Manage RDMA QP Counters with rdma statistic

A queue pair is chewing RDMA bandwidth on a live host, and rdma statistic is how you get real counters instead of a guess. It can also bind, unbind or auto-group counters when you actually need to change something, not just read it. The examples use rdma from iproute2 6.1.0-1ubuntu6.4. Give it about fifteen minutes for inspection, longer if you are coordinating around a live workload.

You need the iproute2 package and an RDMA device with a known name and port. Read-only queries run as an ordinary user. The set, bind and unbind commands change kernel counter state and usually need root or the relevant network-management capability. Do not test them on a production port until you know how the application reacts.

1. Check the installed command

Start with the binary and its version. Read-only, no sudo needed:

$ command -v rdma
/usr/bin/rdma
$ rdma -V
rdma utility, iproute2-6.1.0
$ rdma statistic help
Usage: rdma [ OPTIONS ] statistic { COMMAND | help }

The installed help is the contract for this host, not the manpage. This iproute2 build advertises qp as the supported object, type as the automatic-mode criterion, and cntn, lqpn and pid as filters. The manpage is broader and covers MR counters, qp-type, and optional link counters, but do not assume those work here: check rdma statistic help on the target machine first.

2. Find the device name and port

Use the RDMA device name, not the Ethernet interface name. List devices without changing anything:

$ rdma dev show
$ rdma link show

Output is host-specific. A link is written as DEVICE/PORT, for example mlx5_2/1; replace DEVICE/PORT below with a real value from your own output. Both commands empty? Stop here, there is no local port to inspect, and installing a package or guessing a device name will not create one.

$ RDMA_PORT='DEVICE/PORT'
$ printf '%s\n' "$RDMA_PORT"
DEVICE/PORT

3. Read the default and QP counters

These are ordinary read-only queries:

$ rdma statistic show
$ rdma statistic show link "$RDMA_PORT"
$ rdma statistic qp show
$ rdma statistic qp show link "$RDMA_PORT"

Records print when the kernel and driver expose them. On the test host, with no RDMA device, this returned no text and status 0, so an empty result on your host is not proof a counter exists. On a real host, check the device and port in the output before acting on a record.

Filter only when you know the value. The installed help supports counter number cntn, local QP number lqpn, and process ID pid:

$ rdma statistic qp show link "$RDMA_PORT" pid 30489
$ rdma statistic qp show link "$RDMA_PORT" lqpn 178

Those values are examples, not defaults. A filter that returns nothing can mean there is no matching record right now, or that the field is not available on this port. Do not turn a filter into a state change by adding set, bind or unbind until you have actually identified the target.

4. Check the current QP mode

Read the automatic-binding mode before you touch it:

$ rdma statistic qp mode
$ rdma statistic qp mode link "$RDMA_PORT"

The first form reports modes across every device; the second narrows to one link. Record the result. A mode query that prints nothing usually means the driver has no report for that link, so double-check the device name and port rather than assuming a default.

5. Turn on automatic QP counters, deliberately

$ sudo rdma statistic qp set link "$RDMA_PORT" auto type on

The intended effect is one counter grouping per QP type, for new QPs from this point on. It does not retroactively bring every existing object under the new policy. Re-read the mode, then create or observe a test workload:

$ sudo rdma statistic qp mode link "$RDMA_PORT"
$ rdma statistic qp show link "$RDMA_PORT"

There is no config file behind this: the setting lives in kernel and driver state. To stop automatic mode for this scope, use the documented off form:

$ sudo rdma statistic qp set link "$RDMA_PORT" auto off
$ rdma statistic qp mode link "$RDMA_PORT"

That is the recovery action. Counters that remain may still be manually reachable, so check the next listing rather than assuming that turning automatic mode off deletes everything at once.

6. Bind one QP manually

Useful when you want one QP's statistics without automatic grouping. Get the QP's local number from the workload or a filtered listing first. Omit the counter number and let the kernel allocate one:

$ sudo rdma statistic qp bind link "$RDMA_PORT" lqpn 178
$ rdma statistic qp show link "$RDMA_PORT" lqpn 178

The manpage documents an omitted counter ID as a request for a new counter. Already have a counter number? Use the explicit form:

$ sudo rdma statistic qp bind link "$RDMA_PORT" lqpn 178 cntn 4

Confirm cntn 4 actually belongs to the same device and port before using it explicitly. A command that succeeds is not a substitute for checking the resulting listing.

7. Unbind and release a manual counter

$ sudo rdma statistic qp unbind link "$RDMA_PORT" cntn 4 lqpn 178
$ rdma statistic qp show link "$RDMA_PORT" lqpn 178

Omit the QP identifier and the documented behaviour unbinds every QP from that counter:

$ sudo rdma statistic qp unbind link "$RDMA_PORT" cntn 4

Do not use that second form as a cleanup shortcut on a shared port. The manpage says an empty counter is released automatically by the kernel, but an unbind can wipe out live measurements for every QP using it. Unbound the wrong QP or counter? Stop making changes and restore the binding with the same bind form, using verified IDs.

8. Diagnose an empty or rejected result

Keep three questions separate: does the device exist, does the driver expose this statistic, and is the object present right now? Re-run the read-only checks:

$ rdma dev show
$ rdma link show
$ rdma statistic help
$ rdma statistic qp show link "$RDMA_PORT"
$ printf 'status: %s\n' "$?"

• Unknown object or filter: the command syntax is probably ahead of the installed utility.
• Valid syntax, no records: no matching QP or counter is allocated right now.
• Permission error on a change command: you need the right privilege, not a guessed counter ID.

Do not repeatedly retry a state-changing command against a live workload while changing several arguments at once. Go back to the read-only listing and verify one value at a time. If an older manpage example for MR or optional counters matters to your deployment, treat it as a compatibility question: check the target's iproute2 version and its own help before relying on it. This guide does not claim those commands exist in iproute2 6.1.0.