Inspect RDMA Devices, Links and Counters with rdma

An empty result from rdma looks exactly like a broken command, and telling the two apart matters when you're already chasing a fault. This covers a safe RDMA inspection workflow on Linux: identify the tool, list devices and ports, inspect resources, check namespace mode, and pull counters in a form scripts can actually consume, in about fifteen minutes.

The examples use rdma from iproute2 6.1.0-1ubuntu6.4, installed here on Ubuntu.

1. Confirm the installed command

Check the binary and its version first, so you're not quietly relying on documentation for a different iproute2 release:

$ command -v rdma
/usr/bin/rdma
$ rdma -V
rdma utility, iproute2-6.1.0

The package revision and the tool's reported version are related but not identical. Record both if you're attaching output to an incident:

$ dpkg-query -W -f='${Package} ${Version}\n' iproute2
iproute2 6.1.0-1ubuntu6.4

Checkpoint: if command -v finds nothing, install or repair iproute2 through your normal system management process. Don't copy examples from another host until this passes.

2. List RDMA devices and ports

Ask for the device list, then the link list. Leaving out a device or port selects everything:

$ rdma dev show
$ rdma link show

On a host with no RDMA devices, these can produce no output at all and still exit 0. That's different from a failed invocation, so capture the status straight away if you're scripting the check:

$ rdma dev show
$ printf 'exit status: %s\n' "$?"
exit status: 0

When a link does exist, it's addressed as DEVICE/PORT, such as mlx5_2/1. Use the exact names your host prints rather than guessing a vendor-specific one.

3. Request machine-readable output

The global -j flag requests JSON, handy for feeding device, link, resource or statistic output into another tool:

$ rdma -j dev show
[]
$ rdma -j link show
[]
$ rdma -j resource show
[]
$ rdma -j statistic show
[]

Don't assume every object serialises to JSON just because -j was accepted. On this installed 6.1.0 build, rdma -j system show still prints the plain text line netns shared copy-on-fork on. Treat the output contract as object-specific, and test the exact command your parser will actually run.

4. Inspect resources without changing them

Resource reporting covers objects like queue pairs, completion queues, memory regions, protection domains, contexts and shared receive queues. Start broad, then narrow using a device or port from step 2:

$ rdma resource show
$ rdma resource show qp link DEVICE
$ rdma resource show qp link DEVICE/PORT
$ rdma resource show qp link DEVICE/PORT lqpn 0-6

Replace DEVICE and PORT with real values; port 0 is illegal in this syntax. The special port selector - asks for queue pairs with no assigned port yet:

$ rdma resource show qp link DEVICE/- -d
$ rdma resource show qp link DEVICE/- -dd

Use cm_id, cq, ctx or srq when those are the objects relevant to the fault. For example, rdma resource show cm_id dst-port 7174 filters connection-manager IDs by destination port, and rdma resource show cq pid PID filters completion queues by process ID.

5. Check network namespace mode

Read the RDMA subsystem mode before you go investigating a namespace-related problem:

$ rdma system show
netns shared copy-on-fork on

In shared mode, RDMA devices are accessible in every network namespace. In exclusive mode, a device is visible in exactly one. The output can also include the kernel's copy-on-fork state, so parse the whole line rather than just expecting the word shared or exclusive.

Warning: rdma system set netns changes subsystem behaviour. The manpage warns against switching mode while RDMA traffic is active, and going from shared to exclusive can fail with EBUSY when active namespaces and devices exist. Don't run either of these as a diagnostic shortcut. If a design genuinely needs a change, schedule a maintenance window, document the current mode, and plan namespace and device recovery together first.

6. Inspect hardware counters

Read the default counters for all devices, or narrow to one link:

$ rdma statistic show
$ rdma statistic show link DEVICE/PORT
$ rdma statistic qp show link DEVICE/PORT
$ rdma statistic qp show link DEVICE/PORT pid PID

Counter support is driver-specific. A missing counter, an empty result or an error from one device doesn't prove the rdma command itself is broken; check the device and port spelling, then compare against rdma statistic help and the driver documentation.

Optional counters are state changes, not simple queries.

These can alter what telemetry is even available, so use elevated privileges only when the host requires it, and record the previous state. The documented recovery is the matching unset, followed by a fresh rdma statistic mode link DEVICE/PORT check.

7. Use a batch file for repeatable read-only checks

For a short diagnostic bundle, feed commands to the -b batch option on standard input:

$ printf '%s\n' 'dev show' 'link show' 'system show' | rdma -b -
netns shared copy-on-fork on

The device and link commands print nothing on this host, while the system command reports its status. Without -force, the first batch error stops processing. With -force, processing continues, but the final return status is still non-zero if anything failed. Keep batch input controlled and review it before running it: a batch can just as easily contain state-changing commands like link deletion, namespace changes and counter configuration.

Done means