Switch RDMA Network Namespace Mode with rdma system

RDMA devices vanishing the moment you create a network namespace usually means one rdma system setting is wrong. It controls whether RDMA devices stay visible everywhere or get locked to a single namespace. The examples use the installed rdma utility from iproute2 6.1.0-1ubuntu6.4, where the current mode reports as netns shared copy-on-fork on.

Allow about fifteen minutes for an inspection or a planned change. You need an RDMA-capable Linux host, the rdma command from iproute2, and enough privilege to change kernel RDMA state. Reading the mode is unprivileged; changing it returned Operation not permitted without elevated privilege on the test host.

1. Confirm the installed command

Start read-only, so you are not assuming another distribution's iproute2 behaves the same way:

$ command -v rdma
/usr/bin/rdma
$ dpkg-query -W -f='${Package} ${Version}\n' iproute2
iproute2 6.1.0-1ubuntu6.4
$ rdma -V
rdma utility, iproute2-6.1.0
$ rdma system help
Usage: rdma system show [ netns ]
       rdma system set netns { shared | exclusive }

The command is rdma system. The shorter rdma sys spelling also works on this install, but the full form is easier to recognise in a script.

2. Check the current mode

Ask before you touch anything:

$ rdma system show
netns shared copy-on-fork on

The word that matters is shared: in shared mode, an RDMA device is accessible from every network namespace. The trailing copy-on-fork on is status text from this version of the utility, not a value you pass to rdma system set. The set command only accepts shared or exclusive.

3. Choose shared or exclusive, deliberately

• Use shared when RDMA devices should stay visible across network namespaces and isolation is not required. This is the mode on the tested host.
• Use exclusive when a dedicated RDMA device must belong to one network namespace. Set this before creating that namespace: the manpage warns that switching from shared to exclusive fails with EBUSY when active namespaces and one or more RDMA devices already exist.

This is a system-wide choice, not a per-device switch. Do not change it while RDMA traffic is live unless you have deliberately accepted the risk: it can affect how applications find devices, so schedule it with the same care as any other networking change.

4. Apply the chosen mode

# Keep RDMA devices visible in every network namespace
$ sudo rdma system set netns shared

# Or isolate an RDMA device to one network namespace
$ sudo rdma system set netns exclusive

Do not add copy-on-fork, a device name or a namespace name; none of those are accepted arguments here. Keep shared or exclusive literal rather than interpolating an unchecked shell variable. Without enough privilege, the tested utility reports:

$ rdma system set netns shared
error: Operation not permitted

That means the change did not happen. Re-run with the privilege your host's policy requires, then verify in the next step. Do not treat sudo as a fix for EBUSY: an active namespace or RDMA device still needs handling safely.

5. Verify the result

Always read the mode back after a successful set:

$ rdma system show
netns exclusive copy-on-fork on

Expect netns shared or netns exclusive depending on what you chose. The rest of the line can carry version-specific status text, so check the mode field, not the whole line byte for byte:

mode=$(rdma system show | awk '$1 == "netns" { print $2; exit }')
case "$mode" in
    shared|exclusive) printf 'RDMA netns mode: %s\n' "$mode" ;;
    *) printf 'Unexpected rdma system output\n' >&2; exit 1 ;;
esac

That script validates the mode field only, not whether an application can actually open an RDMA device. Test the real workload separately, inside the namespace where it is meant to run.

6. Recover from a wrong choice

Return to the intended value during a maintenance window:

$ sudo rdma system set netns shared
$ rdma system show
netns shared copy-on-fork on

Swap in exclusive only when isolation is the deliberate target and its prerequisites are met. The undo is not a substitute for stopping traffic first. If a change fails, keep the last verified mode, inspect active namespaces and RDMA users, and plan the retry rather than repeating the command.

For a failed exclusive attempt, remember the documented EBUSY boundary: shared mode stays in effect when the change is rejected. Confirm with rdma system show before touching anything else.