Diagnose and Recover Devlink Health Reporters Safely

A NIC firmware has thrown a fatal error, traffic has stalled, and the driver says it has a health reporter. devlink health shows you how to identify that reporter, read its status, collect diagnostics, inspect or clear its saved dump, and judge whether recovery is safe. It uses iproute2 6.1.0-1ubuntu6.4 from Ubuntu and takes about 10 minutes once you have a device and reporter. Finding the right hardware identifier may take longer.

Before you start

• You need: a Linux host with a devlink-capable driver, the devlink command, and a reporter name the driver has registered.
• Swap the examples. Replace pci/0000:00:09.0, 1 and tx below with values from your host. Do not copy the example identifier as if it identified your hardware.
• Privileges. Reading health status is normally an ordinary command. Recovery, testing, configuration changes and dump management may need elevated privileges, depending on the kernel, driver and your account. Use sudo when the unprivileged command is rejected.
• Timing. If the device carries production traffic, these operations belong in a maintenance window.

Checkpoint: stop here if you do not know the device and reporter. The next step discovers both.

1. List the available reporters

Run the broad query first. It reports the health reporters registered on devlink devices and ports, with their status and configuration.

devlink health show

On an idle host this may print nothing. That is a useful result: this kernel currently exposes no health reporters through devlink. If the command itself is missing, install the distribution's iproute2 package. Confirm the local utility version with:

devlink -V

Keep the complete device path and reporter name from the output.

• Device reporter: a path such as pci/0000:00:09.0.
• Port reporter: the same path with a port index appended, such as pci/0000:00:09.0/1. Do not confuse the port index with a Linux network interface name.

2. Inspect one reporter

Once you have a target, narrow the output to that reporter. This is the least disruptive way to confirm the path and name are right.

devlink health show pci/0000:00:09.0 reporter fw_fatal
devlink health show pci/0000:00:09.0/1 reporter tx

A failed command usually means the device path, port index or reporter name does not match what the driver registered. Go back to devlink health show and copy the spelling exactly. Some reporters support recovery or dumps and others do not, so capability is reporter-specific.

Checkpoint: continue only when the targeted show query identifies the reporter you intend to operate on.

3. Collect diagnostics before changing state

Ask the reporter for diagnostics. This requests data from the driver and does not itself start recovery.

sudo devlink health diagnose pci/0000:00:09.0 reporter fw_fatal

For a port reporter, keep the port suffix:

sudo devlink health diagnose pci/0000:00:09.0/1 reporter tx

Save important output in your incident record. If the driver reports that diagnosis is unsupported, use the status and dump operations it does support. An empty response is not proof that the underlying fault is gone.

4. Show the saved dump

Devlink keeps one dump per reporter. Displaying it can generate a new dump when none is stored, either because the reporter captured one automatically after an error or because this is the first request.

sudo devlink health dump show pci/0000:00:09.0/1 reporter tx

So dump show is more than a passive file read: the first request can ask the driver to create data. Record the output before attempting recovery, because a later event may replace the useful context.

Checkpoint: you now have the reporter identity and, where supported, diagnostic and dump evidence. Ask the device owner before any operation that can affect traffic.

5. Recover only with a deliberate change window

Warning: recovery asks the driver's reporter to perform its recovery operation. It can reset hardware, interrupt traffic or change device state. Confirm that the fault is understood, that a link or service interruption is acceptable, and that you have an alternate access path.

For a device reporter, the form is:

sudo devlink health recover pci/0000:00:09.0 reporter fw_fatal

For a port reporter, use the port path:

sudo devlink health recover pci/0000:00:09.0/1 reporter tx

On success, the reporter's recovery counter increases. Verify the result rather than assuming a successful command means a healthy service:

devlink health show pci/0000:00:09.0/1 reporter tx

Follow that with the device or application checks that suit your environment. If recovery fails, keep the error, diagnostics and dump output. Repeated recovery attempts can hide the original failure and may increase disruption.

6. Use a test event with care

health test triggers a test event on a device reporter. It is for controlled validation of the reporter path, not a general health check. The man page specifies a device path for this operation:

sudo devlink health test pci/0000:00:09.0 reporter fw_fatal

Warning: run it only with the driver's documentation and a maintenance plan to hand. A test can create a dump, invoke automatic recovery or affect traffic, depending on the reporter's implementation. Re-run devlink health show and inspect the dump afterwards to see what happened.

7. Change automatic recovery or dump settings

The set command changes one reporter's policy.

• Grace period: in milliseconds, the interval between consecutive automatic recoveries.
• Boolean settings: control whether devlink automatically recovers from an error and automatically captures a dump.

For example, disable automatic recovery for a device reporter while you investigate it:

sudo devlink health set pci/0000:00:09.0 reporter fw_fatal auto_recover false

Warning: this is a live configuration change and may leave a fault unrecovered. Record the previous setting first:

devlink health show pci/0000:00:09.0 reporter fw_fatal

Restore automatic recovery when the investigation is over:

sudo devlink health set pci/0000:00:09.0 reporter fw_fatal auto_recover true

A recovery interval or automatic dump policy takes the same form:

sudo devlink health set pci/0000:00:09.0 reporter fw_fatal grace_period 3500
sudo devlink health set pci/0000:00:09.0 reporter fw_fatal auto_dump true

Some parameters are unsupported when a reporter lacks the matching recovery or dump method. Treat an unsupported-parameter error as a capability boundary, not a cue to guess another spelling.

8. Clear a dump only after preserving it

Warning: clearing deletes the saved dump for that reporter, and it cannot bring back evidence you did not save elsewhere. Use it only after your incident record holds the useful output:

sudo devlink health dump clear pci/0000:00:09.0/1 reporter tx

Clearing lets a new dump be generated on the next dump show. Verify the resulting state with:

sudo devlink health dump show pci/0000:00:09.0/1 reporter tx

If the driver supports dump generation, this may immediately create fresh output. If not, the command reports that limitation.

Common traps

• Wrong scope: a reporter can belong to the device or to a port. Use the exact path shown by the broad query.
• Assuming every reporter has every feature: diagnosis, recovery, testing and dumps depend on driver support.
• Discarding evidence: dump show can create data, and dump clear removes the saved copy. Capture output before clearing.
• Confusing a command error with a hardware diagnosis: check the path, reporter spelling and privilege level before interpreting the driver's response.

Done means

• Target identified: you know the exact devlink device or port and reporter.
• Evidence captured: status, diagnostics and any useful dump were saved before recovery.
• Risk respected: you treated recovery and testing as potentially disruptive.
• Result verified: you re-checked the reporter after recovery or policy changes.
• Evidence preserved: you saved incident output before clearing a dump.