Debug and Safely Replay udev Events with udevadm

By the end of this guide you will be able to inspect a device, check udev rules, simulate an event, and replay only a narrowly selected event set. These examples target the installed systemd 255 build. Allow about 10 minutes. You need a shell; several commands that change daemon state or replay events require root.

Before you begin

Keep a second terminal available if you are investigating a device being added or removed. Do not start with a system-wide trigger. A trigger asks the kernel to emit events again, which can run rules and cause device changes. Begin with --dry-run, then add a subsystem or device match.

The examples use /dev/null because it is present on ordinary Linux systems. Replace it with the device you actually need to investigate, such as /dev/sdb, and check the path carefully before using a privileged command.

1. Inspect the device database

Query the udev database with a device node, a /sys/ path, or a systemd device unit. This asks udev what it already knows; it does not replay an event.

udevadm info --query=property --property=DEVNAME,DEVPATH /dev/null
udevadm info --query=property --property=DEVNAME --value /dev/null

The first command prints named properties. On this machine it includes DEVPATH=/devices/virtual/mem/null and DEVNAME=/dev/null. The second prints only /dev/null. The --value form is useful in scripts, but only with a property query and not with --export.

For rule writing, udevadm info --attribute-walk /dev/null prints matching sysfs attributes up the device chain. Treat the output as clues for a rule, not as a reason to copy every attribute into one. Prefer stable identifiers and test the resulting rule.

2. Validate a rule before installing it

udevadm verify checks udev rules for syntactic, semantic, and stylistic errors. You can pass one or more files. The following harmless example checks a temporary file and does not install or load it:

rule_file=$(mktemp /tmp/udevadm-guide-XXXX.rules)
printf '%s\n' 'SUBSYSTEM=="misc", KERNEL=="null", TAG+="udevadm-guide-example"' > "$rule_file"
udevadm verify "$rule_file"
rm -- "$rule_file"

Expected output contains Success: 1 and Fail: 0. The temporary file is only a validation fixture. If you are checking the live rules, run udevadm verify without a filename to scan the rule directories that the daemon processes.

Do not confuse verification with activation. A valid rule still needs to be placed in a udev rules directory, then the daemon must reload its rules. Reloading does not apply new rules to devices that already exist.

3. Simulate one event

Use udevadm test to simulate rule processing for one sysfs device. It prints debug output and does not run programs specified by RUN rules. The command is therefore useful for seeing properties and rule decisions, but its output can differ from a real event.

udevadm test --action=add /sys/devices/virtual/mem/null

Look near the end for properties such as DEVPATH, DEVNAME, MAJOR, and MINOR. The installed command also reports that it is for debugging only. A failed simulation is not proof that the real device is broken: permissions, timing, and event context can differ.

To test a built-in operation, first list the supported built-ins, then provide a command and a device path:

udevadm test-builtin --help
udevadm test-builtin hwdb /sys/devices/virtual/mem/null

Use the exact device path and built-in name shown by your installed command. The available list is version-specific.

4. Reload rules, then replay narrowly

After installing a corrected rule, reload the daemon's rules and related databases. This needs elevated privileges:

sudo udevadm control --reload

Check the daemon separately:

sudo udevadm control --ping

A successful ping produces no useful data for a human, but exits successfully. A permission error means the command could not contact the daemon from your current context; it does not demonstrate that the daemon is stopped.

Preview a scoped replay before doing it. This lists the matching device without sending the event:

udevadm trigger --dry-run --verbose --subsystem-match=misc --name-match=/dev/null

Only if that list is exactly what you intend, replay the change event and wait for events from this command to finish:

sudo udevadm trigger --subsystem-match=misc --name-match=/dev/null --action=change --settle

Do not use --initialized-nomatch casually during boot. The manpage warns that rules depending on parent devices can leave an unstable final state. For broad recovery work, select the smallest subsystem or parent path that demonstrates the problem.

5. Wait for the result

udevadm settle waits for the whole current event queue, with a default timeout of 120 seconds. trigger --settle waits only for events that trigger command created. If a script needs one device to exist and be initialised, use udevadm wait:

udevadm wait --timeout=10 /dev/null
udevadm settle --timeout=0

The first command checks one device and fails if it is not ready within 10 seconds. The second performs an immediate queue check. A zero timeout does not wait.

Where iocost fits

On this installation, /usr/lib/udev/rules.d/90-iocost.rules applies an I/O cost solution to block disks when the udev property IOCOST_SOLUTIONS is present. The /etc/systemd/iocost.conf configuration can set TargetSolution=, added in systemd 254. Prefer an administrator drop-in such as /etc/systemd/iocost.conf.d/50-local.conf over editing a vendor file. Drop-ins are read in lexical order, with later values overriding earlier single-value settings.

Before selecting a target solution, inspect the device's udev properties and available metadata. A solution missing from a device falls back to the first solution listed in IOCOST_SOLUTIONS. Changing this configuration affects I/O isolation and should be tested during a maintenance window.

Done means

• You can query the target device with udevadm info.
• Your rule passes udevadm verify before installation.
• You used udevadm test to inspect rule processing without running RUN commands.
• You previewed a trigger and restricted it by device or subsystem before using root.
• You waited for the relevant device or event queue and recorded the result.