Inspect, Debug and Replace the dmeventd Event Daemon Safely

A thin pool creeps towards full and dmeventd is the daemon meant to notice, so here is how to check it without touching a live volume. You will confirm which dmeventd is installed, understand what it monitors, and collect useful diagnostics. The examples use the installed LVM2 daemon, version 1.02.185 from package dmeventd 2:1.02.185-3ubuntu3.2.

Allow about 15 minutes for inspection. Replacing a live daemon or changing its LVM policies needs a maintenance window and a tested recovery path.

1. Confirm the binary and version

dmeventd is the device-mapper event monitoring daemon. LVM plugins can react to device events for mirrors, RAID, snapshots, thin pools and VDO pools. The daemon is usually involved indirectly when LVM activates monitored devices, so first establish the executable and package version before comparing behaviour with another host.

$ command -v dmeventd
/usr/sbin/dmeventd
$ dmeventd -V
dmeventd version: 1.02.185 (2022-05-18)
$ dpkg-query -W -f='${Package} ${Version}\n' dmeventd
dmeventd 2:1.02.185-3ubuntu3.2

Your path and package manager output may differ. The version printed by the program is the useful comparison. Do not assume that a newer or older LVM2 release has identical plugin thresholds or restart behaviour.

2. Read the available control surface

The daemon has a small command-line interface. Help is a safe, unprivileged query and is useful when a startup script or service wrapper passes options you did not expect.

$ dmeventd -h
Usage:
dmeventd [-d [-d [-d]]] [-f] [-h] [-l] [-R] [-V] [-?]

• -d, one to three times, increases debug detail sent to syslog.
• -f keeps the daemon in the foreground.
• -l sends logging to standard output and standard error, but the manual says it only works with -f; otherwise it is ignored.

That last detail is an easy distraction trap. If you run dmeventd -l and wait for terminal output, you may conclude that logging is broken when the option was ignored because the daemon was not in foreground mode.

3. Inspect the host without starting a second daemon

Before using foreground mode, check whether a daemon is already running. A second instance can contend with the existing monitor, so do not casually launch one as root.

$ pgrep -a dmeventd || echo 'no dmeventd process found'
$ ps -C dmeventd -o pid,user,args

These commands only inspect processes. A host may start dmeventd through an LVM helper rather than a unit named dmeventd.service, so a missing systemd unit does not by itself prove that monitoring is unavailable. If you need the service owner, inspect the distribution's LVM monitoring unit or the process parent before stopping anything.

Use elevated privileges only when the process information or log source is protected. The version, help and process queries normally work as an ordinary user.

4. Capture foreground diagnostics in a maintenance window

Use foreground mode when you need to observe startup or plugin messages directly. This example adds one debug level and directs messages to the terminal:

# sudo dmeventd -f -l -d

For a less intrusive diagnostic, leave the running daemon alone and inspect the system's configured log destination. The manual says normal debug messages go to syslog. The exact command depends on the host:

$ journalctl -b -t dmeventd --no-pager
$ journalctl -b --grep=dmeventd --no-pager

Either query may return no lines. That is not proof of failure: the daemon may use a different syslog identifier, may not be running, or may not have emitted a message during the current boot. Increase debug detail only for a bounded investigation, because -ddd can be noisy.

5. Understand what the LVM plugins can do

The daemon itself reacts to registered device-mapper events; its LVM plugins provide the storage-specific actions. The installed manual describes these broad behaviours:

• Mirror and RAID plugins attempt to handle device failure automatically.
• Snapshot monitoring warns at 80% full, then repeats at 85%, 90% and 95%. A snapshot that runs out of space becomes invalid and may be unmounted when mounted.
• Thin pool data and metadata monitoring warns at the same thresholds. Above 50% usage, the thin plugin can call its configured command at each further 5% increase.
• VDO pool monitoring can call its configured command above 50%, also at 5% increases.

Thin and VDO handling can use LVM's lvextend --use-policies by default when the relevant autoextend policy allows it. A failed command is retried with increasing delays, up to 42 minutes between retries according to this manpage. That can turn a seemingly quiet capacity problem into repeated background activity.

Review lvm.conf(5) before changing a policy. Do not treat a warning as proof that an automatic extension succeeded. Check the pool and filesystem separately with the LVM and filesystem tools appropriate to your host.

6. Treat plugin commands as privileged automation

The thin and VDO plugins can execute configured commands. These commands run with LVM_RUN_BY_DMEVENTD=1, which prevents an LVM command launched by the plugin from recursively interacting with dmeventd. Thin commands can also receive DMEVENTD_THIN_POOL_DATA and DMEVENTD_THIN_POOL_METADATA; VDO commands receive DMEVENTD_VDO_POOL. Those values are absent while processing an error event.

7. Replace a running daemon only as a planned change

The -R option replaces a running dmeventd instance. The new process obtains the devices and events being monitored from the old process, but the manual requires the running daemon to be version 2.02.77 or newer. This is a service-disrupting operation, not a routine health check.

Before using it, record the running version, confirm that the new binary is compatible with the host's LVM libraries and plugins, and arrange a way to restore the previous package. Then use the distribution's documented service control path or an approved maintenance command. Do not paste sudo dmeventd -R into an incident shell without first checking which process it will replace.

$ pgrep -a dmeventd
$ dmeventd -V
dmeventd version: 1.02.185 (2022-05-18)

If a replacement fails, stop and inspect the daemon process, syslog or journal, and the LVM monitoring path before retrying. Recovery normally means restoring the known-good package or service definition and then verifying monitored devices, not repeatedly launching dmeventd by hand.