Run Berkeley DB Deadlock Detection with db5.3_deadlock

Two processes are each waiting on a lock the other holds, and your Berkeley DB application has stopped dead, which is a job for db5.3_deadlock. You will run its detector against an existing environment, choose how it picks a victim, and stop it cleanly. Allow about fifteen minutes for a one-shot check, longer if you add a supervised background process.

Warning: the detector can abort a lock request. Use it only with an environment whose application owners understand the recovery behaviour.

This guide describes the installed Berkeley DB 5.3.28 utility. On this machine the packages are db5.3-util version 5.3.28+dfsg2-7 and db-util version 1:5.3.21ubuntu2. The executable names are db5.3_deadlock and the compatible alias db_deadlock.

1. Confirm the installed utility

Run these ordinary, read-only checks before touching a database environment:

$ command -v db5.3_deadlock
/usr/bin/db5.3_deadlock
$ db5.3_deadlock -V
Berkeley DB 5.3.28: (September  9, 2013)
$ command -v db_deadlock
/usr/bin/db_deadlock

-V prints the library version and exits. The alias reports the same version here. Do not assume that two different detectors can safely share one environment. Use the name and library version that match the application and its Berkeley DB build.

2. Identify the environment and its owner

The detector needs an already-created Berkeley DB environment, including its shared memory regions. It does not create those regions. Start the application that owns the environment first, then identify its environment home and the account that normally opens it.

The -h option selects the home directory. If you omit it, the utility uses the current working directory unless DB_HOME supplies a home. Make the path explicit in scripts so a changed working directory cannot point the detector at the wrong environment:

$ DB_HOME='/srv/example-db'
$ test -d "$DB_HOME" && echo 'environment home exists'
environment home exists
$ db5.3_deadlock -h "$DB_HOME" -v

Without -t, that command performs one detector pass and exits. The final command is safe only when /srv/example-db is a real, active Berkeley DB environment. Replace the placeholder with your actual path. Do not point it at a directory merely because it contains files with familiar database names.

Run the detector as the account that can open the environment. Use sudo only when your deployment deliberately requires it and the environment permissions demand it. Elevated privileges do not make an absent environment valid, and running as root can create ownership problems for later application starts.

3. Choose the victim policy

When the detector finds a deadlock, one lock request must be rejected so the waiting cycle can move again. With no -a option, this installed utility chooses a request at random. That is a poor default for production, because the interrupted work can differ from incident to incident.

Use one of these selectors when the application team has an agreed policy:

• -a m: choose the locker holding the most locks.
• -a n: choose the locker holding the fewest locks.
• -a o: choose the oldest lock.
• -a W: choose the locker holding the most write locks.
• -a w: choose the locker holding the fewest write locks.
• -a y: choose the youngest lock.

These choices affect which request is aborted. They do not stop the application creating another deadlock. The application must handle the resulting Berkeley DB deadlock error and retry or roll back according to its transaction design.

There is a separate selector, -a e, for a timed-out lock request. Use it only when lock or transaction timeouts have been configured. It is not a general replacement for a deadlock victim policy.

4. Run one controlled pass

For a maintenance check, use an explicit home, an explicit victim policy and verbose output:

$ db5.3_deadlock -h /srv/example-db -a W -v

Verbose mode reports each detector run. The command normally prints nothing useful when no deadlock needs attention, so blank output is not proof that the environment was never checked. Capture the shell status immediately:

$ status=$?
$ printf 'db5.3_deadlock exit status: %s\n' "$status"
db5.3_deadlock exit status: 0

Exit status 0 means the utility completed successfully; a value greater than 0 means an error occurred. If the status is non-zero, check the exact path, permissions, environment state and application logs. Do not keep retrying against an uncertain home directory.

5. Run periodic detection when the workload needs it

For multiple processes or threads where at least one process modifies the database, the detector can run as a background process. The -t value is seconds.microseconds. It sets how often the utility checks whether a process is waiting for a lock:

$ db5.3_deadlock -h /srv/example-db -a W -t 5.000000 -v

Keep this process under the same service manager and account policy as the application. Send its output through that manager or to a reviewed log destination. The detector is part of the database's operational path, so a second unmanaged copy can produce confusing diagnostics and competing policy decisions.

A detector that exits immediately after a one-shot pass is not a daemon. If you expect continuous protection, check that the process stays alive and that its supervisor records a clean status:

$ pgrep -af 'db5.3_deadlock.*srv/example-db'
$ printf 'detector status: %s\n' "$?"
detector status: 0

Tip: the process search is only a quick check and can miss a differently formatted command line. For a deployed process, prefer your service manager's status command.

6. Stop the detector cleanly

Warning: do not use an abrupt kill as the normal shutdown path. The manual says the utility must detach from the Berkeley DB environment and release its resources cleanly. Send an interrupt signal, SIGINT, to the detector process:

$ kill -INT DETECTOR_PID
$ wait DETECTOR_PID
$ printf 'detector exit status: %s\n' "$?"
detector exit status: 0

Replace DETECTOR_PID with the real process ID. If a service manager owns the process, use its normal stop operation only when it sends or permits a graceful interrupt.

The optional -L /path/to/detector.log records startup information, including the process ID and start time. That log file is removed when the utility exits gracefully, so do not use it as permanent audit storage.

Recovery: if a forced termination already happened, let the application owner check the environment before restarting database writers.

7. Diagnose the common traps

• No shared regions: start the Berkeley DB application first. The detector does not create the regions.
• Wrong home: use -h or inspect DB_HOME; the current directory is an easy accidental default.
• Unexpected victim: specify -a rather than accepting random selection, and make sure the application handles an aborted request.
• No repeated checks: add -t sec.usec; omitting it deliberately makes the utility run once.
• Timeout confusion: choose -a e only if the environment has lock or transaction timeouts configured.
• Permission failures: check the service account, directory permissions and environment ownership before considering elevated privileges.

Do not use this utility as a repair or recovery command. It detects lock conflicts and rejects requests. It does not rebuild databases, restore transactions or decide whether application data is correct.

Done means

• Utility confirmed. You checked the installed utility and Berkeley DB library version.
• Right environment. The detector was pointed at an existing, active environment with an explicit home path.
• Policy chosen. A victim policy was picked deliberately, or the random default was accepted knowingly.
• Periodic mode supervised. It uses an explicit -t interval and runs under a supervisor.
• Owner informed. The application owner knows that a deadlock victim's lock request will be rejected.
• Clean stop. The detector is stopped with SIGINT, and forced termination is not used as routine cleanup.