Measure Linux Lock Contention with perf lock

A CPU that is not maxed out but still feels slow is often waiting on a lock, not on work. perf lock records exactly which locks got contended, how often, and for how long, so you stop guessing from CPU graphs alone.

Allow 15 to 30 minutes for a first run, plus whatever time it takes to reproduce the workload. This guide follows the perf lock command documented with Debian's linux-tools-common package.

Before you start

You need a Linux host with perf and its matching kernel tools, a workload that is safe to run, and enough permission to collect the required perf events. The installed package on this machine is linux-tools-common 6.8.0-142.142. Its wrapper currently reports no perf binary installed for kernel 6.8.0-139, so the examples below are checked against the perf-lock(1) manual but cannot be run on this host until matching tools go in.

$ command -v perf
$ perf --version
$ man perf-lock

• Version warning or missing binary? Stop and install the package matching the running kernel through your normal process. Do not paper over it by copying an unrelated binary into a system directory.
• Collection denied as an ordinary user? That is the kernel's perf event security policy, not a bug. It may need a permitted capability or an administrative invocation depending on host policy.
• Handling a capture? Treat it as potentially sensitive: it can expose process names, addresses and call paths. Use a directory with sensible permissions and remove data once it is no longer needed.

1. Record a small, reproducible workload

perf lock record writes perf.data in the current directory, capturing lock events from the child command's start to its exit:

$ mkdir -p "$HOME/perf-lock-run"
$ cd "$HOME/perf-lock-run"
$ perf lock record -- /path/to/workload --one-test-case
$ test -s perf.data && echo "recording created"
recording created

Swap the placeholder for a real, bounded workload, and quote paths and arguments rather than assembling an option string from untrusted input. The -- marker separates perf's own options from the child command; the documented form is perf lock record <command>.

This step writes a new file and nothing else, but an existing perf.data gets replaced without asking. Save an earlier capture first if it still matters:

$ test ! -e perf.data || cp --preserve=all perf.data perf.data.previous

2. Confirm the capture before analysing

$ printf 'record exit status: %s\n' "$?"
record exit status: 0
$ ls -lh perf.data

3. Read the aggregate report

perf lock report reads statistics straight from perf.data:

$ perf lock report
Name              acquired  contended   avg wait (ns)
mutex-name              ...         ...              ...

Names and numbers depend entirely on the workload. What matters is three questions: how often a lock was acquired, how often acquisition had to wait, and how long those waits took. When the default order hides the problem, sort by something else:

$ perf lock report --key=contended
$ perf lock report --key=wait_total
$ perf lock report --key=wait_max

Available keys are acquired, contended, avg_wait, wait_total, wait_max and wait_min. A high acquisition count on its own is not contention. Start with contended or a wait-time key, then check the result against the workload's actual latency.

4. Change the report fields and scope

Keep output small when you are comparing runs. --field takes the same fields as a comma-separated list:

$ perf lock report --field=contended,avg_wait,wait_max
$ perf lock report --threads --field=acquired,contended,avg_wait

--threads switches to per-thread statistics, useful once a process-wide total tells you contention exists but not which worker owns it. --entries=20 caps how much you have to scroll through on a first pass:

$ perf lock report --threads --entries=20 --key=contended

--combine-locks merges several instances by lock class name, which surfaces a class-level hotspot but throws away instance-level detail. Rerun without it before deciding which address or call site to change.

5. Inspect raw events and metadata

When the aggregate is not enough, go straight to the raw data:

$ perf lock script
$ perf lock info --threads
$ perf lock info --map

script dumps raw lock events. info --threads lists the threads present in the capture, and info --map shows the address-to-name map for lock instances. All three are more verbose and can carry identifiers you should not send outside the host without review.

To point at a different capture explicitly:

$ perf lock report --input=/path/to/perf.data
$ perf lock script --input=/path/to/perf.data

Do not assume a bare command is reading the file you meant. The default is perf.data unless standard input is a FIFO. An explicit path is safer in scripts, and essential when several captures sit in the same directory.

6. Use live BPF contention collection only when needed

perf lock contention collects statistics with BPF instead of reading an existing capture:

$ sudo perf lock contention --use-bpf --entries=20
$ sudo perf lock contention --use-bpf --threads --key=contended

These commands can need elevated privileges and can see activity beyond your own shell. Use sudo only when host policy requires it, and get approval before running on a shared production system. The BPF map defaults to a maximum of 16384 entries, adjustable with --map-nr-entries for a bigger workload. --max-stack defaults to 8 and --stack-skip to 3; changing either affects collection cost and attribution, so note any non-default values alongside your results.

Filters cut the result set down. --type-filter=mutex picks a lock type, --lock-filter=NAME picks addresses or names, and --callstack-filter=STRING matches a substring in the call stack. A filter like irq is deliberately broad and can match more than one symbol, so test it against an unfiltered run before trusting an empty result.

7. Clean up and recover from common mistakes

There is no undo for analysis, since reading commands never touch the capture. Only remove captures you have finished reviewing, and keep a copy if the result feeds an incident or a performance investigation:

$ rm -- perf.data

$ mv -- perf.data.previous perf.data