Measure Kernel Work Latency with perf kwork

perf kwork splits a stalled kernel path into two questions: how long did the work run, and how long did it sit queued first. A slow softirq and a delayed one look identical from the outside, and treating one as the other sends you fixing the wrong thing.

Allow about fifteen minutes. You need a shell, a perf build that matches the running kernel, and enough permission for perf events. The examples record one second of a harmless sleep, not production traffic, and nothing here touches kernel settings, services or persistent configuration.

• What you need: a working perf binary, a directory you can write a trace into, and the perf-kwork(1) interface, tested here from linux-tools-common version 6.8.0-142.142.
• What it will not touch: no kernel parameters, no services, nothing persistent. The only output is a local perf.data file.
• One thing to remember: a trace can expose workload and system details, so keep the file private.

1. Check the installed tool before recording

Run the read-only checks first. None of them need elevated privileges:

$ command -v perf
/usr/bin/perf
$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
linux-tools-common 6.8.0-142.142
$ perf --version

Here is the trap: the package and the command can both be present while the version-matched perf binary is missing for the running kernel. On this host the wrapper says perf is not installed for kernel 6.8.0-139, even though linux-tools-common is sitting right there. That is a prerequisite failure, not something to work around by guessing at the output.

• Fix it properly. Install the matching distribution package through your normal change process, then rerun the checks.
• Do not reach for sudo. Reading a version number does not need it.

2. Record a bounded workload

Move to a scratch directory and record one second of an idle sleep, purely so there is something in the trace to look at:

$ mkdir -p "$HOME/perf-kwork-test"
$ cd "$HOME/perf-kwork-test"
$ perf kwork record -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote ... bytes perf.data (... samples) ]

Those last two lines are representative perf progress messages; your byte count and sample count will differ. The -- marks the boundary between perf's own options and the workload it is told to run, here sleep 1. Confirm the trace landed before you try to analyse it:

$ test -s perf.data && echo 'perf.data is ready'
perf.data is ready

This step is normally unprivileged. If perf throws a permissions error, read its diagnostic and check the host's perf policy first, rather than reflexively adding sudo.

3. Inspect runtime and delay separately

Run the default report against the file you just made:

$ perf kwork report
             Runtime start      Runtime end        Cpu     Kwork name                 Runtime     Delaytime
                                                                (TYPE)NAME:NUM             (msec)      (msec)
... individual kernel-work events ...

Runtime is time spent actually executing the work. Delaytime is the gap between the work being raised and it actually starting, the queueing you cannot see any other way. Both are shown in milliseconds with microsecond precision, and your event names, CPU numbers and rows will differ because they describe this kernel and this run, not the one in this guide.

For a compact view, pick a sort key and ask for the summary:

$ perf kwork report --sort runtime --with-summary
$ perf kwork report --sort max --with-summary
$ perf kwork report --sort count --with-summary

The report accepts runtime, max and count as sort keys. Change one filter at a time while you are still learning the output; stack several changes together and an empty result becomes hard to trace back to its cause.

4. Measure latency rather than execution time

When the question is "how long did it wait", switch subcommands:

$ perf kwork latency
$ perf kwork latency --sort avg --with-summary

Latency has its own sort keys: avg, max and count. Do not read a high runtime as a high scheduling delay, or a high delay as proof the work itself is expensive: they are different numbers measuring different halves of the story, so pull up both reports before deciding which one is the actual bottleneck.

To zoom in on one kernel-work item, add --name NAME, matched against something you have already seen in the unfiltered report:

$ perf kwork latency --name SCHED

An empty result here means the name is wrong, not that the item is absent. Drop the filter, copy the exact name from the report, and try again. Guessing at a name from a process name or a kernel-version hunch just wastes the fifteen minutes you budgeted.

5. Narrow by CPU or time window

Use a comma-separated CPU list when one processor is the suspect:

$ perf kwork report --cpu 0,1
$ perf kwork latency --cpu 3

The time filter takes seconds and microseconds, and either end can be left blank:

$ perf kwork report --time 1.000000,2.000000
$ perf kwork latency --time ,1.500000
$ perf kwork latency --time 2.000000,

These are timestamps inside the recorded data, not your wall clock, so a window outside the trace simply returns nothing useful. If that happens, drop --time, read the timestamps the plain report shows you, and pick a window that actually exists.

6. Compare with timehist and top

timehist lays kernel-work events out on a timeline and can attach call chains where they are available:

$ perf kwork timehist
$ perf kwork timehist --call-graph --max-stack 8

--max-stack caps the backtrace depth; the documented default is five frames. Symbols need a matching vmlinux, kallsyms file or symbol directory, pointed to with --vmlinux, --kallsyms or --symfs. Missing symbols make the output thinner, not wrong: it is still a latency report, not a runtime report.

top is for task CPU usage, not per-work latency:

$ perf kwork top
$ perf kwork top --sort runtime

Its sort keys are rate, runtime and tid, separate from the ones above. Worth writing down when you are jumping between reports.

7. Finish safely and clean up deliberately

When you are done, only delete the trace if it holds nothing you might still need:

$ rm -- "$HOME/perf-kwork-test/perf.data"

$ chmod 600 "$HOME/perf-kwork-test/perf.data"
$ ls -l "$HOME/perf-kwork-test/perf.data"

None of the report commands modify the trace itself, so you can rerun them with different filters for as long as perf.data sticks around.