Measure Scheduler Latency with perf sched

A process burning barely any CPU can still feel sluggish if the scheduler keeps making it wait its turn. perf sched records that wait separately from the run time, so you can tell "not enough CPU time" apart from "CPU time arrived too late".

Allow about fifteen minutes. You need a Linux shell and a matching perf executable. Recording scheduler events may need elevated privileges depending on the kernel's perf permissions. Nothing here changes services or persistent configuration, but a recording can carry process names and timing information, so treat the resulting perf.data as sensitive operational data.

Version checkpoint: the local package is linux-tools-common 6.8.0-142.142. On this machine, /usr/bin/perf reports the matching tool for the running kernel, 6.8.0-139, is missing. If you see the same warning, install or select the matching kernel-tools package through your normal change process before treating a missing binary as a scheduler problem.

1. Check the installed command

Start with the read-only checks; none of them need sudo:

$ command -v perf
/usr/bin/perf
$ perf --version
WARNING: perf not found for kernel 6.8.0-139
$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
linux-tools-common 6.8.0-142.142

Your warning and package version may read differently. What matters is whether perf matches the running kernel closely enough for your distribution. If perf --version prints a version rather than a warning, keep that output with your investigation notes.

2. Record a small, repeatable workload

Start short. This records scheduler events for the lifetime of a sleep:

$ perf sched record -- sleep 1

The double hyphen marks the boundary between perf sched options and the workload. This is normally unprivileged on permissive systems. If the kernel denies access to performance events, rerun the same command with sudo only after checking your local perf policy:

$ sudo perf sched record -- sleep 1

Later reports read perf.data by default, so confirm it exists before moving on:

$ test -s perf.data && echo 'recording is non-empty'
recording is non-empty

A recording is a file, not a live dashboard, and it will not warn you before being overwritten. Move it to a deliberate name first, or work in a separate directory:

$ mv perf.data perf-data-scheduler-test.data
$ perf sched timehist -i perf-data-scheduler-test.data

The move is reversible with mv perf-data-scheduler-test.data perf.data if you change your mind. Do not delete a trace until you know its results and retention needs.

3. Read the event-level report

With the default filename, the shortest report command is:

$ perf sched timehist
            time    cpu  task name             wait time  sch delay   run time
                         [tid/pid]                (msec)     (msec)     (msec)
    ...

Three columns matter here. Wait time is the interval between a task being scheduled out and its next scheduled-in event. Scheduling delay is the interval between wakeup and the task actually running. Run time is how long it ran for that event. Timestamps print as seconds, a dot, and microseconds, while these three duration columns are in milliseconds.

Do not read one large value as a system-wide average. Idle tasks, kernel threads and short-lived workload threads all turn up in the same event stream, and the task name is paired with a thread or process ID, the safer thing to filter on.

4. Produce a compact summary

When the event list gets too noisy, ask for a per-thread summary instead:

$ perf sched timehist --summary
      ... summary by thread with min, max, average and relative stddev ...

Rows depend entirely on the trace. The summary reports minimum, maximum and average run times in seconds, plus relative standard deviation. Want both the event list and the summary together?

$ perf sched timehist --with-summary

Both of these analyse the existing recording. Neither records a second workload, and neither changes scheduling policy.

5. Narrow the trace without recording again

Filter by CPU when a trace spans several processors:

$ perf sched timehist --cpu 0,3

For a known process or thread, use its numeric ID:

$ perf sched timehist --pid 12345
$ perf sched timehist --tid 12345

--time takes a timestamp window read off the report itself, in seconds and microseconds, and either end can be left blank:

$ perf sched timehist --time 79371.870000,79371.900000
$ perf sched timehist --time ,79371.900000
$ perf sched timehist --time 79371.870000,

The first form covers only the chosen interval; the other two run from the start or to the end of the file. Copy timestamps carefully: a typo produces an empty report that looks exactly like a workload failure.

6. Inspect the trace in other views

perf sched map draws a text outline of context switches: columns are CPUs, two-letter shortcuts are tasks, an asterisk marks the CPU where an event happened, and a dot marks an idle CPU:

$ perf sched map
$ perf sched map --compact

--compact earns its keep on high-core-count hosts by showing only CPUs with activity. Use --cpus to focus the map on specific CPUs. It stays plain text throughout, so it still works fine in logs and terminals with no colour support.

For the detailed event-by-event trace behind a summary:

$ perf sched script

The manual describes this as an alias of perf script for now. Reach for it once a summary tells you a delay exists but you need the actual event sequence around it.

7. Handle failure and clean up safely

If perf says it cannot find itself for the running kernel, check the kernel and the distribution's matching tools package:

$ uname -r
6.8.0-139-generic
$ apt-cache policy linux-tools-$(uname -r)
  Candidate: ...

If no candidate is available, do not guess a package name or mix tools from an unrelated kernel. Ask the distribution administrator to supply the matching package, then repeat step 1. If recording is denied, check the kernel's perf permission policy and the account running the test before reaching for sudo again.

$ rm -- perf-data-scheduler-test.data

Keep the file instead if someone else needs to reproduce the report, and restrict access to it given the process names and timing information it holds.