Capture a Focused Kernel Trace with trace-cmd

trace-cmd hooks into Ftrace to show you exactly what the kernel is doing, which beats guessing every time something misbehaves under load. This guide finds a scheduler event, records a short trace while a harmless command runs, and reads the resulting trace.dat back as text, in around fifteen minutes. The examples use trace-cmd 3.2.0 from Ubuntu package 3.2-1ubuntu2 on this machine.

You need a Linux kernel with Ftrace event support and the trace-cmd package. Recording normally needs access to the kernel tracing interface, so be ready to use sudo if the unprivileged check says permission is denied.

1. Check the installed command

Confirm which binary and package version you're using first. This is read-only and needs no elevated privileges:

$ command -v trace-cmd
/usr/bin/trace-cmd
$ dpkg-query -W -f='${Package} ${Version}\n' trace-cmd
trace-cmd 3.2-1ubuntu2
$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)

The package version and the upstream trace-cmd version are two separate facts. Keep both in a bug report, because distributions can backport changes independently.

2. Find an event your kernel exposes

Event names are kernel-dependent. Don't copy an event name from an unrelated host and assume it exists here; ask trace-cmd for the scheduler events instead:

$ trace-cmd list -e '^sched:sched_switch$'
 sched:sched_switch

Your output can differ, or the event can be missing entirely if the running kernel was built without it. A broader query shows the whole scheduler subsystem:

$ trace-cmd list -e '^sched:'

3. Record a short trace

Change to a scratch directory first, so the output location is obvious:

$ mkdir -p "$HOME/trace-cmd-demo"
$ cd "$HOME/trace-cmd-demo"

Now record the scheduler switch event while sleep runs for two seconds:

$ sudo trace-cmd record -e sched:sched_switch sleep 2
CPU0 data recorded at offset=0x...
    ... bytes in size (... uncompressed)
CPU1 data recorded at offset=0x...
    ... bytes in size (... uncompressed)
...
CPU7 data recorded at offset=0x...
    ... bytes in size (... uncompressed)

The exact progress lines vary with CPU count and trace-cmd build. What matters is that the command finishes and writes trace.dat. The form matters too: -e enables an event, and the trailing command is the workload to run while recording.

4. Inspect the file without touching kernel tracing

Confirm the file exists, then ask report to decode it:

$ ls -lh trace.dat
-rw------- 1 root root  ... trace.dat
$ sudo trace-cmd report -i trace.dat | sed -n '1,12p'
          <...>-...  [..] ....  ...: sched_switch: ...

Owner and record count are host-specific; look for sched_switch lines, not an exact timestamp or task name. -i makes the input file explicit, without it report looks for trace.dat in the current directory.

For a quick metadata check:

$ sudo trace-cmd report --events -i trace.dat | sed -n '1,20p'

This shows the event formats stored in the file. If a later report fails to parse an event, the recording may simply lack the format information that build of trace-cmd needs. Keep the original file while you investigate.

5. Check and restore the tracing state

Recording should stop when the workload exits, but check the current state rather than assuming it:

$ sudo trace-cmd stat
... tracing status ...

The status text is kernel- and version-dependent. If a failed or interrupted run left recording enabled, stop it and clear the buffers:

$ sudo trace-cmd stop
$ sudo trace-cmd reset

If you only need to discard the current buffers while leaving the tracing configuration in place, trace-cmd clear is the narrower command. Check its effect in your own workflow before using it alongside another tracer.

Common traps

• Don't guess event names. Use trace-cmd list -e; the running kernel decides what's available, not the last host you worked on.
• Don't confuse a tracer plugin with an event. trace-cmd list -t lists tracers, trace-cmd list -e lists trace events.
• Don't use -T casually. It adds a stack trace to every event and can bloat a capture fast.
• Don't leave a broad capture running. Press Ctrl-C to stop an interactive recording, then check with trace-cmd stat and, if needed, stop followed by reset.
• Don't overwrite a useful trace. Use -o another-trace.dat for a separate output file, or copy the existing file before recording again.