Capture a Focused Kernel Trace with trace-cmd

You will discover a scheduler event, record a short trace while a harmless command runs, and read the resulting trace.dat as text. The examples use trace-cmd 3.2.0 from Debian package 3.2-1ubuntu2 on this machine.

Allow about fifteen minutes. You need a Linux kernel with Ftrace event support and the trace-cmd package. Recording normally needs access to the kernel tracing interface, so be ready to use sudo if the unprivileged check says permission is denied.

1. Check the installed command

First confirm which binary and package version you are using. This is read-only and does not need elevated privileges:

$ command -v trace-cmd
/usr/bin/trace-cmd
$ dpkg-query -W -f='${Package} ${Version}\n' trace-cmd
trace-cmd 3.2-1ubuntu2
$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)

The package version and the upstream trace-cmd version are separate pieces of information. Keep both in a bug report because distributions can backport changes.

2. Find an event your kernel exposes

Event names are kernel-dependent. Do not copy an event name from an unrelated host and assume it exists here. Ask trace-cmd for scheduler events:

$ trace-cmd list -e '^sched:sched_switch$'
 sched:sched_switch

Your output can differ, or the event can be absent if the running kernel was built without it. A broader query can show the available scheduler subsystem:

$ trace-cmd list -e '^sched:'

Checkpoint: continue only when the event you intend to record appears in the list. trace-cmd list -t is the corresponding read-only query for available tracer plugins; it is not a list of events.

3. Record a short trace

Change to a scratch directory first so the output location is obvious:

$ mkdir -p "$HOME/trace-cmd-demo"
$ cd "$HOME/trace-cmd-demo"

Now record the scheduler switch event while sleep runs for two seconds:

$ sudo trace-cmd record -e sched:sched_switch sleep 2
CPU0 data recorded at offset=0x...
    ... bytes in size (... uncompressed)
CPU1 data recorded at offset=0x...
    ... bytes in size (... uncompressed)
...
CPU7 data recorded at offset=0x...
    ... bytes in size (... uncompressed)

The exact progress lines vary by CPU count and trace-cmd build. The useful result is that the command finishes and writes trace.dat. The command form matters: -e enables an event, and the final command is the workload to run while recording.

If trace-cmd reports that it cannot open the tracing directory or lacks permission, retrying with sudo is reasonable. Do not make the trace broader to solve a permission error.

4. Inspect the file without changing kernel tracing

Confirm that the file exists, then ask report to decode it:

$ ls -lh trace.dat
-rw------- 1 root root  ... trace.dat
$ sudo trace-cmd report -i trace.dat | sed -n '1,12p'
          <...>-...  [..] ....  ...: sched_switch: ...

The owner and the number of records are host-specific. Look for lines containing sched_switch, not an exact timestamp or task name. The -i option makes the input explicit; without it, report looks for trace.dat in the current directory.

For a quick metadata check, use:

$ sudo trace-cmd report --events -i trace.dat | sed -n '1,20p'

This shows event formats stored in the file. If a later report cannot parse an event, the recording may not contain the format information needed by that trace-cmd build. Keep the original file while investigating.

5. Check and restore the tracing state

Recording should stop when the workload exits, but inspect the current state rather than assuming it:

$ sudo trace-cmd stat
... tracing status ...

The status text is kernel- and version-dependent. If a failed or interrupted run left recording enabled, stop recording and clear the buffers:

$ sudo trace-cmd stop
$ sudo trace-cmd reset

If you only need to discard the current buffers while leaving the tracing configuration in place, trace-cmd clear is the narrower command. Check its effect in your workflow before using it alongside another tracer.

6. Avoid the common traps

• Do not guess event names. Use trace-cmd list -e; the running kernel decides what is available.
• Do not confuse a tracer plugin with an event. trace-cmd list -t lists tracers, while trace-cmd list -e lists trace events.
• Do not use -T casually. It adds a stack trace to each event and can make a capture much larger.
• Do not leave a broad capture running. Press Ctrl-C to stop an interactive recording, then use trace-cmd stat and, if necessary, stop followed by reset.
• Do not overwrite a useful trace. Use -o another-trace.dat for a separate output file, or copy the existing file before recording again.