Watch Kernel Events Live with trace-cmd stream

trace-cmd stream prints selected kernel trace events to your terminal as they happen, with no trace.dat file left behind. It reads the ftrace buffers, converts the binary records in userspace and writes them straight to standard output, so what you see is what happened, live.

Allow about fifteen minutes for a first run. You need a shell, the trace-cmd package and a kernel with the events or tracer you request. The command normally needs elevated privileges to access tracing. The examples use trace-cmd 3.2.0 from package version 3.2-1ubuntu2. Start with a narrow event selection: broad tracing can produce too much output and can alter the workload you are trying to understand.

1. Confirm the Command and Available Events

Check the installed version first:

$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)

Then ask trace-cmd for the events exposed by this kernel:

$ sudo trace-cmd list -e

Look for an event such as sched_switch, or for a subsystem name such as block. The list is host-specific. If this read fails with a permission error, keep the command unchanged and use sudo; if the event is absent, choose an event that the list actually shows.

2. Stream One Event for a Short Test

Run a short live trace of scheduler switches:

$ sudo trace-cmd stream -e sched_switch

The terminal should begin showing human-readable trace records with a task name, CPU number, timestamp and event details. The process continues until you press Ctrl-C. There is no output file to clean up afterwards.

On a quiet system, the screen may appear idle. That is a valid result: the selected event may simply not be occurring at that moment. Generate a small, harmless workload in a second terminal if needed:

$ sleep 2

Stop the stream with Ctrl-C. This is the normal end of a session. trace-cmd resets the tracing state and disables the tracing it enabled when it finishes, so the test should not leave the selected event running.

3. Trace One Command Instead of the Whole System

To focus on one process, put the command at the end of the invocation. The command runs while tracing is active:

$ sudo trace-cmd stream -e sched_switch -- ls -la /tmp

The command output and trace output share the terminal unless you redirect them. The -- separates the trace-cmd options from the command being run, which stops a command argument that starts with a hyphen being mistaken for a trace-cmd option.

For a quieter terminal, send the workload's normal output away while retaining the live trace:

$ sudo trace-cmd stream -e sched_switch -- sh -c 'ls -la /tmp >/dev/null'

trace-cmd filters out its own tracing threads by default. This avoids making the live stream noisier with the work needed to collect it. Use --no-filter only when observing those trace-cmd threads is itself the question.

4. Select Events and Filter Their Fields

Repeat -e to select more than one event. You can also select a subsystem, an event name or a glob expression, subject to what the kernel exposes:

$ sudo trace-cmd stream -e sched_switch -e sched_wakeup

Place -f immediately after the event it filters. For example, this asks the kernel to keep scheduler switch records whose next task has priority 120:

$ sudo trace-cmd stream -e sched_switch -f 'next_prio == 120'

The field name must exist in that event's format, and the kernel decides which C-style comparisons it accepts. If the filter is rejected, remove it and inspect the event format or try a field present on this kernel. Quote expressions containing operators so the shell does not interpret them.

A filter is not the same as selecting a process. To trace only a particular process ID, use -P:

$ sudo trace-cmd stream -e sched_switch -P 12345

Replace 12345 with a live process ID. If the process creates children and the kernel supports child filtering, add -c. If you have a command rather than a PID, use the command form from the previous step.

5. Use Function Tracing Carefully

A useful diagnostic example, restricted with -l:

$ sudo trace-cmd stream -p function -l 'vfs_*'

Keep the function pattern narrow, run it briefly and stop it if the output rate becomes excessive. Check available tracers before using one:

$ sudo trace-cmd list -p

A requested tracer must be supported by the running kernel. If function is not listed, use an event-based command instead.

6. Capture Output Without Confusing It With a Recording

Because stream writes readable records to standard output, ordinary shell redirection can save what you see:

$ sudo trace-cmd stream -e sched_switch > scheduler-live.txt

This is a text capture made by the shell, not a trace-cmd data file. It will not have the metadata needed for a later trace-cmd report. If you need a replayable recording, use trace-cmd record instead: stream is specifically for live output and does not accept record's -o output-file option.

To watch the output and retain it, use a pipeline such as tee:

$ sudo trace-cmd stream -e sched_switch | tee scheduler-live.txt

Stop the pipeline with Ctrl-C. The text file remains, so remove it afterwards if it contains sensitive process names, paths or timing information:

$ rm -- scheduler-live.txt

7. Recover When a Stream Fails or Gets Noisy

• Missing event. Treat that as a host capability problem, not a reason to guess an event name. Re-run trace-cmd list -e and select an available event. The record manual documents -i to ignore missing events, but silently losing part of a diagnostic is usually the wrong choice for a first investigation.
• Overwhelming output. Stop with Ctrl-C, then restart with fewer events, a process filter or an event field filter.
• Avoid --poll. Unless you specifically need it, since it busy-waits while extracting data and can consume a CPU.
• Avoid -r. Unless you understand real-time scheduling, because it changes the priority of capture threads and can change the system being measured.
• After an interrupted run. Verify that no trace-cmd process is still running and inspect the tracing status if your system permits it.

$ pgrep -a trace-cmd || true
$ sudo trace-cmd stat

If a session used the record option -k, tracing state may deliberately be left enabled. Do not use that option with a normal live investigation unless you have a specific recovery plan. A standard stream run should clean up the tracing it enabled when it exits.