Watch Kernel Events Live with trace-cmd stream

You will finish with a live view of selected Linux kernel trace events, filtered to a useful workload and written to your terminal as they happen. The examples use trace-cmd 3.2.0 from package version 3.2-1ubuntu2. This command uses the kernel's ftrace buffers, converts the binary data in userspace and writes readable records to standard output. It does not create a trace.dat file.

Allow about fifteen minutes for a first run. You need a shell, the trace-cmd package and a kernel with the events or tracer you request. The command normally needs elevated privileges to access tracing. Start with a narrow event selection: broad tracing can produce too much output and can alter the workload you are trying to understand.

1. Confirm the command and available events

Check the installed version first:

$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)

Then ask trace-cmd for the events exposed by this kernel:

$ sudo trace-cmd list -e

Look for an event such as sched_switch, or for a subsystem name such as block. The list is host-specific. If this read fails with a permission error, keep the command unchanged and use sudo; if the event is absent, choose an event that the list actually shows.

2. Stream one event for a short test

Run a short live trace of scheduler switches:

$ sudo trace-cmd stream -e sched_switch

The terminal should begin showing human-readable trace records with a task name, CPU number, timestamp and event details. The process continues until you press Ctrl+C. There is no output file to clean up afterwards.

On a quiet system, the screen may appear idle. That is a valid result: the selected event may simply not be occurring at that moment. Generate a small, harmless workload in a second terminal if needed:

$ sleep 2

Stop the stream with Ctrl+C. This is the normal end of a session. trace-cmd resets the tracing state and disables the tracing it enabled when it finishes, so the test should not leave the selected event running.

3. Trace a command rather than the whole system

To focus on one process, put the command at the end of the invocation. The command is run while tracing is active:

$ sudo trace-cmd stream -e sched_switch -- ls -la /tmp

The command output and trace output share the terminal unless you redirect them. The -- separates the trace-cmd options from the command being run, which prevents a command argument that starts with a hyphen from being mistaken for a trace-cmd option.

For a quieter terminal, send the workload's normal output away while retaining the live trace:

$ sudo trace-cmd stream -e sched_switch -- sh -c 'ls -la /tmp >/dev/null'

trace-cmd filters out its own tracing threads by default. This avoids making the live stream noisier with the work needed to collect it. Use --no-filter only when observing those trace-cmd threads is itself the question.

4. Select events and filter their fields

Repeat -e to select more than one event. You can also select a subsystem, an event name or a glob expression, subject to what the kernel exposes:

$ sudo trace-cmd stream -e sched_switch -e sched_wakeup

Place -f immediately after the event it filters. For example, this asks the kernel to keep scheduler switch records whose next task has priority 120:

$ sudo trace-cmd stream -e sched_switch -f 'next_prio == 120'

The field name must exist in that event's format, and the kernel decides which C-style comparisons it accepts. If the filter is rejected, remove it and inspect the event format or try a field present on this kernel. Quote expressions containing operators so the shell does not interpret them.

A filter is not the same as selecting a process. To trace only a particular process ID, use -P:

$ sudo trace-cmd stream -e sched_switch -P 12345

Replace 12345 with a live process ID. If the process creates children and the kernel supports child filtering, add -c. If you have a command rather than a PID, use the command form from the previous step.

5. Use function tracing carefully

The -p option selects a tracer rather than an ordinary event. A useful diagnostic example is:

$ sudo trace-cmd stream -p function -l 'vfs_*'

The function tracer is restricted here with -l. The installed manual warns that enabling function stacks without successfully limiting the function tracer can live-lock a machine. Treat function tracing as a system diagnostic, not as a casual default. Keep the function pattern narrow, run it briefly and stop it if the output rate becomes excessive.

Check available tracers before using one:

$ sudo trace-cmd list -p

A requested tracer must be supported by the running kernel. If function is not listed, use an event-based command instead.

6. Capture output without confusing it with a recording

Because stream writes readable records to standard output, ordinary shell redirection can save what you see:

$ sudo trace-cmd stream -e sched_switch > scheduler-live.txt

This is a text capture made by the shell, not a trace-cmd data file. It will not have the metadata needed for a later trace-cmd report. If you need a replayable recording, use trace-cmd record instead; stream is specifically for live output and does not accept record's -o output-file option.

To watch the output and retain it, use a pipeline such as tee:

$ sudo trace-cmd stream -e sched_switch | tee scheduler-live.txt

Stop the pipeline with Ctrl+C. The text file remains, so remove it afterwards if it contains sensitive process names, paths or timing information:

$ rm -- scheduler-live.txt

7. Recover when a stream fails or becomes noisy

If trace-cmd reports that an event is missing, treat that as a host capability problem, not a reason to guess an event name. Re-run trace-cmd list -e and select an available event. The record manual documents -i to ignore missing events, but silently losing part of a diagnostic is usually the wrong choice for a first investigation.

If output overwhelms the terminal, stop with Ctrl+C, then restart with fewer events, a process filter or an event field filter. Avoid --poll unless you specifically need it: it busy-waits while extracting data and can consume a CPU. Avoid -r unless you understand real-time scheduling, because it changes the priority of capture threads and can change the system being measured.

After an interrupted run, verify that no trace-cmd process is still running and inspect the tracing status if your system permits it:

$ pgrep -a trace-cmd || true
$ sudo trace-cmd stat

If a session used the record option -k, tracing state may deliberately be left enabled. Do not use that option with a normal live investigation unless you have a specific recovery plan. A standard stream run should clean up the tracing it enabled when it exits.