Read Linux Ftrace Buffers with trace-cmd show

trace-cmd show reads whatever is already sitting in the kernel's Ftrace buffer, and the mode you pick decides whether the data survives the read. Get it wrong and a live pipe eats records you wanted to look at twice. You will end up with a repeatable workflow for a static read, a live pipe and a snapshot, and know which one to reach for.

This guide describes trace-cmd 3.2.0, installed from Ubuntu package version 3.2-1ubuntu2. Allow about ten minutes for a read-only inspection. You need a shell, the trace-cmd package and access to the kernel tracing filesystem. Most reads need elevated privileges on a normally configured host.

1. Check the installed command

Confirm which executable will run and record its version. These are ordinary, read-only commands:

$ command -v trace-cmd
/usr/bin/trace-cmd
$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)
$ trace-cmd show --help

The help output shows the command shape: trace-cmd show [-p|-s] [-c cpu] [-B buf] [options]. Under the hood it reads the kernel's trace file, the same basic operation as reading that file with cat.

2. Read the static trace buffer

Start with the default read. On a host where the tracing filesystem is accessible, it prints whatever is currently in the main buffer and then exits:

$ sudo trace-cmd show
          <...-1234  [002] ....  4182.731000: sched_switch: ...
          <...-1234  [002] ....  4182.731100: ...

• The exact lines vary. They depend on the active tracer, enabled events and kernel. An empty result is also a valid outcome.
• The read is static. Repeating it after tracing has stopped shows the same current contents again, subject to other programs changing the buffer in the meantime.
• Do not treat a permission error as an empty trace. An unprivileged process may see Permission denied reading /sys/kernel/tracing/trace. Retry with sudo only if you are authorised to inspect this host's kernel activity, and if that still fails, check that tracing is mounted and that policy permits access.

$ sudo trace-cmd show > trace-buffer.txt
$ wc -l trace-buffer.txt
0 trace-buffer.txt

The count above is only an example. The shell creates or replaces the destination file before trace-cmd even runs, so pick somewhere you are happy to overwrite. The command itself does not alter the kernel buffer.

3. Stream new records with trace_pipe

Use -p when you want records as they arrive:

$ sudo trace-cmd show -p
          <...-1234  [002] ....  4190.102000: sched_switch: ...

This reads trace_pipe, not trace. It is a consuming read: records handed to your command are removed from the pipe and will not come back a second time just because tracing has stopped. The command also blocks when no data is available, so a quiet terminal is expected, not a hang.

Stop the foreground read with Ctrl-C. That stops this reader, not kernel tracing. If another process is also consuming the pipe, records get divided between the two of you, so use one deliberate consumer when you need a complete live stream.

4. Read a snapshot instead

Use -s to read the snapshot buffer:

$ sudo trace-cmd show -s
          <...-1234  [002] ....  4179.004000: sched_switch: ...

A tracing application creates a snapshot by swapping the active buffer with the snapshot buffer. Once no more swaps happen, the snapshot stays put and reading it does not consume it. That makes -s the right tool when a separate workflow has already captured an event and you need to inspect it more than once.

-p and -s are alternatives, not a combination:

$ sudo trace-cmd show -p -s
trace-cmd: cannot use -p and -s together

Diagnostic wording can vary by build. What matters is that the command refuses the conflicting request instead of silently picking one buffer for you.

5. Narrow the read to a CPU or buffer instance

Use -c when a trace is noisy and you only need one CPU's file. Replace 2 with a CPU that actually exists on the host:

$ sudo trace-cmd show -c 2
          <...-1234  [002] ....  4182.731000: sched_switch: ...

CPU numbering is host-specific: check the bracketed CPU field in ordinary output, or the machine's online CPU list, before picking a number. An invalid or unavailable CPU can produce an error or simply no useful records, depending on the kernel and tracefs layout.

If tracing uses an instance buffer, select it with -B and the exact instance name:

$ sudo trace-cmd show -B my-instance
          <...-1234  [002] ....  4182.731000: sched_switch: ...

The instance must already exist. -B does not create it and does not copy data from the main buffer. If you do not know the name, find it from the tracing setup before running this command.

6. Inspect tracing settings without dumping records

The long options report individual settings instead of trace records: --tracing_on, --current_tracer, --buffer_size, --buffer_total_size, --ftrace_filter, --ftrace_notrace, --ftrace_pid, --graph_function, --graph_notrace and --cpumask.

For example, check whether tracing is enabled:

$ sudo trace-cmd show --tracing_on
1

The output is the current value for the selected tracing instance. It does not enable tracing. Add -B NAME when the setting belongs to a named buffer instance, and use -f if you need the full path of the file being displayed:

$ sudo trace-cmd show -f --current_tracer
/sys/kernel/tracing/current_tracer
none

Paths and values vary with the kernel and configuration. Treat these options as inspection aids, not as proof that a particular event or tracer is active unless the returned value actually says so.

7. Recover from the common mistakes

• Permission denied. Verify the executable, retry with authorised elevation, and check the tracing filesystem.
• Unexpectedly empty output. Tell an empty static buffer apart from a blocked -p read: the former exits, the latter waits for new data.
• Empty snapshot. Confirm that another tracing workflow has actually created one before assuming the feature is broken.
• Consumed live records by accident. There is no undo command in trace-cmd show. Stop the reader and arrange a new capture through the tracing workflow that owns the buffer. Do not reach for trace-cmd reset as a recovery step: that is a separate command that disables tracing and clears trace buffers entirely.