Read and Filter a trace-cmd Recording Without Losing the Evidence

A trace file only earns its keep once you can pull the one event out of it that matters, and that is what trace-cmd report is for. You will use it to turn an existing trace.dat recording into readable text, inspect what it contains, and narrow the output to useful events.

The workflow is read-only: it does not start tracing, change kernel tracing settings, or rewrite the recording.

These examples use trace-cmd 3.2.0 from package trace-cmd 3.2-1ubuntu2. Allow about fifteen minutes for a first pass. You need a readable recording made by trace-cmd record, a shell, and enough disk space for any output you redirect to a file. Most commands are unprivileged. Use elevated privileges only if the recording's permissions require it.

1. Check the installed command

Confirm which binary your shell will run and record its version. This is an ordinary, read-only check:

$ command -v trace-cmd
/usr/bin/trace-cmd
$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)
$ dpkg-query -W -f='${Package} ${Version}\n' trace-cmd
trace-cmd 3.2-1ubuntu2

The report command accepts an input file as its final argument or through -i. Unless you name a file, it looks for trace.dat in the current directory. That default is convenient when you are in the recording directory and confusing when you are not.

Checkpoint: put the real path to your recording in a shell variable and verify that it is readable. Do not guess the file name.

$ TRACE_FILE='/path/to/trace.dat'
$ test -r "$TRACE_FILE" && echo 'recording is readable'
recording is readable

2. Produce the first report

Start with a small, inspectable report. Redirecting to a new file keeps the terminal usable for large traces:

$ trace-cmd report -i "$TRACE_FILE" > report.txt
$ test -s report.txt && echo 'report written'
report written
$ sed -n '1,20p' report.txt

A normal report contains a task name and process ID, CPU number, timestamp, event name, and event-specific fields. Function events can also show a caller relationship. Exact lines depend on what was recorded, the kernel, and loaded trace-cmd plugins.

Safety warning: shell redirection truncates an existing destination before trace-cmd starts. Use a new output name, or write to a temporary file and move it only after the command succeeds. The trace input is not changed by this command.

$ TRACE_REPORT="${TRACE_FILE}.report.new"
$ trace-cmd report -i "$TRACE_FILE" > "$TRACE_REPORT"
$ test -s "$TRACE_REPORT" && mv -- "$TRACE_REPORT" "${TRACE_FILE}.report"
$ rm -f -- "$TRACE_REPORT"

The last cleanup is safe here because the temporary name is explicit. If the report fails, inspect the error first and remove only the incomplete temporary file. Never replace the original trace.dat with report text.

3. Discover the recording before filtering it

Use the metadata queries before writing a complicated filter. They can reveal which CPUs contain data, the time range, and the event formats stored in the file:

$ trace-cmd report --cpus -i "$TRACE_FILE"
$ trace-cmd report --first-event -i "$TRACE_FILE"
$ trace-cmd report --last-event -i "$TRACE_FILE"
$ trace-cmd report --events -i "$TRACE_FILE" > event-formats.txt

• The first three commands print recording-specific information and then exit.
• --events lists stored event formats, which is the useful source for filter field names. Do not infer a field name from its abbreviated display value: a report may show a symbol while the format calls the underlying field something else.
• -E has a different purpose: it lists possible events in the file, which is not necessarily the set that actually appears in the recording. Use --events when you need the format and fields for a filter.

4. Select events by name

Use --event when you want matching event names without a field expression. A colon separates the system expression from the event expression:

$ trace-cmd report --event 'sched:sched_switch' -i "$TRACE_FILE"
$ trace-cmd report --event 'read' -i "$TRACE_FILE" > read-events.txt

The first expression matches the sched system and sched_switch event. An expression without a colon can match event names and may also match systems with that text. The argument is a regular expression, so quote it to prevent the shell from interpreting characters that belong to the expression.

5. Filter on event fields

-F applies a field expression. Here is a practical scheduler example that shows scheduler events where the previous task was runnable, or where a wake-up succeeded:

$ trace-cmd report -i "$TRACE_FILE" \
    -F 'sched : prev_state == 0 || success == 1'

The event selection before the colon is the sched system. A field absent from an event evaluates as false, so prev_state is useful for sched_switch while success is useful for sched_wakeup. Values are compared in their recorded form. If output displays R for a numeric state, filter the numeric field value from the event format rather than the displayed letter.

Tip: to test a filter without relying on a large report, add -T. It displays the resulting filter for each event and can expose a misspelt field that would otherwise be silently ignored:

$ trace-cmd report -i "$TRACE_FILE" -T \
    -F 'sched/sched_switch : prev_state == 0'

Put a filter before the first input file when it should apply globally. A filter after -i applies to that input file. This matters when combining recordings. Filters only select output; they do not remove events from the source file.

6. Make timestamps and diagnostics easier to trust

The default timestamp display is normally at microsecond precision even when the recording contains nanosecond timestamps. Add -t for the full timestamp:

$ trace-cmd report -t -i "$TRACE_FILE" > report-full-time.txt
$ trace-cmd report --ts-check -i "$TRACE_FILE" > report-checked.txt

--ts-check warns if timestamps go backwards. That warning is evidence about ordering in the recording, not a repair. Keep the original file and investigate clock sources, merged inputs, or timestamp correction options before drawing a performance conclusion.

For a quick view of the file's recorded metadata, these options are also read-only:

$ trace-cmd report --uname -i "$TRACE_FILE"
$ trace-cmd report --version -i "$TRACE_FILE"
$ trace-cmd report --stat -i "$TRACE_FILE"

Each value is available only when the recording stored it. A missing result does not prove that the host lacks the information; it may simply not have been captured.

7. Diagnose a failure without changing tracing

If the command cannot open the file, check the path and permissions first:

$ ls -l -- "$TRACE_FILE"
$ file -- "$TRACE_FILE"
$ test -r "$TRACE_FILE" && echo readable

Warning: if the file is readable but report rejects it, confirm that it is a trace-cmd recording rather than another trace format, and rerun with the exact installed binary from step 1. Do not use sudo as a general fix: it can hide an ownership problem and may create root-owned report output. Use it only for a deliberate read of a file your account cannot otherwise access, and keep the output in a directory you control.

Plugins can affect event presentation. -N prevents plugins from loading; -L loads only local plugins from ~/.trace-cmd/plugins. Use these when comparing reports, and record which mode you used. They change presentation and processing, not the trace data.

Done means

• You confirmed the installed trace-cmd version and the actual recording path.
• You generated a report without overwriting trace.dat.
• You inspected event formats before choosing field names.
• You can separate event-name matching with --event from field filtering with -F.
• You know whether a filter was global or attached to one input file.
• You checked timestamp ordering and kept the original evidence unchanged.