Read a trace.dat Histogram with trace-cmd hist

Run trace-cmd hist when you have a trace.dat recording and want the events and call chains that fire most often, not a scroll of timestamped lines. Give it a few minutes once the recording is in hand.

1. Check the recording before you read it

Put the recording somewhere readable, then check the path is right. A trace file can hold command names, process IDs and timing data, so treat it as operational data, not a throwaway temp file.

trace_file=/var/tmp/trace.dat
test -r "$trace_file" && printf 'Readable: %s\n' "$trace_file"

You should see one line starting Readable:. Nothing printed and a non-zero exit means the path or permissions are wrong. Do not reach for sudo on reflex: it will not fix a wrong filename, and copying a sensitive recording somewhere less protected just doubles your exposure.

2. Read the default trace.dat

With a file called trace.dat sitting in the current directory, this is all you need:

trace-cmd hist

The output is a histogram, not a chronological event report: common events come first, and where the recording has function-tracer data and call stacks, trace-cmd assembles them into call graphs. The exact rows depend on what was enabled during recording, so do not treat one layout as a stable interface to script against.

Checkpoint: confirm you are looking at the file you think you are before you start reading anything into the task names or call chains.

pwd
ls -lh -- trace.dat
trace-cmd hist

A missing default file fails before any histogram appears. On this installation, that looks like:

trace-cmd: No such file or directory
  can't open /tmp/trace-cmd-hist-no-such.dat

The exit status is non-zero. That is an input failure, nothing more: it does not mean the recording holds no events.

3. Point it at a different recording

Use -i when the recording is not called trace.dat:

trace-cmd hist -i /var/tmp/kernel-trace.dat

A final positional argument works the same way:

trace-cmd hist /var/tmp/kernel-trace.dat

Prefer the explicit -i form in notes and scripts, since it makes the input obvious at a glance; the positional form is fine for a quick look. Both only open the file for reading. If the file has been compressed or renamed away from trace-cmd's own format, this command will not decompress or convert it for you: go back to the original recording, or run the conversion step your trace-cmd version provides.

4. Decide whether task separation helps

By default the histogram is per task, so identical call chains under different tasks or PIDs stay separate. Good for "which task did this", less good when a system-wide pattern gets scattered across dozens of small entries.

trace-cmd hist -P -i /var/tmp/kernel-trace.dat

-P compacts everything, ignoring task and PID, and shows the task name as <all pids>. That answers a different question: which call graphs are common across the whole recording, regardless of who ran them. Keep the per-task view when ownership, or one misbehaving process, is what you actually care about.

Tip: run both views when the cause is not obvious. A pattern that only looks big under -P is probably spread across many tasks; a single large per-task entry deserves a narrower look.

5. Handle failures without touching tracing state

trace-cmd hist only analyses. It never starts or stops tracing, clears buffers or alters the source file, so it is safe to rerun as many times as you need while you try different filters or inputs. The traps that catch people out:

If the file is readable but the command still fails, keep the original recording untouched and capture the full diagnostic along with the exit status:

trace-cmd hist -i /var/tmp/kernel-trace.dat >/tmp/trace-cmd-hist.out 2>&1
status=$?
sed -n '1,120p' /tmp/trace-cmd-hist.out
printf 'trace-cmd hist exit status: %s\n' "$status"

Once you have checked it, the temporary diagnostic can go:

rm -- /tmp/trace-cmd-hist.out

That removal only touches the diagnostic file. It never removes the trace recording itself.

Done means