Profile a Live Linux Command with trace-cmd profile

When a full recording would eat too much disk, trace-cmd profile gives you live counts and timings instead. You run a command under it, it collects kernel events as they happen, and it prints a report of counts and timings the moment the command exits, with no trace.dat file left behind.

Allow 10 minutes for a first, small test and more time if you need to narrow a busy workload.

This guide targets trace-cmd 3.2.0, the version installed here. The command needs access to the kernel tracing filesystem, which on a normal workstation usually means elevated privileges, depending on the distribution's tracing policy. The examples do not change persistent configuration, but tracing can add overhead and can expose activity from other tasks.

1. Check the installed command

Confirm that the profile subcommand is available before designing a trace. This check is ordinary user work and changes nothing.

trace-cmd --version
man trace-cmd-profile

Expected output includes a version line such as trace-cmd version 3.2.0. The manpage describes profile as the live equivalent of recording with profiling enabled and then reporting it, with the data read as events arrive.

2. Run the smallest useful profile

Start with a short command and send the profile output to a file. Replace the example command only after this test works.

sudo trace-cmd profile -p nop -e sched_switch -o /tmp/trace-profile.out -- sleep 1
sed -n '1,35p' /tmp/trace-profile.out

Here -p nop disables the default function graph tracer, -e sched_switch selects one scheduler event, and -o writes the report. The -- separates trace-cmd options from the command being profiled. The command itself is only sleep 1, so the trace lasts about one second.

Depending on the kernel and workload, the report contains lines similar to these:

task: sleep-1234
  Event: sched_switch:S (1) Total: 100012345 Avg: 100012345 Max: 100012345 Min:100012345

Timings are nanoseconds. Counts and totals vary on every run, so use the structure of the output rather than copying its numbers into a test.

Checkpoint: if the command reports Permission denied while opening /sys/kernel/tracing, the profile did not run. Do not treat an empty or partial output file as a successful measurement. Check the mount and permissions first:

test -r /sys/kernel/tracing/events && echo tracing-filesystem-readable || echo tracing-filesystem-not-readable
mount | grep tracing

Use the host's approved tracing method, such as sudo for a local administrator, only when you understand who can see the resulting kernel and task activity. If the tracing filesystem is not mounted, follow your distribution's tracing setup documentation rather than creating ad hoc mounts in a production change window.

3. Profile one task and its descendants

For a real command, broad scheduler data can be distracting. -F follows the named task, and -c is also needed when you want forked children included. Test with a command that creates a short-lived child:

sudo trace-cmd profile -p nop -e sched_switch -F sh -c 'sleep 1' -c -o /tmp/trace-profile-tree.out
grep -E '^(task:|  Event:)' /tmp/trace-profile-tree.out | head -n 20

The exact task names and event counts depend on the shell and kernel. Without -c, following a parent does not automatically mean that forked children are followed. Events such as scheduler switches can still mention another task because their fields refer to both sides of a scheduling decision.

4. Choose the tracer deliberately

With no replacement, profile enables several events and function graph tracing at depth one when the kernel supports it. That gives useful call paths, but it can produce a large report. Use -p nop when event accounting is the question, as in the earlier example.

To count function hits instead of measuring function graph timings, use the function tracer:

sudo trace-cmd profile -p function -l 'vfs_*' -o /tmp/trace-profile-functions.out -- sleep 1
grep '^ *Event:' /tmp/trace-profile-functions.out | head -n 20

The -l filter limits the functions considered. Function tracing counts hits; it does not provide the function graph timings that the default setup is intended to show. Keep the filter narrow because tracing every function can create noise and overhead.

5. Take full control when the defaults are too noisy

Use -S when you want only the tracer and events explicitly named on the command line. For example, this enables function graph tracing, limits it to functions matching kmalloc, and requests stack traces for that filter:

sudo trace-cmd profile -S -p function_graph -l '*kmalloc*' -l '*kmalloc*:stacktrace' -o /tmp/trace-profile-kmalloc.out -- sleep 1
grep -E '^(task:|  Event:)' /tmp/trace-profile-kmalloc.out | head -n 20

Tip: quoting the wildcard is essential. Without quotes, the shell may expand it against filenames in the current directory before trace-cmd sees it. With -S, do not assume scheduler events are present: add each required -e event explicitly, and use trace-cmd list -e to inspect event names available on the machine.

6. Save or watch the report

-o is the straightforward choice when you need a report file. If you want to watch the command's own output while keeping the profile separate, use --stderr and redirect standard error:

sudo trace-cmd profile --stderr -- sleep 1 2> /tmp/trace-profile-stderr.out

The manpage states that --stderr redirects the profile output, not the executed command's output. If you use -o instead, remember that the output path is overwritten according to normal shell and program behaviour. Choose a disposable path such as /tmp for experiments, or make a dated destination after checking that it is safe to replace.

7. Stop cleanly and recover from a failed run

Normally, trace-cmd stops when the profiled command exits and prints the accumulated report. Pressing Ctrl-C interrupts the foreground profile; check whether the report was written before interpreting it. A failed run can leave tracing enabled on some systems, so inspect the status before starting another experiment:

sudo trace-cmd stat

Warning: if your administrator's tracing procedure permits it and no other tracing session is using the system, reset the trace buffers with sudo trace-cmd reset. This is state-changing and can remove data belonging to another session, so do not run it on a shared tracing host without checking ownership first. Remove only your disposable reports when you are finished:

rm -f -- /tmp/trace-profile.out /tmp/trace-profile-tree.out /tmp/trace-profile-functions.out /tmp/trace-profile-kmalloc.out /tmp/trace-profile-stderr.out

Done means