Record a Useful Linux Profile with perf record

perf record is the step everyone rushes to reach the flame graph, and where most bad profiles get baked in.

Allow 10 to 20 minutes for a first profile. You need a shell, perf tools matching the running kernel, and a workload you can run safely. Recording is normally read-only from the workload's point of view, but the resulting data file can carry command names, paths and stack information, so keep it out of shared or sensitive directories.

1. Check the tool and choose a workload

Start by checking the kernel and the event list:

$ uname -r
6.8.0-139-generic
$ perf list | head -n 20

You should see the running kernel release, then available hardware and software events. The package and the executable need to agree with the kernel closely enough for the required PMU support. On this host, /usr/bin/perf reports the tools for kernel 6.8.0-139 are missing, even though the manual comes from linux-tools-common 6.8.0-142.142. Install or select the matching, approved kernel tools before blaming the workload for a failed command.

2. Record one command

Use an explicit output path and an event name. cycles is a common starting event, but availability is processor-dependent, so pick a name perf list actually showed you on this machine:

$ perf record -e cycles -o ./perf.data -- ./path/to/workload ARGUMENT
[workload output appears here]
[perf finishes when the workload exits]

This runs the workload and gathers samples without displaying a profile. The -- separates perf's own options from the command and its arguments; keep it if the workload's first argument looks like an option itself. The output file is the recording, not a text report.

Confirm the file actually turned up:

$ test -s ./perf.data && printf '%s\n' 'perf.data was created'
perf.data was created

If your workload writes its own files, run it from a disposable directory. Check the output path before recording again: a fresh recording at the same location overwrites the evidence you meant to compare it against.

3. Read the recording

perf report is the inspection command. Run it against the file you just made:

$ perf report -i ./perf.data

Expect an interactive report of sampled locations, limited by the symbols actually available on the machine. Exit with q. Without an interactive terminal, check that installed build's help for its non-interactive output option instead of guessing one.

The busiest entries are where the samples landed, not automatically the functions worth rewriting. Check first that the workload and event were the ones you meant to capture. A run that is too short leaves too little evidence, and a mostly idle program's profile can be dominated by startup or library code instead of the path you care about.

4. Add call graphs when names are not enough

Record call graphs for user and kernel space with -g:

$ perf record -e cycles -g -o ./perf-callgraph.data -- ./path/to/workload ARGUMENT
$ test -s ./perf-callgraph.data && printf '%s\n' 'call-graph recording was created'
call-graph recording was created

The default user-space method is fp, which relies on frame pointers. If the workload was built without them, expect wrong stacks. Where the installed build supports it, try DWARF with an explicit stack dump size:

$ perf record -e cycles --call-graph dwarf,4096 -o ./perf-dwarf.data -- ./path/to/workload ARGUMENT

DWARF recording adds user stack data to every sample, which increases overhead and file size. The manual's default stack dump size is 8192 bytes; the example above picks 4096 deliberately for a smaller trial run. Compare the result against the frame-pointer version rather than assuming one method wins outright.

5. Profile an existing process or the whole system

For a process already running, pass its PID:

$ perf record -p 12345 -e cycles -o ./pid-12345.data

Stop the recording with Ctrl-C once the behaviour you need has happened, then verify the output file. Swap in a real PID for 12345.

For all CPUs, add -a:

$ sudo perf record -a -e cycles -o ./system.data -- sleep 10
$ test -s ./system.data && printf '%s\n' 'system recording was created'
system recording was created

This captures system-wide activity for the whole ten-second window, not just your shell command, and other users' activity comes along with it. Elevated privilege may be required by the kernel's perf-event policy; only reach for sudo after a permission failure, and remove or protect the file after analysis according to your data-handling rules.

Common traps

• Event rejected: run perf list and pick an event actually available on this CPU. Do not copy a raw rN event from another processor.
• Permission denied: try the smallest scope first, then check the host's perf-event security policy. -a, kernel events and some filters need more privilege than profiling your own command.
• Empty or tiny data: give the workload enough time to actually sample, and confirm it exercised the path you care about.
• Missing or misleading stacks: use -g, check the workload's frame-pointer build setting, and only try --call-graph dwarf once the unwinder it needs is actually available.
• Unexpected coverage: with no explicit target, perf record defaults to system-wide collection across all CPUs. Put a single command after --, and use -p or -a only when you deliberately mean an existing process or every CPU.