Find Hot Instructions in perf.data with perf annotate

perf annotate reads a perf.data recording and shows exactly which instructions, and with debug information, which source lines, took the sampled time. The whole workflow is read-only: it never alters the recording or the binaries it inspects.

Allow about fifteen minutes for a recording that already exists. You need the perf command, a readable recording made by perf record, and the executable or shared objects present when it was captured. The examples use the installed linux-tools-common package, version 6.8.0-142.142. The local wrapper also warns that a kernel-specific perf binary for 6.8.0-139 is not installed, so this host may need the matching tools package before the commands can run.

1. Check the installed command and recording

Start with ordinary, unprivileged checks. None of these need sudo:

$ command -v perf
/usr/bin/perf
$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
linux-tools-common 6.8.0-142.142
$ test -r perf.data && echo 'recording is readable'
recording is readable

The default input name is perf.data, unless standard input is a FIFO. If the recording has another name, pass it explicitly with -i. Keep the executable and libraries used for the recording available: sample addresses only become useful once perf can match them to the right object files.

2. Open the standard annotation view

Run the basic command from the directory containing the recording:

$ perf annotate --stdio2

--stdio2 selects the non-interactive interface with TUI-style formatting, useful in a terminal capture or a redirected report. The TUI is also available with --tui, but it needs a terminal; when none is present, perf falls back to the stdio interface. For a script or a file, prefer --stdio2 and make colour explicit:

$ perf annotate --stdio2 --stdio-color=never > annotation.txt
$ test -s annotation.txt && echo 'annotation report written'
annotation report written

The report lists symbols found in the recording plus an annotated source or assembly view. Exact percentages, addresses and instruction lines depend on your workload and recording. A high percentage marks code where the sampled event spent more of its measured period; it is not automatic proof that the instruction is the only cause of a slowdown.

3. Select one symbol

A large recording can contain many functions. Select the symbol you want with -s or --symbol:

$ perf annotate --stdio2 --symbol=FUNCTION_NAME

Replace FUNCTION_NAME with the symbol as perf displays it. For C++ code, demangling is on by default; use --no-demangle when you need the original mangled names for an exact lookup. Combine with an input path when needed:

$ perf annotate --stdio2 --input=/path/to/perf.data --symbol=FUNCTION_NAME

If the symbol lives in one shared object, narrow the search with -d or --dsos. Multiple objects are comma-separated with no spaces:

$ perf annotate --stdio2 --dsos=libexample.so --symbol=FUNCTION_NAME

4. Choose source or assembly output

When the object has debug symbols, perf can place source beside assembly. Source interleaving is on by default; request it explicitly with --source, or drop it for the instruction view alone with --no-source:

$ perf annotate --stdio2 --symbol=FUNCTION_NAME --no-source

Use -l or --print-line to print matching source lines. This may be slow, so use it after narrowing to a symbol rather than turning it on across a very large recording:

$ perf annotate --stdio2 --symbol=FUNCTION_NAME --print-line

Long source paths are shortened by default. Add --full-paths for the complete path. If the source was compiled on a different filesystem layout, --prefix-strip and --prefix can remove leading path components and add a replacement prefix; test that mapping against one symbol before using it in an automated report.

No debug information does not make a recording useless. Perf can still show annotated assembly. The source view just needs suitable debug information in the object or in the symbol files you provide.

5. Point perf at symbols and kernel code

For a copied executable tree, use --symfs= to make perf look for symbol files relative to a directory:

$ perf annotate --stdio2 --symfs=/path/to/symbol-tree --symbol=FUNCTION_NAME

For a kernel recording, provide the matching uncompressed vmlinux with -k or --vmlinux. This is a path lookup, not a request to load a new kernel:

$ perf annotate --stdio2 --vmlinux=/path/to/vmlinux --symbol=KERNEL_SYMBOL

--modules loads module symbols, but the installed manual marks it for use only with -k and a live kernel. Treat that as a hard boundary, not a default troubleshooting switch: for a saved kernel recording, make the matching symbol files available instead. --ignore-vmlinux deliberately skips vmlinux files and is only useful when that omission is intended.

6. Make percentages and evidence easier to compare

--percent-limit=LIMIT hides functions below a given overhead percentage in stdio and stdio2. It filters functions, not individual lines inside a selected function. Add --show-total-period when the sum of periods matters, and -n or --show-nr-samples for a sample count per symbol:

$ perf annotate --stdio2 --percent-limit=1 --show-total-period --show-nr-samples

These are presentation choices. They do not improve the recording's quality or turn sampling into a complete execution trace. Compare like with like: keep the event, workload, filters and percentage type consistent when checking a change.

Use --percent-type to state how percentages are calculated. The installed command accepts global-period, local-period, global-hits and local-hits. The local or global part controls scope; period or hits controls the base. Record the choice alongside any conclusion so a later reader does not mistake two different bases for a real regression.

7. Diagnose missing or misleading annotation

Start unfiltered, then add one restriction at a time. Check the recording path with --input, confirm the executable and shared objects match the captured build, and check the selected symbol actually received samples. Use --skip-missing only when you deliberately want perf to omit symbols it cannot annotate; it shortens a report while hiding the reason a symbol disappeared.

For instruction-tracing data, --itrace controls decoding. Its default is all instruction-trace event types this installed manual documents, and --no-itrace disables decoding. Do not add it to an ordinary sampling recording merely because the report looks empty; first establish whether the recording contains instruction-trace data at all. For raw investigation, -D or --dump-raw-trace writes an ASCII dump, which can be much larger than an annotation report.

--verbose adds symbol addresses and other detail. --quiet suppresses warnings, messages and verbose output alike. Keep diagnostics on while investigating a failure; use quiet mode only when a caller has a deliberate policy for handling missing data.