Read perf.data and find hot instructions with perf annotate

You will finish with a repeatable way to inspect a perf.data recording, identify where samples landed inside a symbol, and get source lines when debug information is available. The workflow is read-only: perf annotate reads the recording and related binaries; it does not alter either one.

Allow about fifteen minutes for a recording that already exists. You need the perf command, a readable recording made by perf record, and the executable or shared objects that were present when it was captured. The examples use the installed linux-tools-common package, version 6.8.0-142.142. The local wrapper also warns that a kernel-specific perf binary for 6.8.0-139 is not installed, so this host may need the matching tools package before the commands can run.

1. Check the installed command and recording

Start with ordinary, unprivileged checks. They do not need sudo:

$ command -v perf
/usr/bin/perf
$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
linux-tools-common 6.8.0-142.142
$ test -r perf.data && echo 'recording is readable'
recording is readable

The default input name is perf.data, unless standard input is a FIFO. If the recording has another name, pass it explicitly with -i. Keep the executable and libraries used for the recording available: the sample addresses only become useful when perf can match them to the right object files.

Checkpoint: if perf --version reports a warning about a missing kernel-specific binary, install or select the matching package through your normal system administration process. Do not try to work around a tool and kernel mismatch by guessing paths or running the analysis as root.

2. Open the standard annotation view

Run the basic command from the directory containing the recording:

$ perf annotate --stdio2

--stdio2 selects the non-interactive interface with TUI-style formatting, which is useful in a terminal capture or a redirected report. The TUI is also available with --tui, but it needs a terminal; when no terminal is present, the manual says perf falls back to the stdio interface. For a script or a file, prefer --stdio2 and make colour explicit:

$ perf annotate --stdio2 --stdio-color=never > annotation.txt
$ test -s annotation.txt && echo 'annotation report written'
annotation report written

The report lists symbols found in the recording and an annotated source or assembly view. The exact percentages, addresses and instruction lines depend on your workload and recording. A high percentage marks code where the sampled event spent more of its measured period; it is not automatically proof that the instruction is the only cause of the slowdown.

3. Select one symbol

A large recording can contain many functions. Select the symbol you want with -s or --symbol:

$ perf annotate --stdio2 --symbol=FUNCTION_NAME

Replace FUNCTION_NAME with the symbol as perf displays it. For C++ code, demangling is enabled by default; use --no-demangle when you need the original mangled names for an exact lookup. The option can be combined with an input path:

$ perf annotate --stdio2 --input=/path/to/perf.data --symbol=FUNCTION_NAME

If the symbol is in one shared object, narrow the search with -d or --dsos. Multiple objects are comma-separated with no spaces:

$ perf annotate --stdio2 --dsos=libexample.so --symbol=FUNCTION_NAME

If the command says that no symbol can be annotated, first remove the filters and inspect the general report. A spelling difference, a missing object, or a recording with no samples for that function can all look similar when several filters are applied at once.

4. Choose source or assembly output

When the object has debug symbols, perf can place source beside assembly. Source interleaving is enabled by default and can be requested explicitly with --source. If you want the instruction view without source, use --no-source:

$ perf annotate --stdio2 --symbol=FUNCTION_NAME --no-source

Use -l or --print-line to print matching source lines. The manual warns that this may be slow. Use it after narrowing to a symbol rather than enabling it across a very large recording:

$ perf annotate --stdio2 --symbol=FUNCTION_NAME --print-line

Long source paths can be shortened in the normal display. Add --full-paths when you need the complete path. If the source was compiled in a different filesystem layout, --prefix-strip and --prefix can remove leading path components and add a replacement prefix. Test that mapping against a single symbol before using it in an automated report.

No debug information does not make the recording useless. Perf can still show annotated assembly. The source view needs suitable debug information in the object or in the symbol files you provide.

5. Point perf at symbols and kernel code

For a copied executable tree, use --symfs= to make perf look for symbol files relative to a directory:

$ perf annotate --stdio2 --symfs=/path/to/symbol-tree --symbol=FUNCTION_NAME

For a kernel recording, provide the matching uncompressed vmlinux with -k or --vmlinux. This is a path lookup, not a request to load a new kernel:

$ perf annotate --stdio2 --vmlinux=/path/to/vmlinux --symbol=KERNEL_SYMBOL

--modules loads module symbols, but the installed manual marks it for use only with -k and a live kernel. Treat that as a boundary, not as a default troubleshooting switch. If you are analysing a saved kernel recording, make the matching symbol files available instead. --ignore-vmlinux deliberately skips vmlinux files and is useful only when that omission is intended.

6. Make percentages and evidence easier to compare

For a compact review, --percent-limit=LIMIT hides functions below the given overhead percentage in stdio and stdio2. It filters functions, not individual lines inside a selected function. Add --show-total-period when the sum of periods matters, and -n or --show-nr-samples when you want the sample count for each symbol.

$ perf annotate --stdio2 --percent-limit=1 --show-total-period --show-nr-samples

These are presentation choices. They do not increase the quality of the recording or turn sampling into a complete execution trace. Compare like with like: keep the event, workload, filters and percentage type consistent when checking a change.

Use --percent-type when you need to state how percentages are calculated. The installed command accepts global-period, local-period, global-hits and local-hits. The local or global part controls scope; period or hits controls the base. Record the choice alongside any conclusion so a later reader does not mistake two different bases for a performance regression.

7. Diagnose missing or misleading annotation

Start with an unfiltered command and then add one restriction at a time. Check the recording path with --input, confirm that the executable and shared objects match the captured build, and check that the selected symbol actually received samples. Use --skip-missing only when you intentionally want perf to omit symbols it cannot annotate; it can make a report shorter while hiding the reason a symbol disappeared.

For instruction-tracing data, --itrace controls decoding. Its default is all instruction-trace event types documented by this installed manual, and --no-itrace disables decoding. Do not add it to an ordinary sampling recording merely because the report is empty. First establish whether the recording contains instruction-trace data. If you need raw trace data for investigation, -D or --dump-raw-trace writes an ASCII dump, which can be much larger than an annotation report.

--verbose can add symbol addresses and other detail. --quiet suppresses warnings and messages, and also suppresses verbose output. Keep diagnostics enabled while investigating a failure; use quiet mode only when a caller has a deliberate policy for handling missing data.