Compare perf.data Profiles with perf diff

You will compare two or more perf.data recordings, read the result in the right direction, and narrow it to the code that matters. Allow about twenty minutes, plus the time needed to record two comparable runs. The examples follow the installed perf manual dated 1 September 2026; the installed package is linux-tools-common 6.8.0-142.142.

This guide assumes that the perf executable matching your running kernel is installed and that you already know how to make two recordings of the same workload. Recording is not covered here. Do not compare unrelated workloads and then treat the percentage columns as a benchmark score.

1. Check that the matching perf tool is available

First check the executable and package. These are ordinary, read-only commands:

$ command -v perf
/usr/bin/perf
$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
linux-tools-common 6.8.0-142.142
$ perf --version
perf version 6.8.0

The version lines above show the intended shape, not a promise that your distribution will print exactly the same text. The important check is that perf can run for your kernel. On this machine, invoking perf diff --help reports that the tool for kernel 6.8.0-139 is missing and suggests installing the matching linux-tools package. Stop there if you get the same warning. Installing packages is a system-management action, so use your normal administrator-approved process rather than guessing a package name.

2. Put the baseline first

The simplest comparison uses two files:

$ perf diff /path/to/baseline.perf.data /path/to/candidate.perf.data

The first argument is the baseline. The second is the other run. If you pass more files, the first remains the baseline and each later file is compared with it. The manual describes the operation in terms of matching samples, symbols and events, so a row that exists only in one recording can have an empty side of the comparison.

If you use the conventional names and supply no arguments, perf diff reads perf.data.old as the baseline and perf.data as the other file:

$ perf diff

That default is easy to trip over after a new recording. Make the file names explicit in scripts and notes, or inspect them before running the comparison:

$ ls -lh /path/to/baseline.perf.data /path/to/candidate.perf.data

3. Read the default result without reversing it

The default computation is delta-abs. In practical terms, the reported delta is based on the candidate's percentage minus the baseline's percentage, and the rows are sorted by the absolute size of that result. A positive change means the candidate has a larger share for that matching entry; a negative change means its share is smaller. Always repeat the file order when recording a finding.

To see the raw counts as well as the diff, add verbose output:

$ perf diff --verbose /path/to/baseline.perf.data /path/to/candidate.perf.data

The manual calls this option useful for showing raw counts in addition to the diff. It does not make two recordings statistically comparable. If sampling was short, the workload varied, or the event changed, a tidy-looking percentage may still be noise.

For a plain delta rather than sorting by absolute value, select it explicitly:

$ perf diff --compute delta /path/to/baseline.perf.data /path/to/candidate.perf.data

For a relative comparison, use the ratio method:

$ perf diff --compute ratio /path/to/baseline.perf.data /path/to/candidate.perf.data

A ratio is based on period values, while a delta uses period percentages. Do not call a ratio a percentage improvement unless you have checked what the displayed column represents.

4. Limit the comparison to useful code

Large profiles are easier to inspect when you restrict them to one executable or shared object. The --dsos option accepts a comma-separated list:

$ perf diff --dsos=/path/to/app,/path/to/libexample.so \
    /path/to/baseline.perf.data /path/to/candidate.perf.data

You can filter by command name with --comms, or by symbol with --symbols. Sort keys include comm, dso, symbol, cpu, pid, tid, parent and srcline:

$ perf diff --comms=YOUR_PROGRAM --sort=srcline,symbol \
    /path/to/baseline.perf.data /path/to/candidate.perf.data

Filtering changes the denominator used for displayed percentages. If you want the percentages to retain their original pre-filter meaning, add --percentage=absolute:

$ perf diff --percentage=absolute --dsos=/path/to/app \
    /path/to/baseline.perf.data /path/to/candidate.perf.data

The alternative, --percentage=relative, makes the shown entries sum to 100 percent after filtering. That can be convenient for examining a subsystem, but it can also make a small slice look more significant than it is in the complete profile.

5. Compare only a time window

Use --time when the recordings include warm-up or a known interval you do not want to mix with the steady-state work. Percent ranges are useful when the two runs have different absolute timestamps:

$ perf diff --time 0%-10% \
    /path/to/baseline.perf.data /path/to/candidate.perf.data

10%/2 selects the second ten-percent slice. For timestamp ranges, the manual accepts seconds and nanoseconds, with a colon separating ranges for different data files:

$ perf diff --time '1234.567,1234.789:1235.100,1235.300' \
    /path/to/baseline.perf.data /path/to/candidate.perf.data

Quote a time expression containing spaces or shell punctuation. Use perf script -i FILE to find timestamps in a recording before constructing a timestamp-based window.

6. Diagnose an unhelpful result

If the output is empty, first check that both files contain the event and samples you expected. The differential profile is shown only for events matching both files. If symbols are missing, investigate the recording's symbol files before reaching for --force. That option disables ownership validation; it does not repair missing symbols and removes a useful safety check.

For live-kernel module symbols, the manual provides --kallsyms=FILE and --modules, but warns that --modules should be used only with --kallsyms and a live kernel. This is a boundary, not a suggestion to add both options to every comparison.

There is no state to undo: perf diff reads the files and writes its report to standard output. It does not alter either recording. Redirect output only after choosing a new destination, because shell redirection can truncate an existing report:

$ perf diff --verbose /path/to/baseline.perf.data /path/to/candidate.perf.data \
    > perf-diff-candidate.txt
$ test -s perf-diff-candidate.txt && echo "report written"