Find CPU Hotspots Live with perf top

perf top gives you a live, refreshing view of which symbols are burning CPU right now, better than staring at load average. This guide gets you a live profile plus a small set of filters for turning that broad view into a useful diagnosis. Examples follow the installed perf-top(1) manual from linux-tools-common version 6.8.0-142.142. Allow about fifteen minutes for a first investigation.

You need a shell, a matching perf executable, and a workload that is currently doing something worth measuring. Most examples are ordinary commands. Reading system-wide counters or kernel symbols may require elevated privileges or a suitable perf_event_paranoid policy, so do not change that policy or grant capabilities just to make one experiment work without checking your system's security requirements first.

1. Check the installed tool before profiling

Start with read-only checks. They flag the command and any kernel-tools mismatch, which is a common source of wasted time:

$ command -v perf
$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
$ uname -r
$ perf --version

On this machine, linux-tools-common is 6.8.0-142.142, while the installed wrapper reports it cannot find a kernel-specific perf for kernel 6.8.0-139. If you see that warning, stop and install or otherwise provide the matching package through your normal operating-system process; this guide does not install packages.

2. Confirm the event names available on this host

Event names are hardware and kernel dependent. List the names instead of guessing them:

$ perf list

Look for a hardware CPU event such as cycles. The manual also accepts raw PMU events in the form rN, but raw encodings depend on the PMU format exposed under /sys/bus/event_source/devices/cpu/format/, so use a symbolic event first. If the list is empty or opening an event fails, treat that as an access or hardware-support problem, not a reason to invent an event name.

3. Start a short system-wide view

With no process selector, perf top uses system-wide collection by default. That is useful for finding the busiest code on a test host, but it can expose activity from other users and services:

$ sudo perf top -e cycles

The sudo is conditional: try the same command without it first on a non-sensitive machine. If the kernel denies access, read the error and ask an administrator about the existing perf policy.

The terminal opens an interactive display. You should see a changing table of symbols, shared objects and overhead percentages, with exact rows depending on the workload. Press q or Q to quit. This does not create a file and does not change the workload.

4. Narrow the profile to one process

System-wide output is often too noisy. Find the target process ID, then select it with -p:

$ pgrep -x YOUR_PROGRAM
$ sudo perf top -p YOUR_PID -e cycles

Replace YOUR_PROGRAM and YOUR_PID with values from your host. The process must still exist when perf top opens it. The manual also accepts a comma-separated list of process IDs. If the program creates short-lived workers, process-wide results may shift as those workers appear and disappear, so record the command, PID and time when comparing runs.

To inspect only selected CPUs instead, use a comma-separated list or range:

$ sudo perf top -C 0,1 -e cycles

Do not confuse -C, which selects CPUs, with -p, which selects processes. Both still refresh an interactive display.

5. Make the display easier to read

Refresh interval and row count are safe presentation controls. For example, refresh every two seconds and show 20 functions:

$ sudo perf top -e cycles -d 2 -E 20

• Press d to change the delay interactively.
• Press e to change the number of entries.
• Press f to set a hit-count display filter.
• Use --percent-limit to suppress entries below a percentage; hiding rows does not reduce the measured activity.

For a focused view, filter by symbols, shared objects or commands with --symbols, --dsos or --comms. Filtering changes how the overhead percentage is interpreted: the manual calls the two modes relative and absolute. Relative percentages recalculate from the filtered entries; absolute percentages keep their original share. Check this before comparing a filtered screen with an unfiltered one.

6. Add call graphs only when they answer a question

Use -g or --call-graph when the hot symbol alone is not enough and you need its callers:

$ sudo perf top -p YOUR_PID -e cycles -g

With callchains, the default Children view accumulates the cost of child functions into their parents, so its percentages can add to more than 100 percent. Self is the direct sampled cost at the symbol. Press s to annotate a symbol, and S to return to the full profile. Annotation needs symbols and, for kernel code, may need a suitable vmlinux path supplied with -k.

Call-graph collection and symbol resolution increase overhead and may fail under tighter permissions. Start without -g, identify a real hotspot, then add it for a short confirmation run. Use --no-children if you need the traditional self-overhead ordering.

7. Diagnose a result before changing anything

A high percentage is a lead, not a fix. Repeat the observation with the same workload, event, process selector and delay. A row labelled unknown, an absent kernel symbol or an event-open error can point to missing symbols, insufficient privileges, a mismatched perf binary or unsupported hardware.

Use verbose diagnostics when an event will not open:

$ sudo perf top -v -p YOUR_PID -e cycles