Read Kernel Profiling Data with readprofile

Something feels slow, and readprofile turns a kernel's raw profiling buffer into a list of which functions actually burned the CPU. "Just add more CPU" is not an answer anyone should accept without proof. Allow about ten minutes. The examples here only read data, up to the explicit reset example; nothing enables profiling or changes a service.

Two versions matter before you type anything. The installed executable is util-linux 2.42.4, but the local readprofile(8) page identifies itself as the older util-linux 2.39.3, and the two disagree on defaults: the current binary reports /boot/System.map and the running kernel's versioned map, while the older page describes /usr/src/linux/System.map. Trust the executable's own output on the host where you run it, not the page.

1. Check the executable and kernel interface

Confirm which program your shell will run and record its version:

$ command -v readprofile
/home/linuxbrew/.linuxbrew/sbin/readprofile
$ readprofile --version
readprofile from util-linux 2.42.4

The command reads /proc/profile by default. Check that file before spending time on map-file troubleshooting:

$ test -r /proc/profile && echo 'profiling buffer is readable' || echo 'no readable profiling buffer'
no readable profiling buffer

On a system without /proc/profile, ordinary reads cannot work. The installed command also reports this directly:

$ readprofile --info
readprofile: /proc/profile: No such file or directory

Checkpoint: if you see the second result, stop here unless you have a saved profiling buffer from a kernel that provided this interface. Do not try sudo as a repair: root access does not create a missing proc file.

2. Understand the data and map relationship

The profiling buffer is a binary snapshot of kernel profiling data: on its own, it means nothing. readprofile combines it with a kernel symbol map so counts can be pinned to real C function names. Use the map for the kernel that produced the buffer, not just the newest map you happen to have lying around: a mismatch quietly produces missing or misleading symbols instead of an error.

List candidate maps without changing anything:

$ ls -l /boot/System.map* /usr/src/linux/System.map 2>/dev/null
-rw------- 1 root root 9135527 ... /boot/System.map-6.8.0-142-generic

The exact file list, permissions and sizes vary by host. The current executable's help reports /boot/System.map and a versioned /boot/System.map-6.8.0-139-generic default here. Pass an explicit map when you know which kernel produced the data:

$ readprofile --profile /path/to/profile.freeze --mapfile /boot/System.map-6.8.0-142-generic

Replace both placeholders with real paths. The manpage says a map ending in .gz is decompressed on the fly. A saved buffer is useful when you need to inspect it later, but it is only useful with the correct map.

3. Read ordinary function counts

When /proc/profile exists, the bare command is enough. Three columns come back: clock ticks, the kernel function name and a normalised load value:

$ readprofile
       12 schedule                   0.0042
        8 do_sys_open                 0.0011

Those names and counts are illustrative, not promised output: your kernel, workload and map decide what actually appears. A blank result usually just means nothing had ticks, since zero-count procedures are omitted by default.

To include every symbol in the map, use --all:

$ readprofile --all | less

Use --verbose when addresses matter. The output then includes the RAM address, function name, tick count and normalised load:

$ readprofile --verbose | less

For a quick ranking, sort numerically by the first column:

$ readprofile | sort -nr | less

Do not treat that ranking as a complete performance profile. The manpage warns that profiling is disabled while interrupts are inhibited, and ticks get dumped in a batch when interrupts are re-enabled, so the data can be biased by kernel behaviour in ways that have nothing to do with your workload.

4. Inspect bins or counters when the columns are too coarse

Both exist for investigating the shape of the collected data, not for producing the normal function summary:

$ readprofile --histbin | less
$ readprofile --counters | less

Keep the map and profiling buffer fixed while you compare two readings. Otherwise a changed input can masquerade as a changed kernel.

5. Freeze a buffer for later inspection

The live buffer keeps moving while you read it, which is annoying if you want a stable snapshot to compare against later. If /proc/profile is present and readable, copy it to a root-owned or otherwise protected working location, then pass that copy with --profile:

$ cp -- /proc/profile /tmp/profile.freeze
$ readprofile --profile /tmp/profile.freeze --mapfile /boot/System.map-6.8.0-142-generic

Reading the proc file and writing under /tmp normally need no elevated privileges, but the copy still contains kernel activity data. Use a directory with suitable permissions if that information is sensitive on your system. Remove the temporary copy once you are done:

$ rm -- /tmp/profile.freeze

Warning: that removal is irreversible. If you need an audit record, move the file to an approved evidence location instead of deleting it.

6. Treat reset and multiplier as state-changing operations

Everything up to here has been read-only. These two flags are not.

Warning: --reset clears the profiling counters. It requires root and destroys the current counts, so do not include it in a diagnostic script or paste it into a production shell without recording why:

$ sudo readprofile --reset

There is no undo for cleared counters. The only recovery is to begin a new collection period.

Warning: --multiplier changes the profiling interrupt frequency where the architecture supports it, resets the profiling buffer, and requires superuser privileges:

$ sudo readprofile --multiplier 20

Linux 2.6.16 removed multiplier support for most systems, according to the local page. The installed help still exposes the option, but availability and effect depend on the kernel and architecture: do not reach for it as a generic way to get higher quality data.

7. Know what readprofile cannot enable

No flag in this tool turns profiling on. It has to be enabled by the kernel itself. The local manpage says this needs a reboot and a kernel command-line setting such as profile=2, where the number is the two-exponent used as the profiling step. That is a boot configuration change, not a readprofile option: plan it separately, keep a rollback entry for the previous kernel command line, and expect the profiling buffer to stay unavailable until the machine boots with that support.

Do not confuse --info with enabling profiling. It only reports the sampling step. If no buffer exists, it fails with the same proc-file error shown earlier.

Done means