Measure Kernel Memory Allocation with perf kmem

You will finish with a repeatable way to record kernel memory allocation events and inspect slab or page allocator statistics with perf kmem. The examples use the installed perf-kmem(1) manual from linux-tools-common version 6.8.0-142.142. Allow 15 to 30 minutes, plus the time needed to reproduce the workload you want to measure.

This guide measures a workload. It does not tune the kernel, change allocator settings or delete a data file. Recording kernel tracepoints can require elevated privileges, so start with the checks below as an ordinary user and use sudo only when the recorder reports that access is insufficient.

1. Check the installed perf tool

First establish which executable your shell will run and which package version is installed:

$ command -v perf
/usr/bin/perf
$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
linux-tools-common 6.8.0-142.142

The package version is not necessarily the running kernel version. This machine runs kernel 6.8.0-139, while linux-tools-common is 6.8.0-142.142. On this host the wrapper reports that matching kernel-specific perf tools are missing, so a real recording cannot be verified here. Install the normal matching tools package through your operating system's package process before relying on the later examples.

Checkpoint: a usable installation should let perf kmem reach its subcommand parser instead of stopping with a kernel-tool mismatch warning. Do not confuse that packaging failure with a workload that produced no allocation events.

2. Record one bounded workload

Choose a short, representative command. The recorder writes the perf data file while the command runs, so a five-second sleep is a harmless smoke test but not a useful workload by itself:

$ sudo perf kmem record -o /tmp/perf-kmem.data -- /usr/bin/sleep 5

record is the first variant of perf kmem: it captures kmem events from an arbitrary command. The options after record are passed to perf record, which is why -o selects the output file. The explicit path keeps this test separate from any existing default perf.data in the current directory.

This command may need root because kernel tracepoints and perf events are restricted by the host's security policy. If the unprivileged command works, keep it unprivileged. If it fails, read the error before adding sudo; a missing tracepoint or mismatched tool is not fixed by more privilege.

Checkpoint: confirm that recording completed and that the file exists:

$ test -s /tmp/perf-kmem.data && echo 'recording file is non-empty'
recording file is non-empty
$ ls -lh /tmp/perf-kmem.data
-rw------- 1 root root  ... /tmp/perf-kmem.data

The exact size and ownership vary. Because sudo created the file, you may need elevated privilege to inspect or remove it. Do not overwrite an existing capture accidentally: choose a new output path, or move a valuable capture aside before recording.

3. Read the default allocation statistics

Use stat to analyse the capture. Tell it which file to read with -i:

$ sudo perf kmem -i /tmp/perf-kmem.data stat

The default input for stat is perf.data, unless standard input is a FIFO. Making the input explicit prevents an easy mistake: reading an older capture in the working directory and attributing its numbers to the latest test.

The default sort keys differ by allocator. Slab output uses frag,hit,bytes; page output uses bytes,hit. The report is therefore a summary of the events in the file, not a live view of current memory use. A successful command should print a report rather than alter the capture.

Checkpoint: if you get a missing-file error, check the path and permissions without changing anything:

$ sudo test -r /tmp/perf-kmem.data && echo readable
readable

4. Separate slab and page analysis

Select the allocator before stat. For slab allocations, show the busiest callsites first:

$ sudo perf kmem --slab --sort=bytes,hit stat -i /tmp/perf-kmem.data

For page allocator events, sort by allocated bytes and then hit count:

$ sudo perf kmem --page --sort=bytes,hit stat -i /tmp/perf-kmem.data

The manual lists different sort keys for the two modes. Slab accepts ptr, callsite, bytes, hit, pingpong and frag. Page accepts page, callsite, bytes, hit, order, migtype and gfp. Put a mode option such as --slab or --page before --sort, because the sort vocabulary depends on that choice.

To focus on allocation callsites, request the per-callsite view:

$ sudo perf kmem --caller stat -i /tmp/perf-kmem.data

Use --alloc for per-allocation statistics. Limit a large report with --line=20, and add --verbose when symbol addresses or other extra detail are needed. These are analysis choices; they do not change the recorded data.

5. Measure live pages or a time window

Page analysis normally reports total allocation statistics. Add --live when you need pages still allocated at the end of the captured activity:

$ sudo perf kmem --page --live stat -i /tmp/perf-kmem.data

--live only works with page analysis. It is not a general switch for making slab output current.

You can restrict analysis to a time interval using seconds and microseconds:

$ sudo perf kmem --page --time=12.000000,18.500000 stat -i /tmp/perf-kmem.data

Use an empty start or stop when the interval should run to the beginning or end of the file. For example, --time=,18.500000 starts at the beginning, while --time=12.000000, runs to the end. These times refer to the capture's timestamps, not wall-clock dates.

6. Diagnose results without changing state

An empty or surprisingly small report can mean that the workload did not exercise the selected allocator, the capture window was too short, or the kernel and perf build do not expose the expected events. Repeat the workload with a longer bounded interval and inspect both --slab and --page rather than changing kernel settings blindly.

If recording fails with permission errors, check the host's perf restrictions and security policy with the administrator. If it fails because the event is unavailable, install the perf package matching the running kernel. The local wrapper's warning names the expected packages, such as linux-tools-6.8.0-139-generic and linux-cloud-tools-6.8.0-139-generic; package availability depends on the distribution repositories. Do not claim that a report is valid merely because a file was created.

Captures can contain kernel addresses, process names and workload metadata. Treat perf-kmem.data as diagnostic data: restrict its permissions, avoid sending it to third parties without review, and remove it only after you have finished with the evidence. If the temporary capture is no longer needed, remove that exact file:

$ sudo rm -- /tmp/perf-kmem.data

This removal is irreversible. There is no undo command; rerun the recording step to create a new capture.