Profile Memory Loads and Stores with perf mem

perf mem samples where a command's loads and stores actually land, which is the fastest way to tell a cache-miss problem from a plain CPU-bound one. Recording is the only state-changing part; reading the result is entirely read-only.

Allow about fifteen minutes for a first run. You need the perf program, kernel performance tools that match the running kernel, and a command that does enough memory work to produce samples. The installed package here is linux-tools-common 6.8.0-142.142, but the executable is not available for the running 6.8.0-139 kernel. Check your own installation before starting:

$ command -v perf
$ perf --version

If the command is missing, install the kernel-specific tools through your normal distribution process. A package containing the manpage is not proof the matching executable is there.

1. Check the available memory events

Ask perf mem record what event selectors the local build and processor actually expose:

$ perf mem record -e list

The output is entirely hardware-dependent. Pick a selector from that list if you need something specific; otherwise, skip -e and let perf mem pick its own default. An event mentioned in the manual or working on someone else's machine is not automatically available on this processor.

2. Record a short, representative command

Run the workload after record. This example uses a harmless placeholder; swap in the real operation you want to look at:

$ perf mem record -- /usr/bin/sleep 5
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote perf.data ]

Exact status lines vary by version and workload; what matters is a new perf.data in the current directory. The -- keeps perf's own options from swallowing an argument meant for the workload.

Try it unprivileged first. Recording can need elevated privileges when the kernel's performance-event policy restricts ordinary users, so only reach for sudo after a permissions or counter-access failure, and only for the same read-only profile:

$ sudo perf mem record -- /path/to/workload --safe-test-option

3. Report the recorded samples

With perf.data still in the current directory:

$ perf mem report

This is the normal perf reporting view with memory-access fields added, and it includes both loads and stores by default. Filter to one class with --type load or --type store (short form -t):

$ perf mem report --type load
$ perf mem report --type store

The command's own default is load,store, so a report showing both kinds is expected, not evidence the filter was ignored.

If the data file lives elsewhere, point the report at it directly:

$ perf mem report --input=/path/to/perf.data

Use the exact file the recording produced. Do not reach for --force just to silence an ownership check; that option disables ownership validation entirely, so only use it once you understand why the file belongs to a different user and accepting that is actually safe.

4. Inspect raw samples when a script needs them

For a parseable, one-sample-per-line dump:

$ perf mem report --dump-raw-samples
$ perf mem report --dump-raw-samples --field-separator='|'

The default separator is a space. Pick a separator that cannot occur in the fields you expect, and treat the output as version- and hardware-dependent rather than a stable interchange format. Capture a sample of it before you build automation around specific columns.

5. Limit CPUs only when you have a reason

By default, perf monitors every CPU. Restrict it with --cpu, either a comma-separated list or a range with no spaces:

$ perf mem record --cpu=0,1 -- /path/to/workload
$ perf mem record --cpu=0-2 -- /path/to/workload

CPU numbering and availability are host-specific, so check first:

$ nproc
$ lscpu

Expect an error if you name a CPU that is offline or outside the host's range. There is no undo needed either way: the selection applies only to that recording process, and dropping --cpu next time returns you to all CPUs.

6. Interpret the result without overclaiming

• On Intel systems, the reported memory latency is use-latency. It bundles pipeline queueing delay with memory-subsystem latency, so do not present it as a pure DRAM or cache access time.
• On Arm64, perf mem uses Statistical Profiling Extension sampling, which needs hardware and kernel support and, being statistical, will miss plenty of real memory operations. A short or empty report can mean the sampling hit its limits, not that nothing happened.

Keep the generated perf.data until you have checked the report. If it holds sensitive addresses or names, protect it like any other diagnostic output. Once you are done with it, remove that specific file through your normal retention process; do not delete a whole directory of unrelated profiling data by accident.