Profile KVM Guests with perf kvm Without Losing the Host Context

You will record a short KVM workload, select whether the data describes the guest or the host, and inspect the resulting report. The examples follow the installed perf-kvm(1) manual from the Debian linux-tools-common package, version 6.8.0-142.142. Allow 15 to 30 minutes, plus time to identify a useful workload.

You need a KVM host, the matching kernel-specific perf binary, and enough privilege for the kernel's performance monitoring policy. This guide records performance data; it does not start or stop virtual machines. Do not run a command against a production guest until you have checked its workload and storage impact.

1. Check that the matching perf is installed

perf kvm is a subcommand of perf, not normally a separate executable. Check both the command path and the running kernel:

$ command -v perf
/usr/bin/perf
$ uname -r
6.8.0-139-generic
$ perf --version
perf version 6.8.0

The exact version and path vary. On this machine, /usr/bin/perf is supplied by linux-tools-common version 6.8.0-142.142, but it reports that the kernel-specific tools for 6.8.0-139 are missing. Install the matching tools through your normal system-management process before recording. Do not treat the wrapper's warning as a successful performance test.

Checkpoint: this must work without the "perf not found for kernel" warning before continuing:

$ perf kvm --help

2. Choose guest or host data deliberately

The top-level choice changes what the recording is intended to describe. --guest collects the guest side, while --host collects the host side. The manual's default for perf kvm record is guest collection when neither option is supplied.

There is a second detail that catches people out: the default output name depends on those selections:

• No selector, or --guest: perf.data.guest.
• --host: perf.data.kvm.
• --host --no-guest: perf.data.host.
• --host --guest: perf.data.kvm.

If a script or note refers to a particular file, make the selection and output path explicit rather than relying on a remembered default. --no-guest is accepted by the recording command even though the short option summary mainly describes the positive selectors.

3. Record a bounded workload

Use record followed by the command that represents the work you want to measure. Give the output an explicit name so an older capture cannot be mistaken for the new one:

$ perf kvm --guest record --output=/tmp/kvm-guest.data -- /path/to/workload --test-case=baseline

The command after record is the workload. Replace both placeholders with a real command and its arguments; do not copy the example literally. The recording covers the interval from the workload's start until it exits. The output path is a normal file path, so choose a filesystem with sufficient free space.

For a host profile, change the selector and keep the filename clear:

$ perf kvm --host --no-guest record --output=/tmp/kvm-host.data -- /path/to/workload --test-case=baseline

This writes performance data and may require elevated privileges depending on /proc/sys/kernel/perf_event_paranoid, the selected events and the account running the command. Prefer an unprivileged run first. If policy rejects it, inspect the error and use the least privilege your local policy permits. Do not weaken a production host's monitoring policy just to make one capture run.

4. Report the capture you just made

Pass the same file to report:

$ perf kvm --guest report --input=/tmp/kvm-guest.data

The report reads the recorded samples and presents the performance profile. If you omit --input, do not assume it will select the file you intended: use an explicit path when several captures exist. The --output option is for recording; for a report, redirect normal output with the shell if you need a text file:

$ perf kvm --guest report --input=/tmp/kvm-guest.data > /tmp/kvm-guest-report.txt

Checkpoint: confirm that the data file exists and is non-empty before diagnosing a report failure:

$ test -s /tmp/kvm-guest.data && echo 'capture is non-empty'

5. Use KVM statistics for exits and device activity

Use the separate stat family when the question is about KVM events rather than a general profile. The installed manual documents vmexit, plus mmio and ioport on x86:

$ perf kvm stat record -- /path/to/workload --test-case=baseline
$ perf kvm stat report

stat record records events between the workload's start and end. The report includes handled samples and timing fields such as minimum, maximum and mean time. The default event is vmexit; select another documented event explicitly when that is the question:

$ perf kvm stat report --event=vmexit --vcpu=0

The default VCPU selection is all VCPUs. A VCPU filter is useful when one virtual CPU is suspicious, but it can hide work happening elsewhere. The live form is available when you need changing statistics rather than a saved capture:

$ perf kvm stat live --event=vmexit --display=5

6. Supply guest symbols when names are missing

Guest profiles are much easier to interpret when perf can find the guest kernel and module symbols. The manual supports either a guest vmlinux, or copies of the guest's /proc/kallsyms and /proc/modules. These are inputs to analysis, not files that perf creates for you:

$ perf kvm --guest \
    --guestvmlinux=/srv/guest-symbols/vmlinux \
    report --input=/tmp/kvm-guest.data

For a mounted guest root, add --guestmount=/srv/guestmount. The manual describes mounting guest roots below that directory, commonly with SSHFS. Treat the mount as sensitive: it exposes guest filesystem content to the host account. Unmount it after analysis using the same mount tool, and do not use a writable guest mount when a read-only arrangement is sufficient.

buildid-list can show build IDs from a capture. For guest build IDs, the recording must have used --guestmount:

$ perf kvm --guest buildid-list --input=/tmp/kvm-guest.data

7. Recover from common mistakes

If the command says the matching perf binary is unavailable, install the package for the running kernel and repeat step 1. If recording fails with a permissions error, check the local perf policy and whether the chosen account may monitor the required events. If the report shows unresolved symbols, repeat the analysis with the correct guest symbol inputs; changing the report command cannot invent missing symbols.

If you wrote the capture to the wrong path, the original VM workload is unaffected, but the capture may contain sensitive execution details. Restrict its permissions while it is retained and remove it through your normal data-retention process when no longer needed. Removing a capture is irreversible, so verify that any report or comparison has been saved first.