Join Guest and Host VM Traces with trace-cmd attach

You will combine a guest trace.dat with the host recording that contains its virtual CPU activity. The result is a pair of trace files that trace-cmd report can read together, so guest events appear interleaved with the host events that led to them.

Allow 15 to 30 minutes if tracing is already familiar to you. The preparation is the part that needs care: the host and guest recordings must use the clocks expected by this command, and the PID list must identify the host threads representing the guest vCPUs.

1. Check the installed command

Run these commands as your ordinary user. They do not start tracing or change kernel state:

$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)
$ trace-cmd attach -h
 trace-cmd attach [options] host_file guest_file vcpu_pid,...
          -s  offset,scale,fraction[,timestamp] conversion to sync guest timestamp

The installed help is brief. The manual also documents -c for the number of guest CPUs and the long form of the time-shift values. Keep the manual and the installed help together when checking a different release; option details can change.

2. Prepare recordings with compatible clocks

The host file must contain kvm_exit and kvm_entry events, and must use the tsc2nsec clock. The guest file must use x86-tsc. The manual says guest attachment currently supports x86 only.

These are prerequisites, not repair options. If either recording used another clock, make a new recording with the appropriate setup rather than guessing a conversion value. The local manual's example shows the host recording events from kvm, sched, irq and timer, while the guest records sched, irq and timer. Use only events useful to your investigation, but do not omit the host KVM entry and exit events.

Record the guest while the host recording is active, then stop both recordings. Copy the guest file to the host, using a new name so that the original remains available:

$ scp root@GUEST_HOST:/path/to/trace.dat trace-guest.dat
$ ls -lh trace.dat trace-guest.dat

The scp command may require access approved by your normal administration process. Do not make a trace file world-readable merely to avoid a permission error. If the files are owned by root, copy them to a protected working directory or use a narrowly scoped ownership change approved for your host.

3. Find the host thread IDs for the guest

The final arguments are host process IDs, not guest process IDs. They identify host threads representing the guest's vCPUs. First find the QEMU process for the guest:

$ ps -eo user,pid,cmd | grep '[q]emu-system'
libvirt-qemu  63170 /usr/bin/qemu-system-x86_64 ...

Replace 63170 with the QEMU PID on your host. List its threads:

$ ls /proc/63170/task
1541591 63170 63198 63209 63211 63213

Use the IDs that represent all guest vCPUs. You may include additional threads, because trace-cmd attach searches the host file's KVM entry and exit events to match the supplied PIDs with vCPUs. Including unrelated IDs makes the command harder to review, so start with the vCPU threads you actually identified.

4. Obtain the time-shift values

Use -s when the guest timestamps need conversion to the host timeline. Its value is a comma-separated offset,scale,fraction,timestamp. These values come from each guest vCPU directory under /sys/kernel/kvm/<pid>/vcpu/* according to the manual.

The offset needs special attention: the value passed to trace-cmd attach is the negative of the tsc-offset value shown in that directory. The other fields map to tsc-scaling-ratio, tsc-scaling-ratio-frac-bits and the timestamp at which the conversion starts. The timestamp is normally zero. Reading these files may require elevated privileges, depending on your system's permissions:

$ sudo cat /sys/kernel/kvm/63170-15/vcpu0/tsc-offset
-27950965013436847

For that example, the offset supplied to -s is 27950965013436847. Do not copy that number into a real run unless it came from your VM and recording.

If the conversion values change during a run, provide one -s option for each CPU and include the timestamp for each change. If fewer options are supplied than the guest has CPUs, the last option is reused for the remaining CPUs. One option is reused for every CPU when only one is supplied.

5. Attach without overwriting either input

$ df -h .
$ cp --preserve=all trace.dat trace-host.original.dat
$ cp --preserve=all trace-guest.dat trace-guest.original.dat

For a guest with eight vCPUs, substitute the real host file, guest file, offset and complete PID list. The -c value must describe the guest, not the host:

$ trace-cmd attach -c 8 \
    -s 27950965013436847 \
    trace.dat trace-guest.dat \
    1541591 63170 63198 63209 63211 63213

Successful completion is indicated by a zero exit status. The command may produce little or no output, so check it explicitly:

$ printf 'attach status: %s\n' "$?"
attach status: 0

If you need to try again with different values, restore the originals first:

$ cp --preserve=all trace-host.original.dat trace.dat
$ cp --preserve=all trace-guest.original.dat trace-guest.dat

6. Verify the joined view

Read both files together with trace-cmd report:

$ trace-cmd report -i trace.dat -i trace-guest.dat

Inspect the output for guest events interleaved with the host trace. A successful command does not prove that the clock conversion or PID selection is semantically correct. If events are absent, badly ordered or implausibly far apart, stop and recheck the host clock, guest clock, KVM events, vCPU count, thread IDs and the sign of the offset. Preserve the original recordings while investigating.

Do not use sudo for attachment or reporting unless file permissions genuinely require it. Elevated privileges do not fix an incompatible trace or an incorrect PID list.