Join Guest and Host VM Traces with trace-cmd attach

trace-cmd attach merges a guest's trace.dat into the host recording of its virtual CPU activity. trace-cmd report then reads both files together as one interleaved timeline.

Allow 15 to 30 minutes if tracing is already familiar. Preparation is the part that needs care: the host and guest recordings must use the clocks this command expects, and the PID list must identify the host threads that represent the guest vCPUs.

1. Check the installed command

Run these as your ordinary user. Neither starts tracing nor changes kernel state:

$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)
$ trace-cmd attach -h
 trace-cmd attach [options] host_file guest_file vcpu_pid,...
          -s  offset,scale,fraction[,timestamp] conversion to sync guest timestamp

The installed help is brief; the manual also documents -c for the number of guest CPUs and the long form of the time-shift values. Keep the manual and the installed help side by side when checking a different release, option details can change between them.

2. Prepare recordings with compatible clocks

• The host file needs kvm_exit and kvm_entry events and must use the tsc2nsec clock.
• The guest file must use x86-tsc. The manual says guest attachment currently supports x86 only.

These are prerequisites, not repair options. If either recording used a different clock, make a new recording with the correct setup rather than guessing a conversion value. The local manual's example has the host recording kvm, sched, irq and timer events, while the guest records sched, irq and timer. Use whatever events help your investigation, but never omit the host KVM entry and exit events.

Record the guest while the host recording is active, stop both, then copy the guest file to the host under a new name so the original stays available:

$ scp root@GUEST_HOST:/path/to/trace.dat trace-guest.dat
$ ls -lh trace.dat trace-guest.dat

That scp may need access approved through your normal administration process. Don't make a trace file world-readable just to dodge a permission error. If the files are owned by root, copy them to a protected working directory or use a narrowly scoped ownership change approved for your host.

3. Find the host thread IDs for the guest

The final arguments are host process IDs, not guest ones. They identify the host threads representing the guest's vCPUs. First find the QEMU process for the guest:

$ ps -eo user,pid,cmd | grep '[q]emu-system'
libvirt-qemu  63170 /usr/bin/qemu-system-x86_64 ...

Replace 63170 with the QEMU PID on your host, then list its threads:

$ ls /proc/63170/task
1541591 63170 63198 63209 63211 63213

Use the IDs that represent all the guest vCPUs. You can include extra threads too, since trace-cmd attach searches the host file's KVM entry and exit events to match the supplied PIDs with vCPUs, but including unrelated IDs just makes the command harder to review. Start with the vCPU threads you actually identified.

4. Obtain the time-shift values

Use -s when guest timestamps need converting to the host timeline. Its value is comma-separated: offset,scale,fraction,timestamp. According to the manual, these values come from each guest vCPU directory under /sys/kernel/kvm/<pid>/vcpu/*.

The offset needs special attention: the value you pass to trace-cmd attach is the negative of the tsc-offset shown in that directory. The other fields map to tsc-scaling-ratio, tsc-scaling-ratio-frac-bits, and the timestamp the conversion starts at, normally zero. Reading these files may need elevated privileges depending on your system:

$ sudo cat /sys/kernel/kvm/63170-15/vcpu0/tsc-offset
-27950965013436847

For that example, the offset passed to -s is 27950965013436847, the sign flipped. Don't paste that number into a real run unless it came from your own VM and recording.

5. Attach without overwriting either input

$ df -h .
$ cp --preserve=all trace.dat trace-host.original.dat
$ cp --preserve=all trace-guest.dat trace-guest.original.dat

For a guest with eight vCPUs, substitute the real host file, guest file, offset and complete PID list. The -c value describes the guest, not the host:

$ trace-cmd attach -c 8 \
    -s 27950965013436847 \
    trace.dat trace-guest.dat \
    1541591 63170 63198 63209 63211 63213

A zero exit status means it worked. The command can produce little or no output, so check explicitly:

$ printf 'attach status: %s\n' "$?"
attach status: 0

$ cp --preserve=all trace-host.original.dat trace.dat
$ cp --preserve=all trace-guest.original.dat trace-guest.dat

6. Verify the joined view

Read both files together with trace-cmd report:

$ trace-cmd report -i trace.dat -i trace-guest.dat

Look for guest events interleaved with the host trace. A command that succeeds doesn't prove the clock conversion or PID selection was semantically correct: if events are missing, badly ordered or implausibly far apart, stop and recheck the host clock, guest clock, KVM events, vCPU count, thread IDs and the sign of the offset. Keep the original recordings while you investigate.

Don't use sudo for attaching or reporting unless file permissions genuinely require it. Elevated privileges won't fix an incompatible trace or a wrong PID list.