Capture and Decode a Small Intel PT Trace with perf

You will finish with a small Intel Processor Trace (Intel PT) recording, a decoded report, and a short diagnostic path for the failures that matter. The examples target the perf-intel-pt documentation shipped by linux-tools-common 6.8.0-142.142. Allow 15 to 30 minutes. You need an Intel processor with Intel PT support, the matching perf tools, and enough disk space for the trace.

This workflow starts with one short userspace command. Intel PT records compressed execution packets, not a ready-made text log. perf report, perf script and perf inject decode those packets after recording. Start small because tracing can produce hundreds of megabytes per second per core and decoding can take much longer than capture.

Checkpoint 1: confirm the local tools and hardware

Check the package and command before spending time on a recording:

$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
$ perf --version
$ test -r /sys/bus/event_source/devices/intel_pt/type && cat /sys/bus/event_source/devices/intel_pt/type

The first line should identify the installed package version. The last command prints the Intel PT PMU type when the kernel exposes it. The manpage describes Intel PT support from Broadwell-era Intel Core processors onwards, but the actual availability still depends on the processor, kernel and perf build. If the PMU path is absent, stop here and investigate the host rather than trying random event names.

Checkpoint: verify that perf --version belongs to the tools you intend to use. On this machine, invoking perf also warns when tools for the running kernel are not installed. A warning like that is a reason to install the matching package through your normal system-management process, not to ignore version mismatches in a production investigation.

Checkpoint 2: record one short userspace trace

Run a harmless, bounded command as your normal user:

$ rm -f perf.data
$ perf record -e intel_pt//u -- /bin/echo intel-pt-ok

The event name is intel_pt//. The u modifier limits collection to userspace. The command writes the recording to perf.data and should print intel-pt-ok. The rm removes an old local recording, so do not use it if that file contains evidence you still need. Copy it elsewhere first if necessary.

There is no persistent service or policy change to undo. To discard this new recording after inspection, remove only the known file:

$ rm -f perf.data

For a real workload, replace /bin/echo intel-pt-ok with one bounded command. Avoid starting with a long-running service or system-wide tracing. That is the fastest route to a file that is expensive to decode and difficult to interpret.

Checkpoint 3: produce the first report

Decode the recording with the ordinary report view:

$ perf report

The report is generated from decoded samples. The exact rows depend on the command, available symbols and the perf build, so do not treat a particular percentage or address as a stable fixture. A successful report proves that perf could open and decode enough of the trace; it does not prove that every instruction was captured.

If you want the lower-level stream, use perf script:

$ perf script --itrace=iybxwpe

These itrace letters request instruction, cycle, transaction, branch, ptwrite and power-event related samples as documented by the installed manual. Add flags to the displayed fields when branch direction matters:

$ perf script --itrace=iybxwpe -F+flags

For a compact control-flow view, try call tracing before dumping every instruction:

$ perf script --call-trace

Instruction tracing with --insn-trace can be much slower and may require the XED tool when used with --xed. Narrow a long investigation with a time range and, where needed, -C CPU after you have identified the interesting interval.

Checkpoint 4: add timing only when you need it

The default Intel PT configuration includes TSC timing and disables return compression. You can request cycle packets with a config term when the PMU supports them:

$ perf record -e intel_pt/cyc=1/u -- /bin/echo intel-pt-cycles
$ perf script --itrace=be -F+ipc

IPC values are derived from cycle-count packets when cyc is used, or from MTC packets otherwise. They are not a precise stopwatch for every instruction. The manual warns that Intel PT timing has limited granularity, and that smaller reporting periods create more samples, more work and potentially less useful statistics.

Before enabling optional terms such as mtc or changing psb_period, inspect the host capabilities:

$ grep -H . /sys/bus/event_source/devices/intel_pt/caps/* 2>/dev/null

Use only values the corresponding capability files report. An unsupported term or value should be treated as a hardware limitation, not repaired by guessing.

Checkpoint 5: trace kernel activity deliberately

Kernel tracing needs elevated access and better image handling. The documented pattern captures a copy of kernel data alongside the recording:

$ sudo perf record -o pt-echo --kcore -e intel_pt// -- /bin/echo intel-pt-kernel
$ sudo perf report -i pt-echo

This creates a directory named pt-echo containing a data file and copies of /proc/kcore, /proc/kallsyms and /proc/modules. The copy is made under the same conditions as capture, which helps the decoder match the executed kernel image. Treat the directory as sensitive: it can contain detailed execution data and kernel symbol information. Restrict its permissions and remove it according to your evidence-retention policy when finished.

Do not add sudo to the userspace example merely because kernel tracing exists. Keep the smallest privilege boundary that answers the question. If the command fails because the PMU is unavailable, privilege escalation will not create Intel PT support.

Common failure modes

• Decoder errors: perf may not be able to access an executed image, match side-band mmap or context-switch data, or follow JIT-compiled and self-modifying code. Preserve the exact binaries and symbols used by the workload where possible.
• Lost data: the perf buffer can fill faster than perf copies it. Reduce the workload and trace duration first. A report from a recording with loss is not a complete account of execution.
• Huge output: do not begin with all instructions. Use perf report or --call-trace, then restrict a time range or CPU before requesting instruction output.
• Pipe mode: the manual does not recommend using a pipe as Intel PT output. Auxtrace buffers can arrive out of timestamp order because their head and tail handling differs from ordinary perf buffers. Use a normal perf.data file.