Build a Python Trace Script with perf script

Recording a trace is easy; turning it into a report means writing a handler, and perf script -g python saves you from guessing the signature. This guide records a focused Linux trace, generates a Python handler skeleton from it, and turns that into a small aggregation script. Allow 20 to 30 minutes for a first run.

You need the perf tools, a writable working directory, and enough privilege to record the events you choose. It follows the installed perf-script-python(1) manual from linux-tools-common version 6.8.0-142.142.

The examples use system-wide syscall entry tracing, which can produce a large file quickly. Use a short capture and check the output before extending it. Recording is normally the only step that needs elevated privileges.

1. Check the installed perf tool

Start in an empty directory so the capture and generated script are easy to identify:

$ mkdir -p "$HOME/perf-python-example"
$ cd "$HOME/perf-python-example"
$ perf --version

You should get a version line. On this machine, /usr/bin/perf is present but reports that the kernel-specific linux-tools-6.8.0-139-generic package is missing, so the command cannot run successfully until you install the matching tool through the normal package-management process. Do not confuse the manual's package version with a working perf binary for the running kernel.

2. Record a short event stream

Record syscall-entry events for every CPU for a few seconds:

$ sudo perf record -a -e raw_syscalls:sys_enter -- sleep 5
[ perf record: Captured and wrote ... perf.data (... samples) ]

-a makes collection system-wide. The event is one raw tracepoint for syscall entry, so the event's id field distinguishes calls. The manual's longer interactive example records until you press Ctrl-C; the bounded sleep command here is easier to review and limits the capture.

This creates perf.data in the current directory, and it contains activity from every user and process observed during the capture.

$ test -s perf.data && printf '%s\n' 'perf.data is ready'
perf.data is ready

3. Generate the starter script

Generate handlers from the event types found in the capture:

$ perf script -g python
generated Python script: perf-script.py
$ test -s perf-script.py && printf '%s\n' 'starter script generated'
starter script generated

The file is a diagnostic skeleton. It adds the perf support-module path, imports the helper modules, and creates a function for each event type found in perf.data. Event handlers are named subsystem__event_name, so for this capture the relevant one is raw_syscalls__sys_enter.

The installed manual's generated example uses Python 2-style print statements. Do not blindly bolt on Python 3 syntax or assume that whatever system Python happens to be installed is the interpreter perf will use. Inspect the generated file and test it against the installed perf build first:

$ sed -n '1,120p' perf-script.py
$ perf script -s perf-script.py | sed -n '1,12p'

Expected output is a series of event lines with the event name, CPU, timestamp, process ID, command name and event fields. If it is empty, check that perf.data is in the current directory and that the recording actually collected samples.

4. Replace printing with an aggregation

Copy the starter before editing it, then add a counter and print totals at the end. The manual documents autodict() for this and syscall_name() for turning raw IDs into names:

$ cp --preserve=all perf-script.py syscall-counts.py

syscalls = autodict()

def trace_end():
    for syscall_id, count in sorted(syscalls.iteritems(),
                                    key=lambda item: (item[1], item[0]),
                                    reverse=True):
        print "%s %d" % (syscall_name(syscall_id), count)

def raw_syscalls__sys_enter(event_name, context, common_cpu,
                            common_secs, common_nsecs, common_pid,
                            common_comm, id, args):
    try:
        syscalls[id] += 1
    except TypeError:
        syscalls[id] = 1

Keep the imports and support-path setup from the generated file. Add the counter at module level and replace the generated event handler with the version above. trace_end() runs after every event has been processed, so the report prints once rather than once per syscall.

Run it against the existing capture:

$ perf script -s syscall-counts.py | sed -n '1,12p'
read 923
close 3037
write 455067

The counts and order will differ on your machine. What matters is that the command exits successfully and prints syscall names followed by integer counts. If the handler is never called, regenerate the skeleton from the current perf.data: a script generated for different event types can report unhandled events or no useful totals.

5. Handle failures without losing the capture

Keep perf.data until the script has produced a report you trust. A Python error does not require a new recording; fix the script and rerun:

$ perf script -s syscall-counts.py > syscall-counts.txt
$ status=$?
$ printf 'perf script exit status: %s\n' "$status"
$ test "$status" -eq 0

When you are finished, remove the trace only after checking the report is useful:

$ rm -- perf.data perf-script.py syscall-counts.py syscall-counts.txt

6. Keep a reusable script separate from recording

For a longer-lived tool, record only the tracepoints the script needs. Use perf list to find candidate events and inspect the corresponding files under /sys/kernel/tracing/events/ for their fields. Generate a fresh skeleton after changing the event set, then keep the Python script under version control with a short note naming the required events.

The manual also describes installing paired -record and -report shell scripts under the perf source tree's scripts/python/bin directory. That is a packaging workflow, not a requirement for running a local script. Do not modify a system perf installation merely to make a private script appear in perf script -l.