Trace a Short-Lived Command with perf ftrace

You will finish with a small, repeatable way to observe kernel function activity around one command, then choose a single function when a latency histogram is more useful than a stream of trace lines. The examples follow the installed perf-ftrace manual, dated 1 September 2026, from linux-tools-common version 6.8.0-142.142.

Allow about fifteen minutes. You need a Linux kernel with tracefs and a matching perf executable. Tracing normally needs elevated privileges because it uses kernel ftrace state. The commands below can observe and alter tracing for the duration of a run, so do not start them against a busy production workload without checking the impact first.

1. Check the installed tool and kernel interface

First establish which executable and package you have. These checks are ordinary and read-only:

$ command -v perf
/usr/bin/perf
$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
linux-tools-common 6.8.0-142.142
$ findmnt -T /sys/kernel/tracing
TARGET              SOURCE  FSTYPE  OPTIONS
/sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime

The package version and running kernel do not have to share the same number, but the executable must support your kernel. On this machine, perf --version reports that the kernel-specific 6.8.0-139 tool is missing and exits with status 2. Treat that as a prerequisite failure, not as a tracing result. Install the matching tools through your normal package-management process before continuing. Do not work around a version mismatch by copying an arbitrary perf binary.

Checkpoint: run perf --version. Continue only when it prints a version and exits successfully.

2. Discover functions without starting a trace

perf ftrace trace accepts the -F or --funcs option to list available functions. Pass a narrow pattern rather than dumping every symbol:

$ sudo perf ftrace trace --funcs 'sched_*'
<matching kernel functions are printed here>

The exact list depends on the running kernel and its configuration. An empty list means that the pattern did not match, not that ftrace has no functions. Use a name from the output for the next step. Keep the quotes around a glob: the pattern is for perf, not for the shell.

sudo is shown because reading and configuring tracefs is commonly restricted. If your account already has the required capability, omit it. Do not make tracefs broadly writable just to avoid using controlled elevation.

3. Trace one command with a function filter

Use --trace-funcs (or -T) to select a function or glob pattern. This selects the function tracer and applies a filter through tracefs. Start with a harmless, short-lived command:

$ sudo perf ftrace trace --trace-funcs 'sched_*' /usr/bin/true
<function trace lines appear here>

true may finish before it exercises a useful path, so a slightly longer command can make the output easier to inspect:

$ sudo perf ftrace trace --trace-funcs 'sys_*' /bin/sh -c 'for n in 1 2 3; do printf "%s\n" "$n"; done'
<function trace lines appear here>

The output is text read from the kernel trace pipe and written to standard output. It is not a saved recording. Redirect it to a new file if you need to inspect it later:

$ sudo perf ftrace trace --trace-funcs 'sched_*' /usr/bin/true > /tmp/perf-ftrace-trace.txt
$ test -s /tmp/perf-ftrace-trace.txt && echo 'trace output captured'
trace output captured

Do not use a broad pattern such as * casually. It can produce a large stream, obscure the event you wanted and add measurable overhead. The trace subcommand currently supports one target thread; it is a simple wrapper around ftrace rather than a general multi-process recording tool.

4. Make the trace easier to read

For a call hierarchy, use the function graph tracer and restrict it with --graph-funcs (or -G):

$ sudo perf ftrace trace --graph-funcs 'vfs_read' /usr/bin/cat /etc/hostname
<function graph output appears here>

Graph options can add context without changing the target command. For example, --graph-opts verbose requests process names, PIDs and timestamps. --graph-opts depth=4 limits the maximum graph depth. These settings affect presentation and collection cost, so add one at a time when comparing runs.

To delay collection until after a program starts, use --delay with milliseconds:

$ sudo perf ftrace trace --delay 1000 --trace-funcs 'sched_*' /path/to/your-command --arg VALUE

Replace both placeholders with a command you understand. The delay is useful when startup noise hides the operation under investigation. It does not make the command safe to run indefinitely.

5. Measure one function as a histogram

Use the latency subcommand when you want execution-time distribution rather than individual trace lines. It accepts only one function for the histogram:

$ sudo perf ftrace latency --trace-funcs vfs_read /usr/bin/cat /etc/hostname
<latency histogram appears here>

The default histogram unit is microseconds. Add --use-nsec when nanoseconds are the useful base unit:

$ sudo perf ftrace latency --trace-funcs vfs_read --use-nsec /usr/bin/cat /etc/hostname
<nanosecond latency histogram appears here>

--use-bpf selects BPF measurement instead of the ftrace measurement path and uses the function graph tracer internally. That can require kernel BPF support and additional permissions. Treat a permission or feature error as information about the host, not a reason to disable security controls or run an unreviewed command as root.

6. Recover from common failures

If perf says that the kernel-specific tool is missing, check the running kernel and the available package versions before installing anything:

$ uname -r
6.8.0-139-generic
$ apt-cache policy linux-tools-$(uname -r)
<candidate information, if available>

If a function filter produces no lines, confirm the function name with --funcs, try a narrower known match and make the target command do enough work to reach it. If tracefs is not mounted or permission is denied, stop and ask the system administrator about the host's tracing policy. Do not manually edit tracefs files while debugging a perf command.

When the command exits, perf should stop its temporary collection. If you interrupted it with Ctrl-C, wait for the shell prompt and check that the command has ended before starting another trace. A failed run should not be used as evidence that the kernel path was never reached. Re-run with a smaller filter and capture the exit status:

$ sudo perf ftrace trace --trace-funcs 'sched_*' /usr/bin/true
$ printf 'perf status: %s\n' "$?"
perf status: 0

The status reports whether perf completed successfully; it does not certify that a particular function ran. The function list and trace output are the evidence for that narrower claim.