Trace a Short-Lived Command with perf ftrace

perf ftrace wraps the kernel's ftrace machinery around one command, so you get a real trace without hand-editing tracefs. It follows the installed perf-ftrace manual, dated 1 September 2026, from linux-tools-common version 6.8.0-142.142. Budget about fifteen minutes: you will check the tool, list real function names, trace a harmless command, then move to a latency histogram.

• You need: a Linux kernel with tracefs and a perf executable that matches it.
• Privilege: tracing normally needs elevated access because it touches kernel ftrace state.
• Caution: the commands below can observe and alter tracing for the duration of a run, so do not point them at a busy production workload without checking the impact first.

1. Check the installed tool and kernel interface

These checks are ordinary and read-only:

$ command -v perf
/usr/bin/perf
$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
linux-tools-common 6.8.0-142.142
$ findmnt -T /sys/kernel/tracing
TARGET              SOURCE  FSTYPE  OPTIONS
/sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime

• Version numbers can differ. The package version and running kernel do not have to match, but the executable must support your kernel.
• Watch for a silent mismatch. On this machine, perf --version reports that the kernel-specific 6.8.0-139 tool is missing and exits with status 2. Treat that as a prerequisite failure, not a tracing result.
• Fix it properly. Install the matching tools through your normal package-management process. Do not work around a version mismatch by copying an arbitrary perf binary.

2. Discover functions without starting a trace

perf ftrace trace accepts -F or --funcs to list available functions. Pass a narrow pattern rather than dumping every symbol:

$ sudo perf ftrace trace --funcs 'sched_*'
<matching kernel functions are printed here>

• An empty list means no match, not that ftrace has no functions. The exact list depends on the running kernel and its configuration.
• Use a real name from the output for the next step.
• Keep the quotes around the glob: the pattern is for perf, not the shell.

sudo is shown because reading and configuring tracefs is commonly restricted. If your account already has the required capability, omit it. Do not make tracefs broadly writable just to avoid using controlled elevation.

3. Trace one command with a function filter

Use --trace-funcs (or -T) to select a function or glob pattern. This selects the function tracer and applies a filter through tracefs. Start with a harmless, short-lived command:

$ sudo perf ftrace trace --trace-funcs 'sched_*' /usr/bin/true
<function trace lines appear here>

true may finish before it exercises a useful path, so a slightly longer command can make the output easier to inspect:

$ sudo perf ftrace trace --trace-funcs 'sys_*' /bin/sh -c 'for n in 1 2 3; do printf "%s\n" "$n"; done'
<function trace lines appear here>

The output is text read from the kernel trace pipe and written to standard output. It is not a saved recording. Redirect it to a file if you need to inspect it later:

$ sudo perf ftrace trace --trace-funcs 'sched_*' /usr/bin/true > /tmp/perf-ftrace-trace.txt
$ test -s /tmp/perf-ftrace-trace.txt && echo 'trace output captured'
trace output captured

4. Make the trace easier to read

For a call hierarchy, use the function graph tracer and restrict it with --graph-funcs (or -G):

$ sudo perf ftrace trace --graph-funcs 'vfs_read' /usr/bin/cat /etc/hostname
<function graph output appears here>

• --graph-opts verbose adds process names, PIDs and timestamps.
• --graph-opts depth=4 limits the maximum graph depth.
• Add one at a time when comparing runs: these settings affect presentation and collection cost.

To delay collection until after a program starts, use --delay with milliseconds:

$ sudo perf ftrace trace --delay 1000 --trace-funcs 'sched_*' /path/to/your-command --arg VALUE

Replace both placeholders with a command you understand. The delay is useful when startup noise hides the operation under investigation. It does not make the command safe to run indefinitely.

5. Measure one function as a histogram

Use the latency subcommand when you want an execution-time distribution rather than individual trace lines. It accepts only one function for the histogram:

$ sudo perf ftrace latency --trace-funcs vfs_read /usr/bin/cat /etc/hostname
<latency histogram appears here>

The default histogram unit is microseconds. Add --use-nsec when nanoseconds are the useful base unit:

$ sudo perf ftrace latency --trace-funcs vfs_read --use-nsec /usr/bin/cat /etc/hostname
<nanosecond latency histogram appears here>

Common traps

• Kernel-tool mismatch. If perf says the kernel-specific tool is missing, check the running kernel and available package versions before installing anything:

  $ uname -r
  6.8.0-139-generic
  $ apt-cache policy linux-tools-$(uname -r)
  <candidate information, if available>

• No trace lines at all. Confirm the function name with --funcs, try a narrower known match, and make the target command do enough work to reach it.
• Tracefs unavailable or permission denied. Stop and ask the system administrator about the host's tracing policy. Do not manually edit tracefs files while debugging a perf command.
• An interrupted run. If you stopped it with Ctrl-C, wait for the shell prompt and confirm the command has ended before starting another trace. A failed run is not evidence the kernel path was never reached; re-run with a smaller filter and check the exit status:

  $ sudo perf ftrace trace --trace-funcs 'sched_*' /usr/bin/true
  $ printf 'perf status: %s\n' "$?"
  perf status: 0

  The status reports whether perf completed successfully; it does not certify that a particular function ran. The function list and trace output are the evidence for that narrower claim.