Configure a Focused Ftrace Session with trace-cmd set

You will enable a small, deliberate Ftrace configuration with trace-cmd set, check what the kernel accepted, and restore the tracing state afterwards. Allow 10 to 15 minutes for a first test, plus longer if you need to choose events or function filters.

1. Check the installed command and kernel access

This guide is written against trace-cmd 3.2.0 from Ubuntu package version 3.2-1ubuntu2. The command writes to the kernel's Ftrace interface, so the account normally needs elevated privileges and the kernel must expose the required tracing files.

$ command -v trace-cmd
/usr/bin/trace-cmd
$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)
$ trace-cmd set --help
trace-cmd version 3.2.0 (not-a-git-repo)
usage:
 trace-cmd set [-v][-e event [-f filter]][-p plugin]...

The last output is a syntax check, not a tracing change. List available tracers and events before choosing names:

$ sudo trace-cmd list -p
$ sudo trace-cmd list -e | less

Use names from your own output. A tracer or event is kernel-dependent, and a name that works on one host may not exist on another. If these commands report permission errors, fix access to the tracing filesystem or use an authorised account; do not invent an event name.

2. Enable one event with a bounded scope

Start with one scheduler event rather than -e all. The event argument can be a complete subsystem:event, a subsystem, an event name, a glob, or the keyword all. Confirm that sched_switch appeared in the event listing first.

$ sudo trace-cmd set -e sched_switch
$ printf 'set status: %s\n' "$?"
set status: 0

Normal output varies by kernel and trace-cmd build. Check the exit status immediately if a wrapper needs a strict success test:

A zero status means the command accepted the request. Check the effective event configuration through the trace interface:

$ sudo trace-cmd list -e | grep -E '(^|:)sched_switch$'
$ printf 'event-list status: %s\n' "$?"
event-list status: 0

Do not confuse an enabled event with recorded data. set configures Ftrace; collection and reporting are separate trace-cmd operations. The command also accepts a command after the options, and runs that command after applying the Ftrace state.

3. Add a tracer or a process filter only when needed

Use -p to select a plugin tracer supported by the kernel. Common names include function, function_graph, irqsoff, preemptoff, preemptirqsoff and wakeup. Check availability rather than assuming all of them exist:

$ sudo trace-cmd set -p function
$ sudo trace-cmd list -p | grep '^function$'
$ printf 'tracer-list status: %s\n' "$?"
tracer-list status: 0

Function tracing can produce a great deal of data. Narrow it with one or more -l function filters, or exclude a function with -n:

$ sudo trace-cmd set -p function -l schedule
$ sudo trace-cmd list -p | grep '^function$'
$ printf 'tracer-list status: %s\n' "$?"
tracer-list status: 0

To focus on one process, use its numeric process ID with -P PID. Add -c if you also need children and the running kernel supports that mode:

$ PID='12345'
$ sudo trace-cmd set -e sched_switch -P "$PID" -c

Replace the placeholder with a real PID. Do not paste an untrusted string into the option list. If you use --func-stack, heed the manpage warning: enabling stacks without successfully limiting function tracing can live-lock the machine.

4. Control buffers and output pressure

Ftrace buffers are per CPU. -b SIZE sets the ring buffer size in kilobytes for each CPU, so four CPUs and -b 10000 can reserve about 40 MB. For a long run, -m SIZE sets a maximum per-CPU buffer size and is intended to reduce the risk of exhausting disk space.

$ sudo trace-cmd set -e sched_switch -b 4096 -m 16384
$ sudo trace-cmd list -e | grep -E '(^|:)sched_switch$'
$ printf 'event-list status: %s\n' "$?"
event-list status: 0

These sizes are resource decisions, not cosmetic settings. Start small, watch memory and storage, and remove the limits only after you understand the workload. Use -q or --quiet when normal trace-cmd output would interfere with a wrapper's output. Use --stderr when trace-cmd diagnostics should go to standard error while the command's output remains unchanged.

5. Run a command after applying the state

Put the command after all trace-cmd options. Without --fork, trace-cmd waits for that command to finish; with --fork, it starts the command and returns immediately.

$ sudo trace-cmd set -e sched_switch -- /usr/bin/true
$ printf 'command status: %s\n' "$?"
command status: 0

The command is a separate process action, not a promise that the trace is useful. The exit status above tells you that the requested command returned successfully. Keep the command short for an initial test, and avoid putting a service start or restart in the first run.

6. Restore the tracing state

Do not leave tracing enabled by accident. In a controlled test, reset it when the collection is complete:

$ sudo trace-cmd reset
$ printf 'reset status: %s\n' "$?"
reset status: 0

Reset turns off Ftrace tracing and loses the ring-buffer data and options that were used. Save or report the trace before resetting if you need it. It can also affect other users of Ftrace, so coordinate with another tracing session instead of resetting someone else's work. If you created named buffer instances with -B, read the reset options carefully: deleting or resetting instances is a separate, potentially destructive action.

7. Diagnose the usual failures

• Unknown event: rerun sudo trace-cmd list -e and use an exact installed name. The default is to fail when a requested event is missing. -i tells trace-cmd to ignore missing events, but use it only when portability is more useful than a strict configuration check.
• Unsupported tracer: inspect sudo trace-cmd list -p. The running kernel, not the trace-cmd binary alone, determines availability.
• Permission denied: check the tracing filesystem and your delegated privileges. Running as root cannot make a missing kernel interface appear.
• Too much overhead: remove broad function tracing, reduce event scope, lower buffer sizes, or reset the configuration. Avoid -e all and unrestricted function stacks on a live system.
• Filter or trigger rejected: place -f or -R immediately after the event they modify. The kernel validates filter syntax, so supported fields depend on the event and kernel version.