Add and Remove Linux Dynamic Probes with perf probe

perf probe drops a tracepoint on a live kernel function or a running executable without recompiling anything. It can also change live tracing state, so treat it with the same care as any other production change.

Allow 15 to 30 minutes for a first probe, longer if you need to track down matching debuginfo or source. Start with discovery and --dry-run, and do not add a probe to a production kernel or service until you know its hit rate and have a removal command ready.

1. Check the installed perf and its prerequisites

The local manpage belongs to linux-tools-common version 6.8.0-142.142, documenting the perf 6.8 command family. The installed machine runs kernel 6.8.0-139-generic, but its /usr/bin/perf wrapper reports the matching kernel-specific perf package is missing. Check your own pair before trying to debug a probe error:

$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
linux-tools-common 6.8.0-142.142
$ uname -r
6.8.0-139-generic
$ perf --version
WARNING: perf not found for kernel 6.8.0-139

Your version and warning may read differently. On Ubuntu, install the matching kernel tools through your normal package process if you see this warning. This guide does not install packages or change the host for you.

2. Understand which kind of probe you are making

Without -x or -m, perf probe targets the running kernel. A kernel probe can use a function name, a function offset or return, a source line, a lazy source pattern, and arguments. A probe on an executable or shared library is a uprobe, and uses -x PATH; the manpage also allows an absolute path as the first non-option argument.

• Need source-level kernel variables? They need debuginfo: -k VMLINUX for a vmlinux file with DWARF data, and -s SOURCE for the matching kernel source tree.
• Probing a module? Use -m MODULE. Passing a module file path makes it an offline module, so it does not need to already be loaded.

The smallest useful probe specification is a function name, with an optional event name and group in front of it:

[GROUP:]EVENT=FUNCTION [ARGUMENT ...]

Skip the event name and perf derives it from the function; a return probe gets a __return suffix. Skip the group and the documented defaults are probe for kprobes and probe_<binary> for uprobes.

3. Discover functions or source lines without changing tracing

Use --funcs to list functions available in the kernel or a module; with -x it inspects an executable or shared library instead. Filter when the list is large:

$ sudo perf probe --funcs='tcp_*'
# matching functions are printed by the installed perf
$ perf probe -x /path/to/program --funcs='worker*'
# matching user-space functions are printed by the installed perf

--funcs takes a filter pattern, and its documented default hides names starting with an underscore. That is the filter doing its job, not proof those functions do not exist.

Once debuginfo is available, ask for source lines before committing to a line probe:

$ sudo perf probe --line 'FUNCTION'
$ sudo perf probe --line '/path/to/source.c:100-120'
$ sudo perf probe --vars 'FUNCTION'

The first form shows probeable lines inside a function, the second a source range, and --vars lists local variables available at a probe point. Do not add a variable argument until --vars shows it exists there; --vars itself takes probe syntax but no arguments of its own.

4. Preview the exact change

Quote the whole probe specification in one argument when it contains shell-significant characters. This kernel example is from the documented syntax, with an event name and one recorded local variable:

$ sudo perf probe --dry-run --add='sched_probe=schedule:12 cpu'

The exact diagnostic depends on the kernel, symbols and installed perf. What matters is a successful exit status and no new event actually registered in tracing. If the function or line cannot be resolved, fix the target, debuginfo or source path before removing --dry-run.

For a user-space function, preview against the real executable:

$ perf probe --dry-run -x /path/to/program --add='worker'
# perf validates the uprobe definition without adding it

Keep the target path and function name tied together: a function name from a different build or shared library is not a valid substitute, even if it happens to match by name.

5. Add and verify one probe

Once the dry run succeeds, add the same specification. A kernel probe normally needs elevated privileges because perf uses tracefs and kallsyms; a user-space probe can still need access to the executable and the tracing interface:

$ sudo perf probe --add='sched_probe=schedule:12 cpu'
Added new event:
  probe:sched_probe (on schedule:12 with cpu)
$ sudo perf probe --list='probe:sched_probe'
  probe:sched_probe

Output wording varies by perf release, so verify the event name rather than trusting the sample lines literally. If the name already exists, pick a new one, or use --force only when replacing that event is genuinely intentional. The default cap is 128 probe points per event; a wildcard or lazy pattern can match more than you expected, so consider --max-probes=NUM for those.

For a user-space probe, the same lifecycle applies with -x:

$ sudo perf probe -x /path/to/program --add='worker'
$ sudo perf probe --list='probe_program:worker'

The default uprobe group name is derived from the binary. Read the list output and use its actual GROUP:EVENT value when filtering or deleting later.

6. Remove the probe when the test is over

$ sudo perf probe --list='probe:sched_probe'
  probe:sched_probe
$ sudo perf probe --del='probe:sched_probe'
Removed event: probe:sched_probe
$ sudo perf probe --list='probe:sched_probe'
# no matching live event is listed

--del accepts glob wildcards such as *, ? and character classes. Quote a wildcard so the shell does not expand it, and always list first: something like --del='probe:schedule*' can remove several events at once. There is no general undo beyond re-adding the original definitions, so save those definitions if you will need the probe again.

7. Diagnose permission and matching failures

A failed add or list is usually environmental. Check these in order:

• Tracefs and /proc/kallsyms access. These may need root or a privileged user for --add, --del and live --list; a cached listing is different.
• /proc/sys/kernel/kptr_restrict set to 2. This blocks the kernel symbol information kprobes need. Changing it is a system-wide security decision: ask the administrator rather than flipping it yourself.
• Symbol and debuginfo mismatch. The kernel or module must expose the symbol, and source-level variables need readable, matching debuginfo. A source tree from a different build can point perf at the wrong lines entirely.
• Wrong build for a uprobe. Check the executable or library path is the intended build, and use --target-ns PID when the target process lives in another mount namespace.

Use --verbose for parsed arguments and diagnostics, or --quiet when a script needs to suppress messages; they are mutually exclusive. A successful add means the event was registered, not that it has fired: run your normal perf recording workflow to actually hit it, then remove it afterwards.