Add and remove Linux dynamic probes with perf probe

You will use perf probe to inspect a possible probe point, preview an operation, and then add, list or remove a dynamic tracepoint. The examples cover kernel functions and user-space executables. Allow 15 to 30 minutes for a first probe, longer if you need to locate matching debuginfo or source code.

These commands can change live tracing state and may require root. Start with discovery and --dry-run. Do not add a probe to a production kernel or service until you understand its hit rate and have a removal command ready.

1. Check the installed perf and its prerequisites

The local manpage belongs to linux-tools-common version 6.8.0-142.142. It documents the perf 6.8 command family. The installed machine is running kernel 6.8.0-139-generic, but its /usr/bin/perf wrapper reports that the matching kernel-specific perf package is missing. Check your own pair before trying to debug a probe error:

$ dpkg-query -W -f='${Package} ${Version}\n' linux-tools-common
linux-tools-common 6.8.0-142.142
$ uname -r
6.8.0-139-generic
$ perf --version
WARNING: perf not found for kernel 6.8.0-139

Your version and warning may differ. On Ubuntu, install the matching kernel tools through your normal package-management process if the wrapper gives this warning. This guide does not install packages or change the host.

Checkpoint: continue only when perf --version identifies a usable binary for the running kernel, or when you have deliberately chosen a matching standalone perf binary.

2. Understand which kind of probe you are making

Without -x or -m, perf probe addresses the running kernel. A kernel probe can use a function name, a function offset or return, a source line, a lazy source pattern, and arguments. A probe on an executable or shared library is a uprobe and uses -x PATH. The manpage also permits an absolute path as the first non-option argument.

Kernel source-level variables need debuginfo. Use -k VMLINUX for a vmlinux file with DWARF data and -s SOURCE for the corresponding kernel source tree. For a module, use -m MODULE; passing a module file path makes it an offline module, so it need not already be loaded.

The smallest useful probe specification is a function name. An optional event name and group precede it:

[GROUP:]EVENT=FUNCTION [ARGUMENT ...]

If you omit the event name, perf derives it from the function. A return probe uses the function name with a __return suffix. If you omit the group, the documented defaults are probe for kprobes and probe_<binary> for uprobes.

3. Discover functions or source lines without changing tracing

Use --funcs to list functions available in the kernel or a module. With -x, it can inspect an executable or shared library. Filter the result when the function list is large:

$ sudo perf probe --funcs='tcp_*'
# matching functions are printed by the installed perf
$ perf probe -x /path/to/program --funcs='worker*'
# matching user-space functions are printed by the installed perf

--funcs uses a filter pattern, and its documented default excludes names beginning with an underscore. That is a filter default, not proof that the functions do not exist.

When debuginfo is available, ask for source lines before choosing a line probe:

$ sudo perf probe --line 'FUNCTION'
$ sudo perf probe --line '/path/to/source.c:100-120'
$ sudo perf probe --vars 'FUNCTION'

The first form shows probeable lines in a function, the second shows a source range, and --vars lists local variables at a probe point. Do not add a variable argument until --vars shows that it is available. --vars accepts the probe syntax but does not accept arguments of its own.

4. Preview the exact change

Put the probe specification in one quoted argument when it contains shell-significant characters. The following is a kernel example from the documented syntax, with an event name and one recorded local variable:

$ sudo perf probe --dry-run --add='sched_probe=schedule:12 cpu'

The exact diagnostic depends on the kernel, symbols and installed perf. The important result is a successful exit status and no new event in tracing. If the function or line cannot be resolved, fix the target, debuginfo or source path before removing --dry-run.

For a user-space function, preview against the actual executable:

$ perf probe --dry-run -x /path/to/program --add='worker'
# perf validates the uprobe definition without adding it

Keep the target path and function name tied together. A function name from a different build or shared library is not a valid substitute.

5. Add and verify one probe

After the dry run succeeds, add the same specification. Adding a kernel probe normally needs elevated privileges because perf uses tracefs and kallsyms. A user-space probe can still need access to the executable and tracing interface:

$ sudo perf probe --add='sched_probe=schedule:12 cpu'
Added new event:
  probe:sched_probe (on schedule:12 with cpu)
$ sudo perf probe --list='probe:sched_probe'
  probe:sched_probe

Output wording and formatting vary by perf release. Verify the event name rather than copying the sample lines literally. If the name already exists, use a new name or use --force only when replacing that event is intentional. The default maximum is 128 probe points for one event; a wildcard or lazy pattern can match more than you expected, so consider --max-probes=NUM.

For a user-space probe, use the same lifecycle with -x:

$ sudo perf probe -x /path/to/program --add='worker'
$ sudo perf probe --list='probe_program:worker'

The default uprobe group includes a binary-derived name. Read the list output and use its actual GROUP:EVENT value when filtering or deleting.

6. Remove the probe when the test is over

Removal is a state-changing operation. First list the exact event, then delete that event rather than using a broad wildcard:

$ sudo perf probe --list='probe:sched_probe'
  probe:sched_probe
$ sudo perf probe --del='probe:sched_probe'
Removed event: probe:sched_probe
$ sudo perf probe --list='probe:sched_probe'
# no matching live event is listed

The --del option accepts glob wildcards such as *, ? and character classes. Quote a wildcard so the shell does not expand it, and inspect the list first. For example, --del='probe:schedule*' can remove several events. There is no general undo for a deletion beyond adding the original definitions again, so save those definitions if you will need the probe later.

7. Diagnose permission and matching failures

A failure to add or list a kernel probe is often environmental. Check these boundaries in order:

• Tracefs and /proc/kallsyms may require root or a privileged user for --add, --del and live --list. Cached listing is different.
• /proc/sys/kernel/kptr_restrict set to 2 prevents the kernel symbol information needed by kprobes. Changing it is a system-wide security decision; ask the administrator instead of changing it casually.
• The kernel or module must expose the symbol, and source-level variables require readable matching debuginfo. A source tree from another build can point perf at the wrong lines.
• For uprobes, check that the executable or library path is the intended build. Use --target-ns PID when the target process is in another mount namespace, as documented by perf probe.

Use --verbose for parsed arguments and diagnostics, or --quiet when a script must suppress messages. They are mutually exclusive. A successful add means the event was registered, not that it has been hit; use the event with your normal perf recording workflow and remove it afterwards.