Keep perf Recorders Running with perf daemon

You will configure perf daemon to supervise one or more background perf record sessions, check that they are online, trigger a recording dump, and stop the daemon without leaving it behind. Allow about 20 minutes for a first setup, plus time to choose events and an output location suitable for your workload.

Before you start

This guide follows perf-daemon(1) from the installed linux-tools-common package, version 6.8.0-142.142 on this machine. The local wrapper currently reports that the kernel-specific perf binary for 6.8.0-139 is missing, so these commands cannot be smoke-tested here. Install the matching kernel tools through your normal package manager before relying on them. The command syntax and status examples below come from the installed manual page and may differ in later perf releases.

You need a writable daemon base directory, an event that exists on the target host, and permission to open that event. System-wide recording with -a commonly needs elevated privileges or suitable perf permissions. Perf data can contain process names, paths, timing information and other sensitive details, so restrict the output directory and decide who may read it before starting.

Use a disposable base under /tmp while learning. For a real capture, choose a persistent filesystem with enough space and an explicit retention plan.

1. Check the installed command and event access

First check whether the matching executable is available and inspect the event names on the host:

$ command -v perf
$ perf --version
$ perf list | head

A warning that the kernel-specific perf package is missing is a packaging problem, not a daemon configuration error. Resolve that first. The list from perf list is host-specific. Replace the event in the examples with one that is present and permitted on your machine.

Checkpoint: do not start a daemon until perf --version succeeds and you have chosen the event and scope you intend to record.

2. Create a small daemon configuration

The daemon reads standard perf configuration syntax. The daemon.base value is the parent directory for daemon state and all session data. Each session-NAME.run value is a perf record command line without the word record. This example uses a temporary configuration file and one session:

$ cat > /tmp/perf-daemon.conf <<'EOF'
[daemon]
base=/tmp/perf-daemon-data

[session-cycles]
run = -m 10M -e cycles --overwrite --switch-output -a
EOF

The session combines a 10 MiB ring buffer, the cycles event, overwrite mode, periodic output switching, and all-CPU recording. The last part is why the example may need sudo. If your event or capture scope needs a different perf record command, change only the value after run =. Do not include perf record there.

Check the file before starting anything:

$ sed -n '1,20p' /tmp/perf-daemon.conf
[daemon]
base=/tmp/perf-daemon-data

[session-cycles]
run = -m 10M -e cycles --overwrite --switch-output -a

3. Start the daemon

Start with the configuration path and an explicit base path. Supplying --base makes the instance location clear and ensures that a second instance cannot silently share the same daemon directory:

$ sudo perf daemon --config=/tmp/perf-daemon.conf --base=/tmp/perf-daemon-data start

sudo is required only when the chosen recording scope or event requires it. If your permissions allow the capture as an ordinary user, omit it consistently from the start, status, signal and stop commands. Mixing users can make the daemon state inaccessible to the account that started it.

The start command creates the daemon process and returns it to the background. The base directory is locked so only one daemon instance can use it at a time. Starting again against the same base should be treated as an error condition rather than a second independent recorder.

Checkpoint: ask the daemon for its short status:

$ sudo perf daemon --config=/tmp/perf-daemon.conf --base=/tmp/perf-daemon-data
[603349:daemon] base: /tmp/perf-daemon-data
[603350:cycles] perf record -m 10M -e cycles --overwrite --switch-output -a

Process IDs in the output vary. The useful checks are that the daemon base is the directory you selected and that the expected session appears.

4. Inspect paths and confirm control access

Use verbose status when you need to see the files created for the daemon and session:

$ sudo perf daemon --config=/tmp/perf-daemon.conf --base=/tmp/perf-daemon-data -v
[603349:daemon] base: /tmp/perf-daemon-data
  output:  /tmp/perf-daemon-data/output
  lock:    /tmp/perf-daemon-data/lock
  up:      1 minutes
[603350:cycles] perf record -m 10M -e cycles --overwrite --switch-output -a
  base:    /tmp/perf-daemon-data/session-cycles
  output:  /tmp/perf-daemon-data/session-cycles/output
  control: /tmp/perf-daemon-data/session-cycles/control
  ack:     /tmp/perf-daemon-data/session-cycles/ack
  up:      1 minutes

The daemon output and lock belong to the daemon. The session directory contains its output, control and acknowledgement files. The up value is elapsed minutes, not a count of samples or output files.

Ping the session to check that its control channel is online:

$ sudo perf daemon --config=/tmp/perf-daemon.conf --base=/tmp/perf-daemon-data ping
OK   cycles

If a session is missing or does not answer, stop and inspect the session output before sending more signals. A successful daemon start does not prove that every child recorder opened its event successfully.

5. Trigger a data dump

The signal command sends the daemon's configured signal to its sessions. Use --session when you want to dump one session:

$ sudo perf daemon --config=/tmp/perf-daemon.conf --base=/tmp/perf-daemon-data signal --session cycles
signal 12 sent to session 'cycles' [603350]
$ tail -2 /tmp/perf-daemon-data/session-cycles/output
[ perf record: dump data: Woken up 1 times ]
[ perf record: Dump perf.data.2020123017013149 ]

The process ID and timestamp vary. Look for a new dump message and a new perf.data file under the session directory. To signal every configured session, omit --session:

$ sudo perf daemon --config=/tmp/perf-daemon.conf --base=/tmp/perf-daemon-data signal

This is a state-changing capture action. Do not use it as a casual health check; use ping for that.

6. Stop and recover cleanly

When the capture is finished, stop all sessions and the daemon:

$ sudo perf daemon --config=/tmp/perf-daemon.conf --base=/tmp/perf-daemon-data stop
$ find /tmp/perf-daemon-data -maxdepth 3 -type f -print

Keep the session directories if you need the captured files. Stopping does not mean you should delete the data. If the temporary capture is disposable, remove only the directory you created after checking the results:

$ rm -rf -- /tmp/perf-daemon-data /tmp/perf-daemon.conf

That removal is irreversible. Never adapt it to a production path or run it until the required perf.data files have been copied or analysed. For a failed start, leave the base in place, read the session output, correct the event or permissions, and retry after confirming that no daemon is still running. The base lock exists to prevent two instances from operating on the same state.