Receive Remote Traces with trace-cmd listen

trace-cmd listen turns one Linux host into a socket that catches a trace-cmd recording sent from another machine. That is exactly what you want when the box doing the tracing cannot spare local disk for the result. The listener starts with trace-cmd listen -p PORT; this guide uses the installed trace-cmd 3.2.0.

• You need: shell access on both hosts, a writable destination directory on the listener, and a network path to the chosen port. Allow about fifteen minutes for a first test, plus whatever time the remote recording itself takes.
• It will not need: root, ordinarily. Only reach for elevated privileges if your destination directory or network policy genuinely requires it.

1. Confirm the installed command

Check the binary and version on the host that will receive the trace:

$ command -v trace-cmd
/usr/bin/trace-cmd
$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)

Your version may differ, and that is fine, just keep it in your notes: option details and diagnostics belong to the installed release, not to every version that ever shipped.

$ trace-cmd listen --help
usage:
 trace-cmd listen -p port[-D][-o file][-d dir][-l logfile]
          Creates a socket to listen for clients.

2. Prepare a dedicated output directory

Make a directory you own and use it exclusively for this, so received files stay separate from unrelated traces and the write permission is trivial to check:

$ mkdir -p "$HOME/trace-cmd-received"
$ test -w "$HOME/trace-cmd-received" && echo 'destination is writable'
destination is writable

Do not use a shared temp directory for anything sensitive unless you actually understand its access policy, and do not make the directory world-writable as a shortcut. If an administrator-managed directory is required, ask for the least access that does the job.

3. Start a foreground listener

-p sets the listening port and -d sets the directory for received files. Run it in the foreground first so any messages stay visible:

$ trace-cmd listen -p 54321 -d "$HOME/trace-cmd-received"

The command stays attached to the terminal while it waits, so leave this window open and use a second one for the remote recording. There is nothing to verify yet beyond the process itself and the directory:

$ pgrep -af 'trace-cmd listen -p 54321'
$ find "$HOME/trace-cmd-received" -maxdepth 1 -type f -print

An empty file list before a client connects is normal. Stop this foreground test with Ctrl-C whenever you need to change an option; restarting the command is the whole undo procedure, since nothing persistent gets created.

4. Send a short recording from the remote host

On the other machine, replace LISTENER_HOST with a name or address it can actually reach. -N sends the recording straight to the listener instead of writing the usual local output file:

$ trace-cmd record -e sched_switch -e irq_handler_entry -N LISTENER_HOST:54321 sleep 5

This records two representative kernel events for five seconds. Event availability is host-specific, so if the recorder rejects one, list events on that host and pick from what it actually reports. The default transport is UDP; when you need TCP delivery instead, the record manpage documents -t alongside -N:

$ trace-cmd record -t -e sched_switch -N LISTENER_HOST:54321 sleep 5

Reach for TCP only when its delivery behaviour actually fits the trace and network. It will not fix a blocked port or a wrong address.

$ find "$HOME/trace-cmd-received" -maxdepth 1 -type f -printf '%f\n'
trace.REMOTE_HOST:REMOTE_PORT.dat

The host and port portion is generated from the remote connection, so your real filename will differ. Do not hardcode that example name into a script.

5. Choose a predictable filename when you need one

By default the listener names files trace.HOST:PORT.dat. Pass -o when your workflow needs a fixed name instead of that generated one:

$ trace-cmd listen -p 54321 -d "$HOME/trace-cmd-received" -o overnight-trace

This only overrides the trace part of the generated filename. Confirm the actual result with find after a client connects rather than guessing from the command line alone, and if several clients might connect, settle your naming and storage plan before you commit to one fixed name.

6. Run it in the background only after testing

-D daemonises the listener, and -l sends its messages to a log file instead of standard output:

$ trace-cmd listen -D -p 54321 -d "$HOME/trace-cmd-received" -l "$HOME/trace-cmd-received/listen.log"

Confirm both the process and the log exist before asking the remote host to record anything:

$ pgrep -af 'trace-cmd listen'
$ test -f "$HOME/trace-cmd-received/listen.log" && echo 'listener log exists'
listener log exists

Daemon mode is not a service manager: it gives you no restart policy, no boot ordering, no log rotation and no access control of its own. Build a proper service definition, tested in a maintenance window, if this receiver needs to survive a reboot. To stop a test daemon, find its exact process with pgrep and kill that one; do not kill every trace-cmd process on a shared machine.

7. Use vsockets for a guest VM

-V selects a vsocket instead of a normal network socket, for tracing between a host and a guest VM that supports the matching vsocket setup:

$ trace-cmd listen -V -p 54321 -d "$HOME/trace-cmd-received"

The recorder on the other end has to use the vsocket form too, with the right guest or host context. Treat the network and vsocket examples as alternatives: do not bolt -V onto a setup just because a normal TCP or UDP connection is not working.

8. Diagnose a missing file or a failed connection

1. Check the listener is still running. Look at the foreground terminal, or find the daemon process with pgrep.
2. Check both sides agree on the port and that the remote host resolves LISTENER_HOST to the right machine.
3. Check the firewall and any network ACL on the chosen port, and change policy only through your normal reviewed process.
4. Check the destination is writable with test -w; a listener that cannot write cannot produce anything useful.
5. Read the listener output or the -l log, and turn up diagnostics with --verbose=debug if you need more:

$ trace-cmd listen --verbose=debug -p 54321 -d "$HOME/trace-cmd-received"

Supported log levels are none, critical, error, warning, info, debug and all, with info as the default. Debug output can contain operational detail, so drop back to the normal level once the fault is understood.

If the listener is reachable but the recorder still fails, test the recorder on its own without touching the listener's directory or service state: check event names and the recorder's own error output. A successful connection is no guarantee that every requested event actually exists on the remote kernel.