Receive Remote Traces with trace-cmd listen

You will run a trace-cmd listener on one Linux host and receive a recording from another host into a trace data file. The examples use the installed trace-cmd 3.2.0, where the listener is started with trace-cmd listen -p PORT.

Allow about fifteen minutes for a first test, plus the time needed to produce the recording on the remote machine. You need shell access to both hosts, a writable destination directory on the listener, and a network path between the chosen port and the remote recorder. The normal listener command is unprivileged. Use elevated privileges only if your chosen directory or network policy genuinely requires them.

1. Confirm the installed command

Check the binary and version on the host that will receive the trace:

$ command -v trace-cmd
/usr/bin/trace-cmd
$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)

Your path and version may differ. Keep the version in your notes when troubleshooting: option details and diagnostics belong to the installed release, not to every older or newer package.

Checkpoint: ask the installed subcommand for its syntax. This does not open a socket:

$ trace-cmd listen --help
usage:
 trace-cmd listen -p port[-D][-o file][-d dir][-l logfile]
          Creates a socket to listen for clients.

2. Prepare a dedicated output directory

Make a directory owned by your account, then enter it. This keeps received files separate from unrelated traces and makes the listener's write permission easy to check:

$ mkdir -p "$HOME/trace-cmd-received"
$ test -w "$HOME/trace-cmd-received" && echo 'destination is writable'
destination is writable

Do not use a shared temporary directory for sensitive traces unless its access policy is understood. If an administrator-created directory is required, ask the administrator to grant the least access needed. Do not make it world-writable as a shortcut.

Checkpoint: choose a high, unused port such as 54321, and record the listener host's address. The remote recorder will need both values.

3. Start a foreground listener

Start in the foreground first. The -p option selects the listening port and -d selects the directory for received data files:

$ trace-cmd listen -p 54321 -d "$HOME/trace-cmd-received"

The command normally stays attached to the terminal while it waits. That is useful for the first run because messages remain visible. Leave this terminal open and use a second terminal for the remote recording.

There is no completed recording to verify yet. A useful local check is to confirm that the process is still running and that the directory is reachable:

$ pgrep -af 'trace-cmd listen -p 54321'
$ find "$HOME/trace-cmd-received" -maxdepth 1 -type f -print

The file list can be empty before a client connects. Stop this foreground test with Ctrl-C when you need to change its options. No persistent service configuration is created, so restarting the command is the undo operation.

4. Send a short recording from the remote host

On the other host, replace LISTENER_HOST with a DNS name or address that it can reach. -N sends the recording to the listener instead of writing the normal local output file:

$ trace-cmd record -e sched_switch -e irq_handler_entry -N LISTENER_HOST:54321 sleep 5

This example records two representative kernel events for five seconds. Event availability is host-specific, so if the recorder rejects an event, list the events on that host and choose events it reports as available. The listener manpage describes the default transport as UDP. The recording manpage also documents -t with -N when TCP delivery is needed:

$ trace-cmd record -t -e sched_switch -N LISTENER_HOST:54321 sleep 5

Use TCP only when its delivery behaviour fits the trace and network. It is not a cure for a blocked port or an incorrect address.

Checkpoint: return to the listener terminal. After the remote command sends data, look in the destination directory:

$ find "$HOME/trace-cmd-received" -maxdepth 1 -type f -printf '%f\n'
trace.REMOTE_HOST:REMOTE_PORT.dat

The host and port portion is generated from the remote connection, so the exact filename will differ. Do not copy that example filename into a script as a fixed value.

5. Choose a predictable filename when needed

By default, the listener uses a name based on trace.HOST:PORT.dat. Pass -o when your workflow needs an output filename instead of that default:

$ trace-cmd listen -p 54321 -d "$HOME/trace-cmd-received" -o overnight-trace

The option overrides the default trace part used for a client file. Test the result with find after a client has connected rather than assuming the final name from the command line alone. If several clients can connect, plan the naming and storage policy before using one fixed name.

Do not point -o at a valuable existing file without checking the installed behaviour and your backup policy. A received trace can replace or conflict with an existing output. Keep the original remote data and use a new destination while testing.

6. Run it in the background only after testing

The -D option makes the listener daemonise:

$ trace-cmd listen -D -p 54321 -d "$HOME/trace-cmd-received" -l "$HOME/trace-cmd-received/listen.log"

Here -l sends output messages to a log file instead of standard output. Check that the process and log exist before asking the remote host to record:

$ pgrep -af 'trace-cmd listen'
$ test -f "$HOME/trace-cmd-received/listen.log" && echo 'listener log exists'
listener log exists

Daemon mode is not a service manager. It does not give you restart policy, boot ordering, log rotation or access control. If this receiver must survive reboots, create a reviewed service definition separately and test it in a maintenance window. To stop a daemon started for this test, identify its exact process with pgrep and terminate that process; do not kill every trace-cmd process on a shared machine.

7. Use vsockets for a guest VM

The -V option selects a vsocket rather than a normal network socket. Use it when tracing between a host and guest VM that support the relevant vsocket setup:

$ trace-cmd listen -V -p 54321 -d "$HOME/trace-cmd-received"

The corresponding recorder must use the vsocket form and the correct guest or host context. The network example and the vsocket example are alternatives; do not add -V merely to solve a normal TCP or UDP connectivity problem.

8. Diagnose a missing file or failed connection

Work through these checks in order:

1. Confirm that the listener is still running in the foreground terminal, or find the exact daemon process with pgrep.
2. Confirm that both hosts use the same port and that the remote host resolves LISTENER_HOST to the intended machine.
3. Check the listener firewall and any network ACL for the chosen port. Change policy only through your normal reviewed administration process.
4. Check the destination directory with test -w; a listener that cannot write cannot produce a useful file.
5. Read the listener output or -l log. Increase diagnostics with --verbose=debug if needed:

$ trace-cmd listen --verbose=debug -p 54321 -d "$HOME/trace-cmd-received"

The supported log levels include none, critical, error, warning, info, debug and all. The default is info. Debug output can contain operational details, so return to the normal level after the fault is understood.

If the listener is reachable but the recorder fails, test the recorder without changing the listener's directory or service state. Check the event names and the recorder's own error output. A successful connection does not prove that every requested trace event exists on the remote kernel.