Run trace-cmd-agent safely for remote kernel tracing

You will prepare a trace-cmd agent that accepts tracing control from one other machine, with the transport chosen explicitly. The normal default is a virtual-machine vsocket. TCP is available for a specified client, but the installed manual warns that this mode is very insecure and should be limited to a trusted network and a completely trusted controller.

Allow about 15 minutes for a foreground smoke test and a further review of the VM or network boundary. This guide uses trace-cmd 3.2.0 from the installed trace-cmd package, version 3.2-1ubuntu2.

1. Check the installed command

Run the checks as the account that will operate the agent. Starting an agent can expose tracing control, so do not test it on a shared host or as a casual background process.

$ command -v trace-cmd
/usr/bin/trace-cmd
$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)
$ trace-cmd agent --help
trace-cmd version 3.2.0 (not-a-git-repo)
...
 trace-cmd agent -p port[-D][-N IP][-P cid]

Your version line may differ on another machine. Keep the local help output with the deployment notes. The agent subcommand is part of trace-cmd; there is no separate trace-cmd-agent executable to launch.

2. Choose the transport and controller

Use the default vsocket when the controller and agent are connected through a virtual-machine setup that provides vsock. The manpage describes the agent as listening on a vsocket for virtual machines and then passing trace data to the controlling connection. A vsocket setup still needs a defined guest or host boundary; it is not a reason to treat every local process as trusted.

Use TCP only when you have a specific controller address and a network design that protects the connection. The -N value is not a bind address: it names the one client host or IP address that may connect. The manual also says that any process on that client can control the agent. That is a broad trust decision, not authentication for individual users.

3. Select a listening port

The -p option supplies the port number to listen on. Pick a documented, permitted port that does not collide with another service. The option is required by the installed command's usage even when the default vsocket transport is used.

$ TRACE_PORT='REPLACE_WITH_YOUR_PORT'
$ printf 'agent port: %s\n' "$TRACE_PORT"
agent port: REPLACE_WITH_YOUR_PORT

Replace the placeholder with a numeric port before launching the agent. Do not paste the placeholder into a production command. Port selection alone does not restrict who may control a vsocket agent; access still comes from the transport and its surrounding VM configuration.

4. Run a foreground vsocket smoke test

For a first test, leave out -D. Start the agent in the foreground so its lifetime is visible and so you can stop the test from the terminal with the normal interrupt for that shell process. This example changes runtime state by opening a listening endpoint, but it does not edit a configuration file.

$ sudo trace-cmd agent -p REPLACE_WITH_YOUR_PORT

Use sudo only if the account needs it for the tracing facilities on your host. If the agent starts successfully, it normally waits for a controlling connection rather than printing a completed result. That wait is the expected foreground state. Keep the terminal open while the trusted controller performs its own connection test.

5. Use TCP only with an explicit trusted client

For a TCP test, combine -N with -p. Replace the two obvious placeholders with the controller's address and the chosen port. The address should be an exact host name or IP address, not a whole subnet.

$ sudo trace-cmd agent -N CONTROLLER_HOST_OR_IP -p REPLACE_WITH_YOUR_PORT

The agent accepts a connection only from the client supplied to -N, according to the installed manual. This is an allow-list for the client host, not encryption, user authentication or per-command authorisation. A compromised or multi-user controller can therefore control the agent. Put the endpoint behind the narrowest network path available and treat the connection as sensitive.

Do not assume that a successful TCP connection proves a safe deployment. Verify the source address seen by your network controls, check the controller's ownership, and keep the agent in the foreground until the boundary has been reviewed. The recovery action is simple: stop the foreground process and remove any temporary network exposure you created. Do not leave a test listener running because the command returned no error.

6. Add daemon mode only after the test works

The -D option makes the agent daemonise and run in the background. Add it only after the foreground command has used the right transport, port and controller boundary.

$ sudo trace-cmd agent -N CONTROLLER_HOST_OR_IP -p REPLACE_WITH_YOUR_PORT -D

There is no foreground terminal to act as your checkpoint after -D. Use the process manager or service wrapper that owns the command, and confirm its process and listening endpoint using the host's normal operational checks. If you have no reliable owner for the background process, leave out -D rather than creating an unmanaged listener.

To undo this change, use the stop or restart mechanism of the process manager that launched the agent, after confirming its exact process or service identity. Avoid broad commands such as killing every trace-cmd process: that could interrupt an unrelated recording or controller.

7. Treat proxy mode as a separate VM design

The -P option makes the agent also act as a proxy server. Its argument is a context ID, or CID, for the client guest that it will allow to connect. The CID is a vsock concept, not a TCP port and not a hostname.

$ sudo trace-cmd agent -P REPLACE_WITH_CLIENT_CID -p REPLACE_WITH_YOUR_PORT

Use this only when the host and guest tracing path has been designed around vsock and you have confirmed the guest's CID from your VM tooling. Do not substitute the guest's IP address. Because proxy mode changes which VM endpoint can reach the agent, test it in the foreground first and document the host, guest, CID and port together.

8. Turn on focused diagnostics when needed

The optional --verbose setting accepts named levels none, critical, error, warning, info, debug and all, or the identifiers 0 through 6. If no level is supplied, the manual says it defaults to info. A selected level also enables the preceding levels.

$ sudo trace-cmd agent -N CONTROLLER_HOST_OR_IP -p REPLACE_WITH_YOUR_PORT --verbose=warning

Use a level that answers the current question, then return to the default once the test is complete. The manual's example shows the syntax on trace-cmd listen, but the option is documented for the agent as well. Diagnostics do not add authentication or repair a transport mismatch.