Trace a Linux Command with strace Without Drowning in Output

Point strace at a real program and thousands of lines scroll past before you can read any of them. That is why most people give up on it after one try. This guide sticks to a small set of repeatable commands for seeing which files a program touches, where it fails, which children it spawns, and which system calls dominate a run. The examples use strace 6.8 from Ubuntu package strace 6.8-0ubuntu2.

Allow about fifteen minutes. You need a shell and a command that is safe to run. The first examples are ordinary user commands; attaching to a process owned by someone else may need elevated privileges and a suitable ptrace policy.

1. Confirm the installed build

Check the version before you copy an option out of a guide. It also gives you something to compare against when a trace from another machine looks different:

$ strace --version
strace -- version 6.8
...
Optional features enabled: stack-trace=libunwind stack-demangle m32-mpers mx32-mpers

The copyright lines and feature list vary by build. The checkpoint that matters is the version line. If your system reports something older, run strace --help and read its local manpage before borrowing an option from elsewhere.

2. Trace a harmless command

Run a command directly after strace. The trace goes to standard error by default, while the command's normal output keeps its usual destination:

$ strace /bin/cat /dev/null
execve("/bin/cat", ["/bin/cat", "/dev/null"], ...) = 0
...
exit_group(0) = ?

The exact lines depend on the kernel, libraries and architecture. Each one names a system call, shows decoded arguments, and gives its return value. An error commonly shows up as a negative result with an errno, such as = -1 ENOENT (No such file or directory).

$ strace -o /tmp/cat.trace /bin/cat /dev/null
$ sed -n '1,12p' /tmp/cat.trace

-o only changes where strace writes its own trace; it does not redirect the traced command's output. The file is ordinary temporary state, so remove it once you are done inspecting it:

$ rm -- /tmp/cat.trace

3. Filter for file activity

A complete trace is often too noisy to answer one narrow question. Use the %file trace set to pick out system calls that take file names:

$ strace -e trace=%file /bin/cat /etc/hostname
execve("/bin/cat", ["/bin/cat", "/etc/hostname"], ...) = 0
openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, ...}, AT_EMPTY_PATH) = 0
... 
close(3) = 0

The set is broader than one guessed syscall name and covers calls such as openat, stat and unlink. The exact call sequence and decoded fields can differ on your machine. To identify the path behind a file descriptor, add -y:

$ strace -y -e trace=%file /bin/cat /etc/hostname 2>/tmp/hostname.trace
$ rg 'hostname|open|close' /tmp/hostname.trace

Do not paste traces into tickets without checking them first for credentials, tokens and private paths. Arguments can carry environment data, command arguments and file contents you did not mean to share.

4. Find failed calls

Use the status qualifier when the question is simply "what failed?" The keyword failed selects calls that returned an error:

$ strace -e status=failed /bin/cat /path/that/does/not/exist
openat(AT_FDCWD, "/path/that/does/not/exist", O_RDONLY) = -1 ENOENT (No such file or directory)
/bin/cat: /path/that/does/not/exist: No such file or directory

The traced command still reports its own normal failure on top of that. strace itself exits with the traced command's status, so check that status right away if a script depends on it:

$ strace -o /tmp/missing.trace /bin/cat /path/that/does/not/exist
$ printf 'command status: %s\n' "$?"
command status: 1

Keep the trace file if you need it, then delete it deliberately. Do not use a broad wildcard in a shared temporary directory to clean up.

5. Include children and separate their logs

Processes often fork or clone helpers. Add -f to follow children, or -ff to give each process its own file, suffixed with its process ID:

$ mkdir -p /tmp/strace-example
$ strace -ff -o /tmp/strace-example/trace sh -c 'printf "%s\n" child'
child
$ find /tmp/strace-example -maxdepth 1 -type f -name 'trace.*' -print

Expect one or more paths such as /tmp/strace-example/trace.12345, with a different process ID each time. -ff is incompatible with -c, because there is then no single per-process summary to keep. Remove the example directory once you are finished:

$ rm -rf -- /tmp/strace-example

That removal is safe only for the directory you just created. Never substitute a broader or unverified path.

6. Summarise instead of reading every line

Use -c when you want counts, errors and time broken down by system call rather than the raw trace:

$ strace -c /bin/cat /etc/hostname
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
... 
------ ----------- ----------- --------- --------- ----------------
100.00    ...                     ...         ... total

The numbers and rows depend entirely on the workload. The default summary reports system time, meaning CPU time spent in the kernel, not elapsed wall-clock time; add -w when elapsed time per call is what you actually need. Add -S calls to sort by call count instead of the default time ordering:

$ strace -c -S calls /bin/true

Use the summary to pick a smaller filter, then go back to ordinary output when you need arguments and return values as well.

7. Add timing and longer strings only when needed

-T appends the time spent inside each system call. -tt adds wall-clock timestamps with microsecond precision. They answer different questions, so pick the one you actually need:

$ strace -T -tt -o /tmp/timed.trace /bin/true
$ sed -n '1,8p' /tmp/timed.trace

String arguments are truncated at 32 bytes by default. Raise that limit with -s when truncation is actually the problem:

$ strace -s 256 -e trace=read,write /bin/echo 'a deliberately inspectable argument'

More output is not automatically more useful. Start narrow with a filter, and only add detail for the one call you are actually investigating.

8. Attach carefully, and know how to stop

To watch an already-running process, use -p PID. Find the PID from a trusted source, then interrupt strace with Ctrl-C once you have enough data:

$ pgrep -x sleep
12345
$ strace -p 12345 -o /tmp/sleep.trace
^C

Swap in the real PID for 12345. Attaching can require sudo and can affect the process itself: on some platforms, the current call may see a spurious EINTR, and a traced process always runs slower than usual. Do not attach to a production service during a latency-sensitive incident without an operational reason and a rollback plan.

For a command with descendants, Ctrl-C can leave those descendants running after strace itself exits. The manpage documents --kill-on-exit for command tracing, but it is deliberately left out of these examples: it sends SIGKILL to tracees if the tracer exits, and that is not something to add to a long-lived or stateful command without understanding it is irreversible.

Common traps

• strace runs the command directly, not through a shell. A script without a shebang can fail with ENOEXEC as a result; supply the shell explicitly instead, for example strace -o /tmp/script.trace sh ./script.
• A filtered trace is not the whole story. Trace only open and you can miss the actual variant in use, such as openat; prefer documented sets such as %file instead.
• Library calls are not system calls. A program can call a library function that eventually reaches a differently named kernel call, so read the arguments and return values right at the user/kernel boundary.
• Set-user-ID programs do not keep their effective privileges while traced. A clean trace here is not proof the untraced, security-sensitive execution behaves identically.