Articles / Go

  • go
  • strace
  • linux
  • syscalls
  • runtime
  • debugging

Why strace Shows Syscalls Your Go Program Never Made

Run a Go program that only prints one line under strace and you get a wall of output: clone, futex, epoll_create1, dozens of rt_sigaction calls. Your code made exactly one write. The rest came from the Go runtime, which is doing its own housekeeping before and around main.

That is normal, but it is confusing the first time, especially if you are hunting a real bug. So here is a field guide to what is yours and what is the runtime's.

Start with the smallest possible program

package main

import "fmt"

func main() {
	fmt.Println("hello")
}

Build it and trace it. The -f flag matters: Go runs goroutines on several OS threads, and without -f you only see the first one.

go build -o hello .
strace -f -o trace.txt ./hello
strace -f -c ./hello

The second command prints a summary table of counts instead of a log. It is the quickest way to see that futex and rt_sigaction dominate, and your write is a single line near the bottom.

What the runtime does on its own

Roughly in the order you will meet them:

  • execve, then arch_prctl and sched_getaffinity: the process starts and the runtime asks how many CPUs it may use, which sets the default GOMAXPROCS.
  • mmap and madvise: the runtime reserves address space for its heap arenas and later hands memory back to the kernel.
  • rt_sigaction in a long run, plus sigaltstack: the runtime installs handlers for essentially every signal so it can turn crashes into panics and traces.
  • clone: new OS threads, for sysmon (a background monitor) and for goroutines that need to run in parallel or block in syscalls.
  • futex: threads parking and waking each other. This is the runtime's scheduler talking to itself.
  • epoll_create1, eventfd2, epoll_ctl, epoll_pwait: the network poller, set up lazily when something first needs it.

None of these is a bug. If you see futex thousands of times in a program that is idle, that is threads going to sleep and waking, not your code spinning.

The signal you never sent

Quick detour, because this one looks alarming. A busy Go program under strace -f can show lines like --- SIGURG ... ---, followed by rt_sigreturn, and sometimes a tgkill call from another thread.

That is asynchronous preemption. If a goroutine hogs a thread for too long, the runtime sends SIGURG to that thread with tgkill so the scheduler can take control back. SIGURG was picked because it is rarely used for anything else. Nothing is wrong; you are watching the scheduler be fair.

The epoll_ctl that fails on purpose

Open an ordinary file with os.Open and you will see something like this:

openat(AT_FDCWD, "data.txt", O_RDONLY|O_CLOEXEC) = 3
epoll_ctl(4, EPOLL_CTL_ADD, 3, {...}) = -1 EPERM (Operation not permitted)

The os package tries to register every file with the poller, in case it is a pipe, socket or terminal that supports non-blocking I/O. Regular files cannot be polled, so the kernel says EPERM, and Go quietly falls back to ordinary blocking reads. That failed call is expected, not an error in your program.

The syscall that is missing

The reverse also happens. Call time.Now() in a loop and you will not see a clock_gettime syscall at all. Go uses the vDSO, a small chunk of kernel-provided code mapped into your process, which reads the clock without entering the kernel. strace only sees real syscalls, so it never sees these.

Filtering down to what you wrote

When you are debugging, ask strace for only the calls you care about:

strace -f -e trace=file ./hello
strace -f -e trace=network ./server
strace -f -e trace=openat,read,write ./hello
strace -f -e trace='!futex,rt_sigaction,rt_sigprocmask' ./hello

The trace=file and trace=network groups are the most useful. The last form inverts the list, which is handy when you want everything except the scheduler noise. Quote the ! so your shell does not treat it as history expansion.

Use -ff -o out instead of -f if you want one file per thread, named by thread ID. Go moves goroutines between threads, so a single request can hop across several files, but it keeps the futex chatter of each thread apart.

Telling runtime calls from yours

  1. Look at the arguments. A write(1, "hello\n", 6) carries your data; a futex(0xc000..., FUTEX_WAIT_PRIVATE, ...) does not.
  2. Check the timing. Runtime setup clusters at the very start and at exit (exit_group); yours sit in between.
  3. Use strace -k if your strace was built with stack-trace support, to print the user-space stack for each call. It shows whether the call came from runtime. frames or from yours.
  4. Compare against the summary from -c. A count that grows with load is yours; a fixed pile at startup is not.

One caveat: exact call names and counts vary by Go version, kernel and architecture. The runtime has swapped pipes for eventfd in the poller before, for example, so do not treat any particular trace as a fixed list.