Articles / Systems Programming

976 words

  • ebpf
  • go
  • linux
  • tcp
  • tracing
  • networking

eBPF for the Curious: Tracing TCP Connections From Go Without tcpdump

A service I run kept opening connections I couldn't account for. Not malicious, just unexplained: a dependency doing something at startup that wasn't in any log.

The instinct is to reach for tcpdump, but the box was a locked-down container with no packet capture tooling installed, no easy way to add it, and honestly more capability than I needed. I didn't want packets. I wanted a stream of "who connected to what, and when the state changed" that I could parse in Go and feed into a metric. That's a small eBPF program away.

This is a walkthrough of hooking a kernel tracepoint that fires on every TCP state transition, and reading the results straight into a Go process using cilium/ebpf, without shelling out to anything.

A tracepoint is a stable hook; a kprobe is not

There are two common places to hook TCP activity in eBPF: kprobes attached to internal functions like tcp_v4_connect, and the sock:inet_sock_set_state tracepoint.

Kprobes attach to whatever a given kernel version happens to name its internal functions. That is not a stable interface and can change (or get inlined away) between point releases. Tracepoints are a maintained, stable ABI: the kernel developers already committed to keeping the format of inet_sock_set_state steady because tools depend on it.

That tracepoint fires every time a TCP socket changes state: SYN_SENT, ESTABLISHED, CLOSE_WAIT, and so on. It covers both outbound connections your process initiates and inbound ones it accepts. It's the same hook the BCC tcplife and tcpstates tools use. One hook point, both directions, no missed events from only watching connect().

The eBPF program is restricted C, checked by the verifier

eBPF programs are written in a restricted subset of C, compiled to BPF bytecode with clang, and loaded into the kernel. There a verifier checks they can't loop forever, read out of bounds, or crash anything.

Field access into kernel structs is done via CO-RE (Compile Once, Run Everywhere), which resolves struct offsets against the running kernel's BTF at load time rather than baking them in at compile time. That's what lets a single compiled object run unmodified across kernel versions, instead of BCC's older approach of recompiling on the target machine.

You need a vmlinux.h describing your kernel's types, generated once with:

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

The program itself, trace_tcp.c:

//go:build ignore

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

char __license[] SEC("license") = "GPL";

struct event {
	__u32 saddr;
	__u32 daddr;
	__u16 sport;
	__u16 dport;
	int oldstate;
	int newstate;
};

struct {
	__uint(type, BPF_MAP_TYPE_RINGBUF);
	__uint(max_entries, 256 * 1024);
} events SEC(".maps");

SEC("tracepoint/sock/inet_sock_set_state")
int trace_tcp_state(struct trace_event_raw_inet_sock_set_state *ctx)
{
	if (ctx->protocol != IPPROTO_TCP || ctx->family != AF_INET)
		return 0;

	struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
	if (!e)
		return 0;

	e->saddr = *(__u32 *)&ctx->saddr[0];
	e->daddr = *(__u32 *)&ctx->daddr[0];
	e->sport = ctx->sport;
	e->dport = ctx->dport;
	e->oldstate = ctx->oldstate;
	e->newstate = ctx->newstate;

	bpf_ringbuf_submit(e, 0);
	return 0;
}

The ring buffer map is the modern replacement for perf buffers. Compared with those, you get:

  • one shared buffer instead of one per CPU;
  • no lost-event bookkeeping;
  • a plain bpf_ringbuf_reserve / bpf_ringbuf_submit pair instead of copying into a separate output map.

It's been available since kernel 5.8, which by now covers most things you'd be deploying to.

Note the mandatory null check after bpf_ringbuf_reserve. The verifier will reject the program outright without it, since it has no way to prove the pointer is valid otherwise. This isn't defensive style, it's a hard requirement.

bpf2go turns the C file into Go bindings

cilium/ebpf ships a code generator, bpf2go, that compiles the C file and produces Go bindings for the maps and programs inside it:

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cc clang -cflags "-O2 -g" TcpTrace trace_tcp.c -- -I./headers

Running go generate gives you a tcptrace_bpfel.go with a tcpTraceObjects struct holding a typed handle to TraceTcpState and Events. Loading and attaching it:

package main

import (
	"bytes"
	"encoding/binary"
	"errors"
	"log"
	"net"

	"github.com/cilium/ebpf/link"
	"github.com/cilium/ebpf/ringbuf"
	"github.com/cilium/ebpf/rlimit"
)

type event struct {
	Saddr    uint32
	Daddr    uint32
	Sport    uint16
	Dport    uint16
	Oldstate int32
	Newstate int32
}

var tcpStates = map[int32]string{
	1: "ESTABLISHED", 2: "SYN_SENT", 3: "SYN_RECV", 4: "FIN_WAIT1",
	5: "FIN_WAIT2", 6: "TIME_WAIT", 7: "CLOSE", 8: "CLOSE_WAIT",
	9: "LAST_ACK", 10: "LISTEN", 11: "CLOSING",
}

func main() {
	if err := rlimit.RemoveMemlock(); err != nil {
		log.Fatal(err)
	}

	var objs tcpTraceObjects
	if err := loadTcpTraceObjects(&objs, nil); err != nil {
		log.Fatalf("loading objects: %v", err)
	}
	defer objs.Close()

	tp, err := link.Tracepoint("sock", "inet_sock_set_state", objs.TraceTcpState, nil)
	if err != nil {
		log.Fatalf("attaching tracepoint: %v", err)
	}
	defer tp.Close()

	rd, err := ringbuf.NewReader(objs.Events)
	if err != nil {
		log.Fatalf("opening ringbuf reader: %v", err)
	}
	defer rd.Close()

	log.Println("tracing TCP state transitions, Ctrl-C to stop")

	var e event
	for {
		record, err := rd.Read()
		if err != nil {
			if errors.Is(err, ringbuf.ErrClosed) {
				return
			}
			log.Printf("reading ringbuf: %v", err)
			continue
		}

		if err := binary.Read(bytes.NewReader(record.RawSample), binary.LittleEndian, &e); err != nil {
			log.Printf("parsing event: %v", err)
			continue
		}

		log.Printf("%s:%d -> %s:%d  %s -> %s",
			addrToIP(e.Saddr), e.Sport,
			addrToIP(e.Daddr), e.Dport,
			tcpStates[e.Oldstate], tcpStates[e.Newstate])
	}
}

func addrToIP(addr uint32) net.IP {
	ip := make(net.IP, 4)
	binary.LittleEndian.PutUint32(ip, addr)
	return ip
}

No CGO_ENABLED=1, no linking against libbcc, no on-target compiler. The resulting binary is a static Go executable that happens to carry a small BPF object embedded in it.

Hang on, why do the addresses come out right?

This bit tripped me up the first time. The kernel's saddr field in the tracepoint is a raw 4-byte array in network byte order, the way an IPv4 address is normally written (192, 168, 1, 1 as consecutive bytes).

The C code above reinterprets those four bytes as a __u32 by pointer cast. That does no swapping: it just takes whatever bytes are in memory and reads them as an integer using the machine's native endianness. On x86 and arm64, that's little-endian, so the numeric value ends up as if you'd called binary.LittleEndian.Uint32 on the raw bytes.

The fix isn't to "correct" the byte order somewhere, it's to be consistent. Since the C side effectively produced a little-endian encoding of the raw bytes, decoding the struct with binary.LittleEndian in Go and then writing it back out with binary.LittleEndian.PutUint32 reproduces the original byte sequence exactly.

Two wrongs cancelling out, but deterministically so, not a coincidence. If you used binary.BigEndian anywhere in that chain you'd get every address reversed. It would look plausible enough (still four valid-looking octets) that you might not notice for a while.

It needs capabilities, and it sees the whole host

Loading BPF programs needs CAP_BPF plus CAP_PERFMON on kernels 5.8 and later (pre-5.8, it's the coarser CAP_SYS_ADMIN).

Attaching to a tracepoint is inherently system-wide: it sees every socket on the host, not just your container's. So this genuinely can't be scoped down to "just my process" the way a syscall filter can.

In a container you'll need:

  • those capabilities added explicitly;
  • /sys/kernel/debug and /sys/kernel/tracing available;
  • a kernel with CONFIG_DEBUG_INFO_BTF=y, so the CO-RE relocations have anything to resolve against.

Most modern distribution kernels ship BTF by default. Stripped-down custom kernels sometimes don't, and you'll get a clear load-time error rather than a silent failure.

Lifecycle events, not packets

No payload, no packet-level detail, nothing about retransmits or window sizes. If you need that, you still want tcpdump or a proper packet capture.

What you get instead is a near-zero-overhead stream of connection lifecycle events, in-process, structured as Go values from the moment they leave the kernel. For "what is this thing actually connecting to", that turned out to be exactly the shape of answer I needed, and a lot less invasive than handing a container NET_RAW and a copy of libpcap.

← All articles