Blog / Go

950 words

  • go
  • net-http
  • connstate
  • connection-tracking
  • observability
  • debugging

Go's http.Server ConnState: Watching Connections Change State

Most Go HTTP instrumentation lives in middleware, which means it only sees requests. That is fine until the question you actually have is about connections: how many clients are sitting there holding a socket open and saying nothing? Which ones are keep-alive connections doing nothing? Why does the "open connections" number on a dashboard never come back down after a WebSocket upgrade?

Middleware cannot answer any of that, because a connection that never sends a complete request never reaches your handler. The hook that can is http.Server.ConnState, and it is smaller and stranger than it first looks.

The five states

The field is a plain callback: ConnState func(net.Conn, http.ConnState). The server calls it whenever a connection moves between these states:

  • StateNew: accepted, expected to send a request straight away.
  • StateActive: has read at least one byte of a request.
  • StateIdle: finished a request, now waiting in keep-alive for the next one.
  • StateHijacked: a handler called Hijack(). Terminal.
  • StateClosed: closed. Also terminal.

The normal life of an HTTP/1.1 connection is new, active, idle, active, idle, and so on, ending in closed. A connection that connects and then says nothing goes new, closed (once a timeout or the client gives up). Simple enough. The interesting bits are in the details.

A tracker that keeps a snapshot

Counters are the obvious thing to build, but I prefer a map of connection to (state, since when), because it lets you ask "which connections have been stuck for a while" rather than only "how many". Here is a version I ran on Go 1.22:

type tracker struct {
	mu    sync.Mutex
	state map[net.Conn]http.ConnState
	since map[net.Conn]time.Time
}

func newTracker() *tracker {
	return &tracker{
		state: make(map[net.Conn]http.ConnState),
		since: make(map[net.Conn]time.Time),
	}
}

func (t *tracker) hook(c net.Conn, s http.ConnState) {
	t.mu.Lock()
	defer t.mu.Unlock()
	switch s {
	case http.StateClosed, http.StateHijacked:
		delete(t.state, c)
		delete(t.since, c)
	default:
		t.state[c] = s
		t.since[c] = time.Now()
	}
}

func (t *tracker) stuckNew(d time.Duration) []string {
	t.mu.Lock()
	defer t.mu.Unlock()
	var addrs []string
	for c, s := range t.state {
		if s == http.StateNew && time.Since(t.since[c]) > d {
			addrs = append(addrs, c.RemoteAddr().String())
		}
	}
	return addrs
}

Wire it in with srv := &http.Server{Handler: mux, ConnState: t.hook}. A net.Conn is an interface holding a pointer, so it is a perfectly good map key.

To try it, I started a server and opened three raw TCP connections: one that sends nothing, one that sends a normal GET /, and one that requests a handler which hijacks the socket. With a print statement in the hook, the output was:

hook: 127.0.0.1:41992 new
hook: 127.0.0.1:41996 new
hook: 127.0.0.1:42000 new
hook: 127.0.0.1:42000 active
hook: 127.0.0.1:42000 hijacked
hook: 127.0.0.1:41996 active
hook: 127.0.0.1:41996 idle
counts: map[new:1 idle:1] stuck: [127.0.0.1:41992]

There is the silent client, still in new, and stuckNew names it. That is the connection-level view of the slow-client problem: it is invisible to request logging, and it shows up here immediately. (If you have not set ReadHeaderTimeout, this is also how you will discover you needed to.)

Hijacked connections never close

Look at the hijacked connection in that output: active, then hijacked, and nothing after. The documentation is explicit that hijacked connections do not transition to StateClosed. The server has handed the socket to your code and no longer knows when you close it.

So the classic bug goes like this. You increment an "open connections" gauge on StateNew and decrement it on StateClosed. Every WebSocket upgrade increments it and never decrements it, and the graph creeps upwards until someone decides you have a leak. You do not; you have a gauge that is wrong. Treat StateHijacked as a decrement too (as the tracker above does), and if you still care about the hijacked sockets, count them separately and let the code that owns them report when they end.

Which goroutine calls you

Actually, this bit is interesting. Reading net/http/server.go, the StateNew hook is called from the accept loop itself, just before the per-connection goroutine is started:

c := srv.newConn(rw)
c.setState(c.rwc, StateNew, runHooks) // before Serve can return
go c.serve(connCtx)

The later transitions happen on the connection's own goroutine, but StateNew runs inline in the loop that accepts sockets. A hook that takes a lock held by a slow reader, does a network call, or writes to a blocked log pipe will therefore stall the acceptance of every new connection, not just its own. Keep the hook to a mutex, a map write and nothing else. If you want to ship the events somewhere, push them onto a buffered channel with a non-blocking send and drop on overflow; losing a metric is better than losing the listener.

The same logic applies to the lock in my tracker. It is held for a map write, so it is fine, but a stuckNew scan over a very large map holds it for longer. With tens of thousands of connections, copy the entries out under the lock and do the filtering afterwards, or keep per-state counters as well.

What it cannot tell you

The hook describes connections, not requests, and the documentation says as much. For HTTP/1.x the two line up closely enough: active fires before the request reaches a handler, and the connection returns to idle once the handler finishes. For HTTP/2, one TCP connection carries many concurrent requests, so StateActive fires when the count goes from zero to one active request and it only moves away once every request has completed. A connection with one long-lived stream and a hundred short ones is simply "active" the whole time.

So do not use ConnState for per-request timing or per-request counting; that is what middleware is for. Use it for what only it can see:

  • connections that never became a request (stuck in new);
  • how many keep-alive connections are parked in idle, which is a decent proxy for whether your IdleTimeout is sensible;
  • churn: a high rate of new to closed with few requests per connection usually means clients that are not reusing connections;
  • hijack counts, so upgraded connections stop being an unexplained hole in your numbers.

One more small thing: the net.Conn you receive is what the server holds, so with ListenAndServeTLS it is a *tls.Conn. You can type-assert it, but the handshake happens after StateNew, so ConnectionState() will not be useful until later transitions. For anything TLS-specific at connection time, VerifyConnection is the better place.

← Back to the blog