Blog / Go

800 words

  • go
  • net-http
  • graceful-shutdown
  • websockets
  • http-hijacker
  • debugging

Go's http.Server Shutdown Ignores Hijacked and WebSocket Connections

The bug report usually reads: "we do graceful shutdown, but every deploy chops our WebSocket clients off mid-message". The code looks right. There is a signal handler, a call to srv.Shutdown(ctx), a nice 30 second timeout. And yet the sockets die instantly.

The title of this post is slightly unfair to Go, because Shutdown does not drop those connections. It does something arguably worse: it does not know they exist. What drops them is your process exiting a few microseconds later.

What the docs actually say

The net/http documentation for Server.Shutdown says it "does not attempt to close nor wait for hijacked connections such as WebSockets", and that the caller should notify such connections separately and wait for them if desired. Server.Close carries the same warning. It is one sentence, and it is easy to read past.

The mechanism is straightforward. The server keeps a map of connections it is serving, each with a state: new, active, idle, hijacked, closed. Shutdown closes the listeners, then loops, closing idle connections and checking whether anything is still active. It polls, starting at 1ms and backing off to 500ms. When a handler calls Hijack, the connection is set to StateHijacked and removed from that map. From then on the server has forgotten it. The handler owns the raw net.Conn, and nothing in http.Server can see it, count it, or wait on it.

So the sequence on a SIGTERM is:

  1. Shutdown closes the listener.
  2. It finds zero tracked connections (your WebSockets are not tracked).
  3. It returns nil immediately.
  4. main returns and the process exits, taking every goroutine and socket with it.

The kernel then closes those sockets, so clients see a reset or an abrupt EOF rather than a proper close frame. That is exactly the "graceful shutdown that isn't" symptom.

Libraries do not save you here. Gorilla, coder/websocket and friends all call Hijack (or the ResponseController equivalent) inside the upgrade. Once the handshake completes, the connection is theirs and http.Server is out of the picture.

A note on the request context

You might hope the request context gets cancelled on shutdown. It does not, for ordinary active requests either: Shutdown waits for them, it does not cancel them. If you want handlers to notice shutdown, you have to arrange that yourself, typically by deriving BaseContext from a context you cancel. For hijacked connections that still leaves the waiting part, because nothing waits for a goroutine you spawned after hijacking.

The fix: track them yourself

You need three things: a registry of hijacked connections, a way to tell each one to wind down, and a WaitGroup so you can wait. Here is a deliberately plain version using a raw hijack and a line-based protocol, so it needs no third-party packages. Swap the inner loop for your WebSocket library's read loop and the structure is the same.

package main

import (
	"context"
	"errors"
	"log"
	"net"
	"net/http"
	"os/signal"
	"sync"
	"syscall"
	"time"
)

type hub struct {
	mu      sync.Mutex
	wg      sync.WaitGroup
	conns   map[net.Conn]struct{}
	closing bool
}

func newHub() *hub { return &hub{conns: make(map[net.Conn]struct{})} }

// add refuses new connections once shutdown has begun, which also
// keeps wg.Add from racing with wg.Wait.
func (h *hub) add(c net.Conn) bool {
	h.mu.Lock()
	defer h.mu.Unlock()
	if h.closing {
		return false
	}
	h.conns[c] = struct{}{}
	h.wg.Add(1)
	return true
}

func (h *hub) remove(c net.Conn) {
	h.mu.Lock()
	delete(h.conns, c)
	h.mu.Unlock()
	h.wg.Done()
}

// shutdown pokes every reader awake, then waits for them to finish
// or for ctx to expire, at which point it closes whatever is left.
func (h *hub) shutdown(ctx context.Context) error {
	h.mu.Lock()
	h.closing = true
	for c := range h.conns {
		c.SetReadDeadline(time.Now()) // unblocks a pending Read
	}
	h.mu.Unlock()

	done := make(chan struct{})
	go func() { h.wg.Wait(); close(done) }()

	select {
	case <-done:
		return nil
	case <-ctx.Done():
		h.mu.Lock()
		for c := range h.conns {
			c.Close()
		}
		h.mu.Unlock()
		return ctx.Err()
	}
}

func (h *hub) handle(w http.ResponseWriter, r *http.Request) {
	conn, brw, err := http.NewResponseController(w).Hijack()
	if err != nil {
		http.Error(w, "cannot hijack", http.StatusInternalServerError)
		return
	}
	defer conn.Close()

	if !h.add(conn) {
		brw.WriteString("HTTP/1.1 503 Service Unavailable\r\nConnection: close\r\n\r\n")
		brw.Flush()
		return
	}
	defer h.remove(conn)

	brw.WriteString("HTTP/1.1 101 Switching Protocols\r\nUpgrade: echo\r\nConnection: Upgrade\r\n\r\n")
	brw.Flush()

	for {
		line, err := brw.ReadString('\n')
		if err != nil {
			var ne net.Error
			if errors.As(err, &ne) && ne.Timeout() {
				brw.WriteString("server going away\n")
				brw.Flush()
			}
			return
		}
		brw.WriteString(line)
		brw.Flush()
	}
}

func main() {
	h := newHub()
	mux := http.NewServeMux()
	mux.HandleFunc("/echo", h.handle)
	srv := &http.Server{Addr: ":8080", Handler: mux}

	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
	defer stop()

	go func() {
		if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
			log.Fatal(err)
		}
	}()

	<-ctx.Done()

	sctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
	defer cancel()
	if err := srv.Shutdown(sctx); err != nil {
		log.Printf("http shutdown: %v", err)
	}
	if err := h.shutdown(sctx); err != nil {
		log.Printf("hijacked conns still open: %v", err)
	}
}

Why the ordering matters

Calling srv.Shutdown first is deliberate. It closes the listeners and waits for in-flight requests, and an in-flight request is precisely the thing that might be about to call Hijack. Once it returns, no new hijack can start, so the hub's set of connections is complete when you tell it to drain.

The read-deadline trick is the tidy bit. Setting a deadline in the past makes a blocked Read return a timeout error straight away, without closing the socket, which leaves you free to write a goodbye message first. With a real WebSocket library that is where you would send a close frame (status 1001, "going away") and wait briefly for the peer to answer. Most libraries expose a Close method that does that handshake for you.

Actually, one more wrinkle: Server.RegisterOnShutdown exists for exactly this and fires your function when Shutdown starts. Its callbacks run in their own goroutines and Shutdown does not wait for them. So it is a good place to kick off the "tell clients to go away" step early, but it does not replace the WaitGroup. I kept the explicit sequence above because it is easier to reason about.

Checking it works

Run the server, connect with nc localhost 8080, and send GET /echo HTTP/1.1, a Host: header and a blank line. You get the 101 back, then anything you type is echoed. Send SIGTERM from another terminal and the client should receive "server going away" and a clean EOF. Remove the h.shutdown call and repeat: the process exits instantly and nc just stops.

Two deployment details worth remembering. In Kubernetes the pod gets SIGTERM, then SIGKILL after terminationGracePeriodSeconds (30 seconds by default), so your drain timeout must be shorter than that. And clients need to reconnect sensibly, ideally with jitter, or a rolling deploy just moves the stampede from one pod to the next.

← Back to the blog