Blog / Go

998 words

  • go
  • net-http
  • server-sent-events
  • reverse-proxy
  • networking
  • debugging

Go SSE Behind a Proxy: Why Dead Clients Never Cancel the Context

The usual advice for a Go Server-Sent Events handler is to select on r.Context().Done() and return when it fires. On a laptop, hitting the server directly with curl, that works perfectly: Ctrl-C, the connection closes, the context is cancelled, the goroutine exits. Then you deploy it behind nginx or a cloud load balancer, and a few days later the goroutine count only goes up.

The reason is that the context is cancelled when your connection dies, and your connection is to the proxy, not to the browser. If a phone walks into a lift and never sends a FIN or RST, the proxy has no idea either. It keeps the upstream connection to you open, it keeps accepting your writes into a socket buffer, and your handler carries on happily writing to nobody.

What Go can and cannot see

For HTTP/1.x, the server does a background read on the connection while your handler runs. If that read returns EOF or an error, the request context is cancelled. For HTTP/2 it is the stream being reset or the connection closing. Both are events on the connection between Go and its immediate peer.

So there are three situations, and only the first is the easy one:

  • The client closes cleanly and the proxy notices, then closes the upstream side. Go sees EOF, the context fires. (nginx does this by default; proxy_ignore_client_abort is off.)
  • The client vanishes silently. The proxy learns nothing until its own TCP retransmissions give up, which on Linux with default settings can be a quarter of an hour or more. Until then, nothing reaches Go.
  • The proxy itself buffers your output, so writes succeed and the flush "works" without anything reaching the client at all.

That second case is the one that leaks. The proxy will eventually clean up, and when it does you get the context cancellation you wanted, just late. If you can live with "eventually", stop reading. If you hold per-client state, subscriptions, or database listeners, the delay matters.

Flusher.Flush swallows the error

Most tutorials call w.(http.Flusher).Flush(). That method returns nothing. If the write to the socket failed, you will not hear about it from there. http.NewResponseController(w).Flush() (Go 1.20 and later) returns an error, so use that and treat a non-nil result as "this client is gone".

The second half is the write deadline. If the proxy stops reading from you, your write blocks once the socket buffer fills, and a blocked write does not observe the context. Setting a short per-write deadline turns that into an error you can act on. It also sidesteps Server.WriteTimeout, which would otherwise kill a legitimately long stream: the deadline is reset on every event instead.

Heartbeats do two jobs

An SSE comment line (one starting with a colon) is ignored by clients and is the traditional keep-alive. It stops proxies with a 60 second idle timeout from cutting a quiet stream; nginx's proxy_read_timeout and the default idle timeout on AWS load balancers are both 60 seconds, though check your own proxy rather than trusting me.

The same write also exercises the connection, so a dead upstream shows up as a flush error. But there is a catch, and actually this bit is interesting: EventSource in the browser never surfaces comment lines to JavaScript. There is no event, no callback, nothing. So if you want the client to notice a stalled stream (which it should), the heartbeat has to be a real named event, not a comment. Both requirements are met by event: hb.

Making the server certain, not just eventually correct

Flush errors only tell you what the proxy tells you. To learn that the browser is gone, you need a signal from the browser. The cheapest one is a lease: the client POSTs a ping every 15 seconds on a separate request, the server records the time, and the stream handler drops any client that has missed three.

package main

import (
	"io"
	"net/http"
	"sync"
	"sync/atomic"
	"time"
)

type client struct {
	events chan string
	seen   atomic.Int64 // unix nanoseconds of last ping
}

func (c *client) touch()              { c.seen.Store(time.Now().UnixNano()) }
func (c *client) idle() time.Duration { return time.Since(time.Unix(0, c.seen.Load())) }

type hub struct {
	mu      sync.Mutex
	clients map[string]*client
}

func (h *hub) add(id string) *client {
	c := &client{events: make(chan string, 16)}
	c.touch()
	h.mu.Lock()
	h.clients[id] = c
	h.mu.Unlock()
	return c
}

func (h *hub) remove(id string, c *client) {
	h.mu.Lock()
	defer h.mu.Unlock()
	if h.clients[id] == c { // a reconnect may already have replaced us
		delete(h.clients, id)
	}
}

func (h *hub) ping(w http.ResponseWriter, r *http.Request) {
	h.mu.Lock()
	c := h.clients[r.URL.Query().Get("id")]
	h.mu.Unlock()
	if c == nil {
		http.Error(w, "unknown stream", http.StatusGone)
		return
	}
	c.touch()
	w.WriteHeader(http.StatusNoContent)
}

func (h *hub) stream(w http.ResponseWriter, r *http.Request) {
	id := r.URL.Query().Get("id")
	if id == "" {
		http.Error(w, "id required", http.StatusBadRequest)
		return
	}
	c := h.add(id)
	defer h.remove(id, c)

	rc := http.NewResponseController(w)
	send := func(frame string) error {
		if err := rc.SetWriteDeadline(time.Now().Add(10 * time.Second)); err != nil {
			return err
		}
		if _, err := io.WriteString(w, frame); err != nil {
			return err
		}
		return rc.Flush()
	}

	hdr := w.Header()
	hdr.Set("Content-Type", "text/event-stream")
	hdr.Set("Cache-Control", "no-cache")
	hdr.Set("X-Accel-Buffering", "no") // nginx: do not buffer this response

	if err := send("retry: 5000\n\n"); err != nil {
		return
	}
	tick := time.NewTicker(15 * time.Second)
	defer tick.Stop()
	for {
		select {
		case <-r.Context().Done():
			return
		case msg := <-c.events:
			if send("data: "+msg+"\n\n") != nil {
				return
			}
		case <-tick.C:
			if c.idle() > 45*time.Second {
				return // client stopped pinging: treat as dead
			}
			if send("event: hb\ndata: 1\n\n") != nil {
				return
			}
		}
	}
}

A few things in there are deliberate. msg must not contain a newline, or it will split the frame (JSON-encode it first). The channel is buffered and the publisher should use a non-blocking send, otherwise one stuck client stalls whoever is broadcasting. And remove compares pointers because the browser may already have reconnected under the same id before the old handler notices it is dead.

The X-Accel-Buffering: no header is nginx-specific; other proxies have their own switch, and some (older Apache mod_proxy setups, corporate appliances) buffer regardless. If events arrive in clumps of many at once, that is the first place to look.

The client half

The browser needs the mirror image: a watchdog that reconnects if no hb arrives, plus the ping. EventSource will reconnect on a clean error by itself, but a stalled connection that never errors just sits there.

const id = crypto.randomUUID();
let es, watchdog;

function arm() {
  clearTimeout(watchdog);
  watchdog = setTimeout(connect, 40000); // ~2.5 missed heartbeats
}

function connect() {
  if (es) es.close();
  es = new EventSource("/events?id=" + id);
  es.onopen = arm;
  es.onmessage = (e) => { arm(); render(e.data); };
  es.addEventListener("hb", arm);
  arm();
}

setInterval(() => {
  fetch("/ping?id=" + id, { method: "POST", keepalive: true })
    .then((r) => { if (r.status === 410) connect(); })
    .catch(() => {}); // offline: the watchdog deals with it
}, 15000);

connect();

Note the 410 from the ping handler: if the server has already dropped the lease (because of a deploy, say), the client finds out on its next ping rather than waiting for the watchdog.

Checking it actually works

The honest test is to break the network path rather than close the tab. Run the server behind nginx, open a stream, then block the client with iptables -j DROP on the client side (or pull a cable). Closing a tab sends a FIN and tells you nothing. Without the lease, the Go handler stays alive until the proxy gives up. With it, the handler should exit about 45 to 60 seconds after the last ping, and you can confirm that by logging the reason for every return from stream (context, flush error, lease expiry). Those three log lines are worth having in production too: the ratio between them says a lot about how your clients and your proxy actually behave.

One limit worth knowing: the lease proves the browser is alive, not that this particular stream is delivering. That is what the client watchdog is for, and why both halves exist.

← Back to the blog