Go's http.TimeoutHandler: Why Your Handler Keeps Running After a 503

The bug report goes something like this: a client got a 503 "Timeout" from the API, retried, and now there are two invoices. Or two rows. Or one row and a confused customer. The server logs show the first request's handler finishing its work perfectly happily, a few seconds after the client had already been told it failed.

This is not a bug in http.TimeoutHandler. It is documented behaviour, just easy to read past. The name suggests "stop this handler after N seconds". What it actually does is "stop waiting for this handler after N seconds". Those are very different promises, and Go has no way to make the first one, because goroutines cannot be killed from the outside.

What the wrapper really does

Reading the source (I checked against Go 1.22.2, but this shape has been stable for a long time), ServeHTTP on the timeout handler does roughly this:

Wraps the request context with context.WithTimeout and swaps it into the request.
Starts your handler in a new goroutine, handing it a private timeoutWriter that buffers everything in memory.
Waits in a select for the handler to finish, a panic, or the context to end.
If the context ends first, writes a 503 to the real ResponseWriter and returns. Your goroutine is left alone.

Step 4 is the whole story. The 503 goes out, the connection carries on with its life, and your handler is still executing somewhere with a context that has been cancelled. Whether it notices depends entirely on whether anything in it looks at that context.

Watching it happen

Here is a small program with two handlers: one that respects the context, and one that does not.

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"time"
)

func aware(w http.ResponseWriter, r *http.Request) {
	select {
	case <-time.After(2 * time.Second):
		fmt.Println("aware: finished the work anyway")
	case <-r.Context().Done():
		fmt.Println("aware: cancelled:", r.Context().Err())
	}
	_, err := io.WriteString(w, "done")
	fmt.Println("aware: write error:", err)
}

func blind(w http.ResponseWriter, r *http.Request) {
	time.Sleep(500 * time.Millisecond)
	fmt.Println("blind: ran to the end, ctx err:", r.Context().Err())
	_, err := io.WriteString(w, "done")
	fmt.Println("blind: write error:", err)
}

func main() {
	for name, h := range map[string]http.HandlerFunc{"aware": aware, "blind": blind} {
		srv := httptest.NewServer(http.TimeoutHandler(h, 100*time.Millisecond, "too slow\n"))
		resp, err := http.Get(srv.URL)
		if err != nil {
			fmt.Println(err)
			continue
		}
		b, _ := io.ReadAll(resp.Body)
		fmt.Printf("%s: client saw %d %q\n", name, resp.StatusCode, b)
		time.Sleep(time.Second) // give stragglers time to print
		srv.Close()
	}
}

The output, in both cases, starts with the client seeing 503 "too slow\n" after 100ms. Then:

The aware handler prints cancelled: context deadline exceeded at about the 100ms mark, and its write returns http: Handler timeout.
The blind handler sleeps through the deadline, prints ran to the end, ctx err: context deadline exceeded at 500ms, and its write returns the same error.

That write error is http.ErrHandlerTimeout, and it is the only signal a handler that ignores the context ever gets. The response bytes go nowhere; they were only ever going into a buffer. But anything else the handler did (the database insert, the outgoing API call, the file on disk) happened for real.

The panic that vanishes

Actually, this bit is the one that made me put the source down and go and make tea. The wrapper does propagate panics from the handler goroutine, but only while it is still waiting. The channel that carries the panic has a buffer of one, and nobody reads from it once ServeHTTP has returned.

So if your handler panics after the timeout has fired, the goroutine's recover catches it, drops it into the buffered channel, and that is the end of it. I tested this with a handler that sleeps for 300ms behind a 100ms timeout and then panics. The client got its 503, the process carried on, and nothing was logged. No stack trace, no http: panic serving line. A panic before the timeout gets the normal treatment; the same panic a moment later evaporates.

If you rely on panic logs to find bugs in slow handlers, the slow handlers are precisely the ones that will hide them from you. Put your own recover with logging inside the wrapped handler if that matters.

Making the handler actually stop

The fix is the boring one: thread r.Context() through everything that can block, and check it before doing anything you cannot take back.

func createOrder(db *sql.DB) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		ctx := r.Context()

		tx, err := db.BeginTx(ctx, nil)
		if err != nil {
			http.Error(w, "begin failed", http.StatusInternalServerError)
			return
		}
		defer tx.Rollback()

		if _, err := tx.ExecContext(ctx, "INSERT INTO orders (item) VALUES ($1)", "widget"); err != nil {
			http.Error(w, "insert failed", http.StatusInternalServerError)
			return
		}

		// Last chance to back out before the point of no return.
		if err := ctx.Err(); err != nil {
			return
		}
		if err := tx.Commit(); err != nil {
			http.Error(w, "commit failed", http.StatusInternalServerError)
			return
		}
		w.WriteHeader(http.StatusCreated)
	}
}

Two things are doing the work here. database/sql honours the context on BeginTx and ExecContext, and cancelling the context passed to BeginTx rolls the transaction back on its own. The explicit ctx.Err() check just before Commit narrows the window where a client has been told "failed" while the row is committed. It narrows it; it does not close it, because the deadline can still fire between the check and the commit. There is no way to make "tell the client" and "commit" atomic across a timeout.

Which leads to the uncomfortable part. A 503 from this wrapper means "I stopped waiting", not "nothing happened". For anything with side effects, the client needs to be able to retry safely. That means an idempotency key on the request, checked server side, rather than hoping the timeout landed before the commit.

Other things it costs you

The whole response is buffered. The timeoutWriter holds the body in a bytes.Buffer and only copies it out when the handler finishes. A handler streaming a large download through this wrapper holds all of it in memory first.
No Flusher, no Hijacker. I checked with type assertions on the writer the handler receives: both come back false. Server-sent events and WebSockets behind a TimeoutHandler will not work. It does forward Pusher, for what that is worth in a world where HTTP/2 push has been all but abandoned.
It is a per-route wrapper, not a server setting. Apply it to the routes that need a ceiling; wrapping a mux that includes a streaming endpoint is how you discover the previous point in production.
The timeout is not WriteTimeout. Server.WriteTimeout is a deadline on the connection and will simply cut the socket; the client sees a reset, not a tidy 503. They solve different problems, and you often want both.

A quick way to audit a handler

For each blocking call in a handler, ask whether it takes a context, and whether that context descends from r.Context(). The usual offenders are http.Get (use http.NewRequestWithContext), exec.Command (use exec.CommandContext), time.Sleep in retry loops (use a select on ctx.Done()), and any goroutine the handler spawns with context.Background(). That last one is worth grepping for; it is the same trap that catches people with plain client disconnects, and TimeoutHandler just gives it a deadline to hide behind.

If a handler genuinely must finish once started (an audit write, say), then cancelling it is the wrong goal, and the honest design is to accept the request, do the work in a background job with its own lifecycle, and answer the client with a 202 and something to poll.