Articles / Go

1034 words

  • go
  • net-http
  • expect-continue
  • uploads
  • http-transport
  • security

Rejecting an Upload Before It Arrives with Expect: 100-continue

A client with a bad token starts pushing 2 GiB at your upload endpoint, and you find out it was unauthorised after the last byte lands. You can return a 401 the moment the headers arrive, but by then the client is already mid-flight with the body.

HTTP/1.1 has a mechanism for exactly this, Expect: 100-continue, and Go's net/http supports both ends of it. It works in a slightly odd way, though, so it is worth seeing on the wire.

The client asks permission before sending the body

The exchange goes like this:

  1. The client sends the request headers, including Expect: 100-continue and a Content-Length, and holds the body back.
  2. The server answers with either an interim 100 Continue ("go on, send it") or a final status such as 401 or 413 ("don't bother").
  3. Only after a 100 does the client send the body.

Here is a raw TCP client talking to a Go server, once with no credentials and once with them. I sent the headers and never sent a body, to see what came back:

raw, bad auth:
  "HTTP/1.1 401 Unauthorized\r\n"
  "Content-Type: text/plain; charset=utf-8\r\n"
  "X-Content-Type-Options: nosniff\r\n"
  "Content-Length: 13\r\n"
  "Connection: close\r\n"
  "\r\n"
  "unauthorised\n"

raw, good auth:
  "HTTP/1.1 100 Continue\r\n"
  "\r\n"

(I trimmed the Date header. This was Go 1.22.2; the source references below are from that version too.)

The server sends the 100 when you first read the body

This is the bit that catches people: Go does not write 100 Continue when the headers are parsed. When the server sees the Expect header on an HTTP/1.1 request with a non-zero Content-Length, it wraps r.Body in an internal expectContinueReader.

The first call to Read on that wrapper writes the interim response, and then reads from the real body. So the handler decides the outcome, by what it touches first:

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
)

const maxUpload = 1 << 20 // 1 MiB

func upload(w http.ResponseWriter, r *http.Request) {
	// Nothing has touched r.Body yet, so no 100 has gone out.
	if r.Header.Get("Authorization") != "Bearer secret" {
		http.Error(w, "unauthorised", http.StatusUnauthorized)
		return
	}
	if r.ContentLength > maxUpload {
		http.Error(w, "too large", http.StatusRequestEntityTooLarge)
		return
	}

	r.Body = http.MaxBytesReader(w, r.Body, maxUpload)
	n, err := io.Copy(io.Discard, r.Body) // first Read: 100 Continue goes out here
	if err != nil {
		http.Error(w, "bad upload", http.StatusBadRequest)
		return
	}
	fmt.Fprintf(w, "got %d bytes\n", n)
}

func main() {
	http.HandleFunc("POST /upload", upload)
	log.Fatal(http.ListenAndServe(":8080", nil))
}

Every check before the io.Copy is free: no 100, no body on the wire. If you call http.Error or WriteHeader first, Go also cancels the pending automatic 100, so the client only ever sees the final status.

Anything that reads early spoils it

The Read is what triggers the 100, so any code that reads the body before your checks have run defeats the whole exercise. The usual suspects:

  • a logging or audit middleware that peeks at the body;
  • r.ParseForm(), r.FormValue() and r.ParseMultipartForm(), which all read the body for POST requests;
  • a JSON decoder that runs before the auth check "because it's only the request struct".

Put authentication, routing decisions and the Content-Length check in middleware that runs strictly before anything reads the body. That is the whole trick, and it is entirely about ordering.

A rejection closes the connection, on purpose

Look again at the 401 above: Connection: close. If the handler writes its headers and the server has not seen the request body reach EOF, Go marks the connection to close after the reply.

There is a comment in server.go explaining the reasoning: if the client is offering a large body you do not intend to use, it is better to drop the connection than to read and discard it.

That is a trade. You save the upload, and you pay for a fresh TCP (and probably TLS) handshake on the client's next request. For a 2 GiB body that is a bargain. For a 4 KiB body it is a slight loss, which is a fair reason not to bother with the header for small payloads.

The client relies on that close

Actually, this bit is interesting: the client side depends on that Connection: close. In Transport, when a final response arrives while the client is waiting, it skips the body only if the response is marked to close the connection (resp.Close).

If the server rejects but keeps the connection alive, the Go client sends the body anyway, because it needs to keep the stream in step. So "reject early" is a joint effort: a server that answers 401 and leaves the connection open may still receive the whole upload.

The client half: you set the header yourself

Go's client never adds Expect for you. You set it, and you need a transport with a non-zero ExpectContinueTimeout:

  • http.DefaultTransport has one second.
  • A zero-value &http.Transport{} has none.
req, err := http.NewRequest(http.MethodPost, url, body)
if err != nil {
	return err
}
req.ContentLength = size
req.Header.Set("Expect", "100-continue")

resp, err := http.DefaultClient.Do(req)

I ran this against the handler above, with a 5 MiB body and no token. The wrapped reader reported 0 bytes read, the status was 401 and the whole call took about half a millisecond. The body was never touched.

The timeout costs a second against servers that ignore it

The timeout is how long the client waits for any response before giving up and sending the body regardless, because plenty of servers and intermediaries ignore Expect.

I pointed a client at a raw listener that never sent a 100 and only replied after receiving the full 100 KiB body:

  • With ExpectContinueTimeout: time.Second the request took one second, then succeeded.
  • With the timeout left at zero it finished at once, because a zero timeout means "send the body immediately, don't wait for approval" and the header becomes decorative.

So the price of using Expect: 100-continue against something that does not understand it is a one second stall on every upload. Know that before you turn it on for a client talking through an unknown proxy.

Quick aside: curl does something similar on its own for larger uploads, which is the origin of many "why is my POST exactly one second slow" tickets.

It is a courtesy, not a size limit

Content-Length is a claim by the client, not a fact. Two gaps follow:

  • A chunked upload has no length at all (r.ContentLength is -1), so the "too large" check above passes and you are back to needing http.MaxBytesReader to cap what you actually read.
  • A client that does not send Expect at all gets no benefit: it will be halfway through the body before your handler runs.

Treat early rejection as a courtesy that saves bandwidth for well-behaved clients, and the size limit as the actual defence.

Everything above is HTTP/1.1. HTTP/2 has its own stream-level ways of refusing a body, so I have kept this to the version where the behaviour is visible on the wire in plain text.

← All articles