Why HTTP Servers Ignore Unknown Headers, and When That Bites

Send a server Cache-Controll: no-store and you get a perfectly happy 200. No warning, no error, and no caching rule either. That silence is deliberate: HTTP is built so that a recipient ignores header fields it does not understand. It is also the reason a whole family of bugs and security holes exists.

The rule, and why it exists

RFC 9110 (HTTP Semantics) treats extensibility as a core design goal. New fields can be invented at any time, and older software has to cope. In practice that means two behaviours: an endpoint ignores fields it does not recognise, and a proxy forwards them unless told otherwise. I have not quoted the exact keyword strength (MUST or SHOULD) here, so check the RFC text itself before citing it.

Consider the alternative. If every server rejected unknown headers, nobody could deploy a new client hint or Sec-Fetch-Site until every server on earth had upgraded. Ignoring is what lets the web evolve without a flag day.

Bite one: the typo that does nothing

The same tolerance makes your mistakes invisible. Common offenders:

• Content-Security-Polcy: no policy applied, page looks fine.
• Strict-Transport-Securty: no HSTS, nothing complains.
• Authorisation: a British typist's classic; the credentials never arrive.
• Cache-Controll: your caching rule silently does not exist.

Security headers are the nasty case, because the failure mode is "everything works, you are just unprotected". A test that only checks the status code will never notice.

Catching them in development

Since the protocol will not tell you, make your own code do it. A small Go middleware can log any request header outside an allowlist while you develop. This is deliberately simplified: it is a debugging aid, not a production filter.

package main

import (
	"log"
	"net/http"
)

func warnUnknown(known map[string]bool, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		for name := range r.Header {
			if !known[http.CanonicalHeaderKey(name)] {
				log.Printf("unknown request header %q on %s", name, r.URL.Path)
			}
		}
		next.ServeHTTP(w, r)
	})
}

func main() {
	known := map[string]bool{
		"Accept": true, "Authorization": true, "User-Agent": true,
		"Content-Type": true, "Content-Length": true,
	}
	mux := http.NewServeMux()
	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("ok\n"))
	})
	log.Fatal(http.ListenAndServe(":8080", warnUnknown(known, mux)))
}

For response headers, the equivalent is a test that fetches your page and asserts the exact header names you expect are present. Checking for the typo'd name's absence is not enough.

Bite two: two parsers, two opinions

Ignoring is only safe if everyone ignores the same things. Request smuggling lives in the gap. If a front-end proxy does not recognise a mangled Transfer-Encoding value and treats it as just another header, while the back end does honour it, they disagree on where the request body ends. Same bytes, two interpretations.

The defence is to refuse ambiguous input rather than ignore it. A malformed framing header should earn a 400, not a shrug. Modern servers mostly do this for the headers that control message framing; unknown headers are a different matter, but the principle is the same.

Bite three: the underscore trap

Quick detour, because this one is oddly satisfying. CGI-style back ends turn header names into environment variables by upper-casing them and swapping hyphens for underscores. So X-User-Id and X_User_Id both become HTTP_X_USER_ID. Which one wins depends on the server.

That is an auth bypass waiting to happen if your proxy sets X-User-Id and a client sends the underscore version. Nginx guards against it: by default it ignores header names containing underscores (the underscores_in_headers directive turns them back on). Ignoring unknown things, here, is the safe choice.

Bite four: forwarding is also a rule

A proxy forwards unknown headers, but with one exception that matters. Any field named in the Connection header is hop-by-hop and must be removed before forwarding. So a client can send Connection: X-Forwarded-For and some proxies will strip a header they added themselves, or one a later stage relies on.

Related and more common: client-supplied X-Forwarded-For or X-Real-IP reaching your app unchanged, because the proxy dutifully forwarded an "unknown" field. Your app must only trust such headers when they were set by a proxy you control, and the proxy should overwrite them rather than append blindly.

What to do about it

1. Test security headers by exact name against a real response, in CI.
2. Have the edge proxy strip or overwrite any internal header (X-User-Id and friends) that clients have no business setting.
3. Reject malformed framing headers outright; never "be liberal" with them.
4. Keep underscores out of your own header names.

Ignoring unknown headers is the right default for a protocol that has to last decades. It just means that the protocol will never tell you when you got a name wrong, so you have to.