Blog / Go

924 words

  • go
  • net-http
  • reverse-proxy
  • x-forwarded-for
  • netip
  • security

Why Your Go Server Logs 127.0.0.1 for Every Client Behind a Proxy

You put nginx (or Caddy, or a load balancer) in front of your Go service, and every log line now says the visitor was 127.0.0.1. Or 10.0.3.7. Rate limits hit everyone at once, and your abuse reports name your own proxy.

Nothing is broken. r.RemoteAddr is the address of the TCP peer, and the TCP peer is now the proxy. The real client address lives in a header, and that header is where the interesting mistakes happen.

What RemoteAddr actually holds

The net/http server fills r.RemoteAddr from the connection's remote address, as an "IP:port" string. It knows nothing about proxies. The kernel told it who connected, and that was the proxy on loopback.

Proxies pass the original address along in a header. The de facto standard is X-Forwarded-For, a comma-separated list. There is also the standardised Forwarded header from RFC 7239, which looks like for=192.0.2.60;proto=https, with IPv6 addresses quoted and in brackets. In practice, most deployments still send X-Forwarded-For, so that is what I will use here.

The obvious fix is a spoofing hole

The tempting version takes the first entry of the header:

// WRONG: the client controls the left-hand side
ip := strings.Split(r.Header.Get("X-Forwarded-For"), ",")[0]

Try it against a typical nginx setup using $proxy_add_x_forwarded_for. That variable appends the address nginx saw to whatever the client already sent. So this request:

curl -H 'X-Forwarded-For: 1.2.3.4' https://example.org/

arrives at your Go server as X-Forwarded-For: 1.2.3.4, 203.0.113.9, where the second value is the real client. The leftmost entry is whatever the client chose to type. If you rate limit, ban or audit on that, anyone can dodge the limit, frame someone else, or fill your logs with lies.

Walk from the right and stop at the first stranger

Hang on, why does the right-hand side work? Because each proxy appends the address of the peer it received the connection from. The rightmost entry was written by the proxy nearest you, which you control. The one before it was written by the next proxy along, and so on. Each entry is only as trustworthy as the hop that wrote it.

So the algorithm is:

  1. Take the TCP peer from r.RemoteAddr. If it is not one of your proxies, ignore all headers and use it.
  2. Otherwise walk the X-Forwarded-For list from the right.
  3. Skip entries that are your own trusted proxies.
  4. The first entry that is not trusted is the client. Everything to its left is unverified claim.

Here it is with net/netip, which gives you comparable values and cheap prefix checks:

package main

import (
	"net/http"
	"net/netip"
	"strings"
)

// Adjust to the proxies you actually run.
var trustedProxies = []netip.Prefix{
	netip.MustParsePrefix("127.0.0.0/8"),
	netip.MustParsePrefix("::1/128"),
	netip.MustParsePrefix("10.0.0.0/8"),
}

func isTrusted(a netip.Addr) bool {
	for _, p := range trustedProxies {
		if p.Contains(a) {
			return true
		}
	}
	return false
}

func clientIP(r *http.Request) (netip.Addr, error) {
	ap, err := netip.ParseAddrPort(r.RemoteAddr)
	if err != nil {
		return netip.Addr{}, err
	}
	peer := ap.Addr().Unmap()
	if !isTrusted(peer) {
		return peer, nil // direct connection: headers are just claims
	}

	var hops []string
	for _, v := range r.Header.Values("X-Forwarded-For") {
		hops = append(hops, strings.Split(v, ",")...)
	}
	for i := len(hops) - 1; i >= 0; i-- {
		a, err := netip.ParseAddr(strings.TrimSpace(hops[i]))
		if err != nil {
			break // garbage in the chain: stop trusting it
		}
		if a = a.Unmap(); !isTrusted(a) {
			return a, nil
		}
	}
	return peer, nil
}

Two details matter. r.Header.Values returns every copy of the header, because a proxy is allowed to send several X-Forwarded-For lines rather than one joined line. And Unmap turns ::ffff:10.0.0.5 into plain 10.0.0.5, so the prefix checks do not miss IPv4 addresses that arrive in mapped form.

A quick test that fails if the logic is wrong

This is the sort of function where one off-by-one hands out free spoofing, so it earns a table test:

func TestClientIP(t *testing.T) {
	cases := []struct{ remote, xff, want string }{
		{"127.0.0.1:5000", "203.0.113.9", "203.0.113.9"},
		{"127.0.0.1:5000", "1.2.3.4, 203.0.113.9", "203.0.113.9"},
		{"10.0.0.2:5000", "203.0.113.9, 10.0.0.5", "203.0.113.9"},
		{"198.51.100.7:5000", "1.2.3.4", "198.51.100.7"},
		{"127.0.0.1:5000", "nonsense", "127.0.0.1"},
		{"127.0.0.1:5000", "", "127.0.0.1"},
	}
	for _, c := range cases {
		r := httptest.NewRequest("GET", "/", nil)
		r.RemoteAddr = c.remote
		if c.xff != "" {
			r.Header.Set("X-Forwarded-For", c.xff)
		}
		got, err := clientIP(r)
		if err != nil || got.String() != c.want {
			t.Errorf("%s / %q: got %v, %v; want %s", c.remote, c.xff, got, err, c.want)
		}
	}
}

The second and fourth rows are the important ones. One proves a forged left-hand entry is ignored; the other proves headers from an untrusted peer are ignored completely.

Quick detour: Go's own reverse proxy and this header

If your proxy is itself Go, httputil.ReverseProxy behaves like nginx by default. It appends the peer address to any existing X-Forwarded-For, so the chain grows and the rightmost-first walk still works.

Since Go 1.20 there is also the Rewrite hook, and pr.SetXForwarded() does the opposite: it discards the incoming X-Forwarded-For and sets it to just the peer. That is what you want on the outermost proxy, where nothing upstream is trusted. Appending is right in the middle of a chain; overwriting is right at the edge.

Using it without touching every handler

Do the lookup once in middleware and put the result on the request context, or into a logger. With slog:

func withClientIP(log *slog.Logger, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ip, err := clientIP(r)
		if err != nil {
			http.Error(w, "bad remote address", http.StatusBadRequest)
			return
		}
		log.Info("request", "client", ip.String(), "method", r.Method, "path", r.URL.Path)
		next.ServeHTTP(w, r)
	})
}

Cases where a header is the wrong tool

  • Your proxy address is not stable. A hard-coded prefix list rots when the load balancer's range changes. Load it from config, and keep it tight rather than trusting a whole /8 out of laziness.
  • The proxy sends a different header. Some CDNs use their own (CF-Connecting-IP and similar). Only believe those when the TCP peer is inside that vendor's published ranges, otherwise they are as forgeable as any other header.
  • You need the address at TCP level. The PROXY protocol, which HAProxy and several cloud load balancers support, prepends the original address to the connection itself. Go's standard library does not parse it, so you would wrap the listener with a small library, and you must then accept it only from the proxy. It sidesteps HTTP headers, but the same trust question applies.
  • Ports in the list. A few proxies write ip:port into the header. netip.ParseAddr will reject those and the loop falls back to the peer, which is safe but wrong. Check what yours sends.

A note on what you are now storing

Once the logs show real addresses, they hold more personal data than they did when everything was 127.0.0.1. Under UK GDPR an IP address can be personal data, so the retention period and access to those logs are now worth a thought. That part is an interpretation that depends on what you do with the logs, not a rule I can settle for your case.

The address you log is only ever as honest as the hop that wrote it. Count your proxies, trust exactly those, and read from the right.

← Back to the blog