Articles / Privacy

894 words

  • privacy
  • analytics
  • logging
  • go
  • nginx
  • uk-gdpr

Your Analytics Pipeline Still Holds the IP You Dropped

You wrote the collector so the event struct has no IP field. The warehouse table has no IP column. The privacy notice says you do not store IP addresses. Then you grep the server for your own address and it turns up in six places.

The IP never lived in your event. It lived in the connection that carried the event, and everything that touches a connection is entitled to write it down.

Follow one beacon from browser to disk

Take a single page-view beacon and list every component that sees the client address before your own code runs. Each one is a separate copy with its own retention.

  • CDN or load balancer access logs, usually on by default and often kept for weeks.
  • Your reverse proxy access log (nginx, Caddy, an ingress controller).
  • The WAF or rate limiter, which has to remember addresses to do its job.
  • Your Go handler, via r.RemoteAddr or X-Forwarded-For, passed to a logger, a tracing span or a geo lookup.
  • Error lines such as "failed to decode event from 203.0.113.7".
  • Queue message headers, if a middleware copies request metadata across.
  • Crash reporters and APM agents that attach the whole request to an exception.

Dropping the field from the payload only fixed the last mile.

Quick detour: the geo lookup that keeps more than you meant to

A popular pattern is to resolve the IP to a country and city, store those, and throw the address away. That is a sensible idea, but check what you kept.

A city plus a user agent plus a timestamp is a fairly tight fingerprint in a small audience. If the lookup also returns ASN or postcode-level data, you have kept most of what the address told you. Coarsening the output is worth doing; just do not call the result anonymous because the dotted quad is gone.

Step 1: find every copy before fixing any

Do the search first, because the list will not match your architecture diagram. Hit the collector from a machine whose address you know, then grep everything you operate.

MYIP=$(curl -s https://ifconfig.me)
curl -s -X POST https://collect.example.org/e -d '{"name":"pageview"}'
sudo grep -rF "$MYIP" /var/log /srv 2>/dev/null | head

Repeat for any hosted service with a log viewer (CDN, APM, error tracker, queue console). If you have IPv6, search for that too; dual-stack clients will show up under whichever family they used.

Step 2: stop the proxy writing it

The proxy log is usually the biggest offender and the easiest fix. In nginx you can log a truncated address with a map.

map $remote_addr $remote_addr_anon {
    ~(?P<ip>\d+\.\d+\.\d+)\.    $ip.0;
    ~(?P<ip>[^:]+:[^:]+):       $ip::;
    default                      0.0.0.0;
}

log_format anon '$remote_addr_anon - [$time_local] "$request" $status $body_bytes_sent';
access_log /var/log/nginx/access.log anon;

That zeroes the last IPv4 octet and keeps the first two groups of an IPv6 address. Test it with both families and with a malformed value, because a regex that quietly falls through to default is better than one that logs the original.

Note that $http_x_forwarded_for and $http_user_agent are logged separately if your format includes them. Check the format string, not just the variable you meant to change.

Step 3: make your own logging unable to leak it

In Go, the easy mistake is a logging middleware that dumps r.RemoteAddr next to the path. Put the truncation in one function and make it the only way an address reaches a log line.

package main

import (
	"log/slog"
	"net/http"
	"net/netip"
)

// coarse returns the /24 (IPv4) or /48 (IPv6) containing addr.
func coarse(addr string) string {
	ap, err := netip.ParseAddrPort(addr)
	if err != nil {
		return "invalid"
	}
	ip := ap.Addr().Unmap()
	bits := 24
	if ip.Is6() {
		bits = 48
	}
	p, err := ip.Prefix(bits)
	if err != nil {
		return "invalid"
	}
	return p.String()
}

func logRequests(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		slog.Info("request",
			"method", r.Method,
			"path", r.URL.Path,
			"net", coarse(r.RemoteAddr),
		)
		next.ServeHTTP(w, r)
	})
}

Two details matter here. Unmap turns an IPv4-mapped IPv6 address back into IPv4, so a dual-stack listener does not log a /48 of nothing useful. And r.URL.Path is logged without the query string, because beacons love putting identifiers in there.

Step 4: the places a middleware hides it

Error messages are the sneaky ones. fmt.Errorf("decode from %s: %w", r.RemoteAddr, err) looks helpful and then travels through every layer that logs errors.

  • Search the code for RemoteAddr and X-Forwarded-For, and route every hit through the coarse function or delete it.
  • Check tracing: HTTP instrumentation often records client.address or http.client_ip as a span attribute by default. Look at your exporter's config.
  • Check what your error tracker attaches. Many send request headers and the remote address unless told to scrub them.
  • Check the CDN: most let you disable or truncate client IP in logs, but it is a setting you have to find.

What this means under UK GDPR

This is interpretation, not legal advice. UK GDPR Article 4(1) lists an "online identifier" as one way a person can be identified, and the ICO treats IP addresses as personal data in practice. So each copy above is processing, and each one needs a purpose and a retention period you can defend.

Two principles bite directly. Data minimisation (Article 5(1)(c)) asks whether you need the address at all for that log. Storage limitation (Article 5(1)(e)) asks how long you keep it. "Security logs for 14 days" is an answer. "Nobody checked" is not.

Truncation helps, but a /24 attached to a rich event stream can still single someone out, so treat it as risk reduction rather than anonymisation. If your privacy notice says you do not collect IP addresses, the grep in step 1 is how you find out whether that sentence is true.

Run the grep again

After the changes, repeat the beacon and the search, and add it to a deployment checklist. New components arrive quietly: someone adds an ingress, a sidecar or an APM agent, and the address reappears in a log nobody was watching. A five-line check that fails when your own IP appears on disk is cheap and catches exactly that.

← All articles