Articles / Privacy

791 words

  • logging
  • nginx
  • logrotate
  • uk-gdpr
  • data-retention
  • self-hosting

How Long Should You Keep Web Server Access Logs, Really?

Nobody can give you a legal number. Neither UK GDPR nor the ICO says "keep access logs for 30 days". What the law gives you is a principle, and the job of turning that into a number is yours.

That is annoying, but it is also freeing. If you can say what the logs are for, you can work out how long that purpose needs them. Most small sites end up with a number far shorter than the default they never chose.

The only rule is storage limitation

The relevant text is Article 5(1)(e) of UK GDPR: personal data should be kept in a form that identifies people for no longer than is necessary for the purposes it was processed for. An IP address in an access log is personal data in most situations, so this applies to your nginx logs.

Notice what is missing: a period. "Necessary for the purpose" is the whole test. The regulator's expectation, as I read ICO guidance, is that you decide a period, can explain why, and actually delete things when it expires.

Start from purposes, not from logrotate defaults

Write down why you keep the logs. Usually it is a short list:

  • Debugging: something broke and you want to see the requests around it.
  • Security: spotting scanning, brute-forcing or abuse.
  • Capacity and traffic: how much, to which URLs.
  • Evidence: proving what happened after an incident.

Each purpose has a different shelf life, and that is the point. Debugging rarely needs more than a couple of weeks, because you either noticed the bug or you did not.

My opinionated starting points

These are my own judgement, not a legal requirement, and your circumstances may differ:

  • Raw access logs with full IPs: 7 to 14 days. Enough to debug last week's deploy.
  • Auth failures and security events: 30 to 90 days, ideally in a separate, smaller log, with the reason written down.
  • Traffic statistics: aggregate them (counts per path per day, no IPs) and keep those as long as you like.

The trick is the third line. Most "we need history" arguments are really about trends, and trends do not need an address attached to every request.

Setting it up with logrotate

Debian's nginx package already rotates logs, with 14 daily rotations. That is not a bad default, but check it rather than assume. Here is an explicit version:

/var/log/nginx/*.log {
    daily
    rotate 14
    maxage 14
    missingok
    notifempty
    compress
    delaycompress
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /run/nginx.pid ] && kill -USR1 $(cat /run/nginx.pid)
    endscript
}

rotate 14 keeps 14 files; maxage 14 also removes anything older by date, which matters if rotation ever stalls. The USR1 signal makes nginx reopen its log files so it writes to the new one.

Quick detour: why maxage matters

Hang on, why have both? Because rotate counts files, not time. If logrotate fails for a week (full disk, broken config, cron not running), you still have 14 files, but they now span a lot longer than you promised.

With maxage, the age limit is enforced when rotation next runs, regardless of how many files exist. It is a cheap belt to go with the braces.

Truncate the IP before it hits disk

If you only need rough location or "same visitor" patterns, you do not need the full address. nginx can write a truncated one using a map:

map $remote_addr $ip_anon {
    ~(?P<ip>\d+\.\d+\.\d+)\.\d+$   $ip.0;
    ~(?P<ip>[^:]+:[^:]+:[^:]+):.*$ $ip::;
    default                         0.0.0.0;
}

log_format anon '$ip_anon - [$time_local] "$request" '
                '$status $body_bytes_sent "$http_referer"';

access_log /var/log/nginx/access.log anon;

That zeroes the last IPv4 octet and keeps the first three groups of an IPv6 address. It is a reduction in precision, not magic: truncated addresses combined with other fields can still point at people. It does shrink what a leak or a subject access request would expose.

It also means your 14-day argument becomes much easier to make, because the data is less sensitive to begin with.

Where the logs hide after rotation

Deleting the files in /var/log/nginx is not the end of it. Check these:

  • Backups that sweep up /var/log, which will keep logs as long as the backup does.
  • A log shipper or aggregator with its own, longer retention.
  • journald, if nginx or your app logs there; look at MaxRetentionSec in journald.conf.
  • Copies in a CDN or hosting provider's dashboard.

The retention period you can honestly state is the longest of all of these, not the one in your logrotate file.

Check that it actually happens

A policy that is not enforced is a wish. Two checks are enough:

ls -l --time-style=long-iso /var/log/nginx/
find /var/log/nginx -type f -mtime +14 -print

The second command should print nothing. If it prints files, your retention is longer than you think. Put it in a monthly reminder, or run it from cron and have it complain.

And if a longer period is genuinely needed, for an incident you are investigating, say, keep that specific slice for as long as the investigation needs and no longer. Retention is a default, not a ban on exceptions.

← All articles