Blog / Security

723 words

  • lets-encrypt
  • certbot
  • tls
  • nginx
  • http-01
  • debugging

Why Let's Encrypt Renewal Fails After a Port 80 Redirect Change

The certificate was issued fine, the site has worked for two months, and then it expires. Nothing on the server looks broken. The culprit is usually a change someone made weeks ago to how port 80 behaves, which only gets tested again when renewal runs.

Renewal is invisible by design, so a break in it stays hidden until the old certificate runs out. That delay is what makes this so annoying to debug.

What the HTTP-01 challenge actually does

With the HTTP-01 challenge, Let's Encrypt asks your server for a token at a fixed path. The request is plain HTTP, on port 80, for this URL:

http://example.org/.well-known/acme-challenge/<token>

Your ACME client (certbot, for example) puts the token there, and the validation servers fetch it. If the body matches, you get a certificate. The first request always goes to port 80; there is no way to start it on 443.

The Let's Encrypt challenge types page documents this. It also says redirects are followed, with limits on where they may lead: as I understand the docs, only to ports 80 and 443, and only a limited number of hops.

Quick detour: why a redirect to HTTPS is fine

Hang on, if validation follows a redirect to HTTPS, what certificate does it see? On the first issuance there is not a valid one yet. The answer is that the validator does not check the certificate on that redirected HTTPS hop. A self-signed or expired certificate does not matter.

That is sensible. The challenge proves you control what answers for the name, not that your TLS is already healthy. Otherwise an expired certificate could never be renewed.

Where the redirect goes wrong

The common setup is a blanket "send everything to HTTPS" rule. It works for browsers and it works at first issuance, so nobody questions it. Here are the ways it later bites:

  • Webroot mismatch. Certbot writes the token into the port 80 site's webroot, but the redirect sends the request to the 443 site, which has a different root and returns 404.
  • Redirect to a non-standard port. An app or proxy sends clients to something like :8443, which the validator will not follow.
  • Redirect to a host that does not answer. For example a canonical www name with no DNS record or no listener.
  • A loop or an auth wall. The application catches every path, including /.well-known/, and bounces to a login page.
  • A firewall change. Someone "hardened" things and closed port 80 because "we only serve HTTPS now".

The first one is the classic. Both servers look correct on their own; it is the hand-off between them that fails.

A config that keeps the challenge on port 80

The robust fix is to answer the challenge path directly on port 80, and redirect everything else. Here is an nginx example:

server {
    listen 80;
    listen [::]:80;
    server_name example.org www.example.org;

    location /.well-known/acme-challenge/ {
        root /var/www/acme;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

Then point certbot at that directory:

certbot certonly --webroot -w /var/www/acme -d example.org -d www.example.org

One webroot shared by every site on the machine means one small place to get right. Nothing about the challenge then depends on how the HTTPS vhost is configured.

Test it without waiting for expiry

Do not wait for the real renewal to find out. Two checks cover most of it.

  1. Create a test file and fetch it the way the validator would, following redirects: 
    mkdir -p /var/www/acme/.well-known/acme-challenge
echo ok > /var/www/acme/.well-known/acme-challenge/test
curl -sSL -o /dev/null -w '%{http_code} %{url_effective}\n' \
  http://example.org/.well-known/acme-challenge/test
    You want 200, and an effective URL you recognise. Run it from outside your network, since your own router may answer differently.
  2. Run a dry run, which goes through the real challenge flow against the staging CA: 
    certbot renew --dry-run

Check both IPv4 and IPv6 if you publish AAAA records. Use curl -4 and curl -6 separately. A stale AAAA record pointing at a machine that no longer serves the site is a quiet way to fail.

Make failure loud

The dry run only proves things today. The best defence is to notice a failed renewal while the old certificate still has weeks left.

  • Check that the renewal timer exists: systemctl list-timers | grep certbot.
  • Look at the last run: journalctl -u certbot.service, if your package uses a systemd service.
  • Add an external check on the certificate's expiry date, alerting at 14 days or so.

Certbot attempts renewal well before expiry, so a failure leaves you a window. That window is only useful if something tells you it has opened.

← Back to the blog