Articles / Security

727 words

  • tls
  • x509
  • curl
  • openssl
  • pki
  • debugging

Why Your TLS Chain Works in Chrome but Fails in curl

You deploy a new certificate, load the site in Chrome, see the padlock, and move on. Then a cron job running curl falls over with curl: (60) SSL certificate problem: unable to get local issuer certificate. The certificate is fine. Your server is not sending the whole chain, and Chrome is quietly covering for you.

What the server is supposed to send

A certificate is trusted because it links back to a root CA that the client already holds. In practice the link goes through one or more intermediates:

  • The leaf certificate, issued for your hostname.
  • One or more intermediate certificates, which signed the leaf.
  • The root, which the client already has in its trust store.

TLS says the server sends the leaf plus the intermediates. The root should be left out, since the client must already trust it, and sending it only wastes bytes. If you send just the leaf, the client has nothing to bridge the gap between your certificate and its roots.

Why Chrome shrugs and curl doesn't

Browsers have a few tricks for a missing intermediate. Chrome reads the Authority Information Access (AIA) extension in the leaf, finds the "CA Issuers" URL, and downloads the missing certificate itself. It also caches intermediates it has seen on other sites, so a chain broken on your site can look fine simply because you visited another site first. Firefox takes another route and ships a preloaded set of known intermediates.

curl, built on OpenSSL (or another library), does none of that. It validates exactly what the server sent against the trust store, and fails when the path is incomplete. So do most other non-browser clients: Python requests, Go's net/http, Java, and your monitoring agent.

Quick detour, because this is the daft part: your broken site can work in your browser for months. The test that "passes" depends on what your browser happened to cache. A clean machine or a fresh browser profile is a better test than your daily driver.

Confirming the diagnosis

Ask OpenSSL what the server actually sends:

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null

Look at the top of the output, in the certificate chain section. A broken server shows only one entry:

Certificate chain
 0 s:CN = example.com
   i:C = US, O = Let's Encrypt, CN = R11
---
Verification error: unable to verify the first certificate

The issuer on line 0 (i:) is an intermediate, but there is no entry 1 for it. A healthy server lists entry 1, whose subject (s:) matches the issuer of entry 0. Newer OpenSSL versions print unable to get local issuer certificate with verify error:num=20; older ones say num=21. Same problem either way.

Be sure to pass -servername. Without it, a server hosting several sites may hand you a default certificate and send you chasing the wrong problem.

Fixing it on the server

The fix is to configure the file your server loads as the leaf followed by its intermediates, in that order. Most CAs hand you a "fullchain" file for exactly this. If you have separate files, join them:

cat example.com.crt intermediate.crt > fullchain.pem

Then point the server at the combined file. In nginx:

ssl_certificate     /etc/ssl/example.com/fullchain.pem;
ssl_certificate_key /etc/ssl/example.com/privkey.pem;

Pitfalls worth checking:

  • Order matters: leaf first, then the certificate that signed it, then the one that signed that.
  • Apache older than 2.4.8 wants the intermediates in a separate SSLCertificateChainFile; newer versions accept them in SSLCertificateFile.
  • With certbot, use fullchain.pem, not cert.pem. The latter is the leaf alone and is a classic cause of this exact bug.
  • Reload the server after changing the file; a running process keeps the old chain in memory.

Checking that it worked

Run the s_client command again and confirm entry 1 appears and the output ends with Verify return code: 0 (ok). Then run curl -sv https://example.com/ -o /dev/null from a machine that has never touched the site. If both pass, you are done.

One more trap: if you still get an error with a complete chain, the client's trust store may be missing the root, which is common in minimal containers without ca-certificates installed. That is a different problem with a different fix, and the s_client output (a full chain, but a root it cannot find) is how you tell the two apart.

Should you instead patch every client to fetch AIA issuers? No. Sending the chain costs a couple of kilobytes per handshake, and it makes your site work everywhere with no client changes.

← All articles