Articles / Networking

  • matrix
  • synapse
  • federation
  • well-known
  • self-hosting
  • debugging

Why Matrix Federation Breaks When .well-known Is Wrong

A classic self-hosted Matrix symptom: your users can join rooms on other servers, but nobody can message you back, or remote servers show your users as offline forever. Client login works fine. The cause is usually a small JSON file, or the lack of one, on the wrong host.

The odd part is the asymmetry. Your server finds other servers without trouble, because their discovery is set up properly. They can't find yours.

Two .well-known files, two different jobs

Matrix uses two separate files under /.well-known/matrix/, and mixing them up causes most of the grief:

  • /.well-known/matrix/client is for apps like Element. It says where the client API lives.
  • /.well-known/matrix/server is for other homeservers. It says where federation traffic should go.

Client discovery working tells you nothing about federation. Element will happily log in through the client file while remote servers get a 404 from the server file.

What a remote server actually does

Say your user ID is @andy:example.com, but Synapse runs on matrix.example.com. A remote server has to turn "example.com" into a host and port. Per the Matrix server-to-server specification, the rough order is:

  1. If the name is an IP literal or has an explicit port, use that directly.
  2. Otherwise fetch https://example.com/.well-known/matrix/server.
  3. If that gives valid JSON with an m.server value, connect to the host and port it names.
  4. If it fails, try SRV records (_matrix-fed._tcp, and the older _matrix._tcp).
  5. If that fails too, connect to example.com:8448.

Step 5 is the trap. If your server is on matrix.example.com:443 and nothing answers on example.com:8448, discovery just falls through to a dead end. The remote server gives up and, depending on the implementation, backs off and retries later.

The file itself

The server file is tiny:

{ "m.server": "matrix.example.com:443" }

Serve it from the apex, example.com, not from matrix.example.com. That is the single most common mistake: people put the file on the host where Synapse lives, but remote servers ask the host in the user ID.

Check it the way a remote server would:

curl -sv https://example.com/.well-known/matrix/server
curl -s https://matrix.example.com/_matrix/federation/v1/version
curl -s https://matrix.example.com/_matrix/key/v2/server

The first confirms discovery, the second confirms something federation-shaped answers at the target, and the third confirms the server publishes its signing keys. If you are on a hosted setup you can also use the public federation tester at federationtester.matrix.org, which walks the same steps and shows which one failed.

Ways the file goes wrong

  • It 404s because the web server on the apex domain was never told about it, or a catch-all rule serves your homepage HTML instead.
  • It includes a scheme: "https://matrix.example.com" is not valid, the value is a host with an optional port.
  • It is not valid JSON (trailing comma, smart quotes pasted from a blog, a BOM).
  • The apex domain redirects somewhere unexpected. The spec allows redirects to be followed, but a redirect to a login page or a different domain is a fine way to get nonsense back.
  • The apex serves it, but the certificate on example.com has expired. Nobody looks at that host any more, so nobody notices.

Quick detour: why the certificate must match the target, not your name

This part is neat. Your Matrix identity is example.com, but the TLS connection goes to matrix.example.com. The remote server checks the certificate against the delegated hostname, not the user-ID domain. So the cert on matrix.example.com must be valid for that name, and it must be served on whatever port you named.

The identity itself is held by the signing keys, published at the key endpoint, and the server name in events stays example.com. Delegation moves the connection, not the identity. That is why you can move the actual server without anyone's user ID changing.

Why it keeps failing after you fix it

Remote servers cache what they found. The spec recommends honouring HTTP cache headers and, absent those, caching for around a day, with an upper bound of a couple of days. Failures get cached too, for a shorter time. You can't flush a remote server's cache.

So after fixing the file, expect some servers to recover quickly and others to lag. Setting a sensible Cache-Control on the file helps you in future. If you are about to change delegation, lower it ahead of time rather than after.

The client file, since you are here

For the client side, /.well-known/matrix/client looks like this:

{ "m.homeserver": { "base_url": "https://matrix.example.com" } }

Browser-based clients fetch it with a cross-origin request, so it needs an Access-Control-Allow-Origin: * header. Without it, web clients fail while desktop apps work, which makes for a confusing afternoon.

A short checklist

  • Fetch the server file from the apex domain, over HTTPS, and read the actual body.
  • Confirm the host and port in it answer on the federation endpoint.
  • Confirm the certificate on that host matches that hostname.
  • Confirm the key endpoint returns your signing keys.
  • Then wait out the caches.

If you want to avoid the file entirely, you can run Synapse on the apex domain's port 8448, or publish SRV records. But a well-known file is one static document and it is the most widely used option, so it is worth getting right. I haven't covered every server implementation here; the behaviour above follows the spec, and individual servers differ in how patiently they retry.