Articles / Security

1161 words

  • linux
  • luks2
  • dropbear
  • initramfs
  • cryptsetup
  • ssh

Unlocking an Encrypted Headless Server Over SSH at Boot

Full-disk encryption on a server is a fine idea right up until the server reboots and there is nobody standing in front of it to type the passphrase. If the box is in a rack three sites away, or it is a VPS with no serial console you trust, the machine just sits there at the LUKS prompt forever.

The usual workarounds are either "don't encrypt the boot disk" (defeats the point) or "unlock it automatically with a TPM", which I covered in an earlier article on systemd-cryptenroll. TPM auto-unlock is great for protecting against a stolen disk, but it deliberately removes the human from the loop. That means a stolen or seized running machine gives up its data just as easily as an unencrypted one.

Sometimes you want the opposite trade-off: keep the passphrase out of the machine entirely, but still be able to supply it remotely after a reboot.

The standard way to do this on Debian and Ubuntu is dropbear-initramfs: a stripped-down SSH server built into the early boot image, listening on the network before the root filesystem is even decrypted. You SSH in, run one command, type the passphrase, and the boot continues.

It has been the go-to answer for this problem for the better part of two decades, and it still works. A few details will bite you if you skip them, though:

  • host key algorithms
  • network naming
  • unattended-upgrades

What you need before starting

This assumes you already have a LUKS-encrypted root or boot volume set up with cryptsetup and a working /etc/crypttab, the kind of setup covered in the earlier detached-header LUKS2 article.

You also need genuine out-of-band console access (IPMI, a hypervisor console, a cloud provider's VNC/serial view) for the first few reboots. If dropbear or networking is misconfigured, you want a way to see the stuck prompt rather than discovering it by phoning the datacentre.

Install it, then swap the RSA host key for Ed25519

sudo apt update
sudo apt install dropbear-initramfs cryptsetup-initramfs

Installing the package generates default host keys under /etc/dropbear/initramfs/. Check what got created:

sudo ls /etc/dropbear/initramfs/

Older versions default to RSA and ECDSA host keys. That matters because OpenSSH clients from 8.8 onwards disable the ssh-rsa signature algorithm by default, so a stock RSA host key can produce a confusing "no matching host key type found" error rather than a clean connection.

It is worth generating an Ed25519 host key explicitly and dropping the RSA one:

sudo rm -f /etc/dropbear/initramfs/dropbear_rsa_host_key
sudo dropbearkey -t ed25519 -f /etc/dropbear/initramfs/dropbear_ed25519_host_key

Print the fingerprint now, while you still have a normal login shell, so you can compare it later instead of blindly accepting whatever the initramfs SSH server presents:

sudo dropbearkey -y -f /etc/dropbear/initramfs/dropbear_ed25519_host_key | grep fingerprint

Write that fingerprint down somewhere other than the server itself.

Only public keys get in

The initramfs environment has no concept of user accounts, PAM, or password auth worth trusting, so this is public key only. Add the client key you'll connect with:

sudo mkdir -p /etc/dropbear/initramfs
echo "ssh-ed25519 AAAA...your-key... you@laptop" | sudo tee /etc/dropbear/initramfs/authorized_keys
sudo chmod 600 /etc/dropbear/initramfs/authorized_keys

Reuse a key you already manage carefully. An existing SSH key from a hardware token works fine here, since dropbear's client-side pubkey verification is unremarkable; it's the server side that's limited. Don't reuse a key that's also authorised for unrelated low-value services.

Lock dropbear down to the one job it has

Edit /etc/dropbear/initramfs/dropbear.conf:

DROPBEAR_OPTIONS="-p 2222 -s -j -k -I 60"

Each flag earns its place:

  • -p 2222 puts it on a non-standard port so the flood of automated SSH probes on port 22 doesn't end up in initramfs logs you can't easily inspect.
  • -s disables password authentication entirely: key-only, no exceptions.
  • -j and -k disable local and remote port forwarding, since there is nothing useful to tunnel through at this stage and no reason to expose the option.
  • -I 60 drops idle connections after 60 seconds.

There's no nftables or fail2ban running in initramfs. Dropbear is exposed however you've configured it, unfiltered by anything on the box. Restrict access to it at the network edge (security group, upstream firewall, whatever sits in front of the server) rather than assuming it inherits your normal ruleset.

Early-boot networking lives on the kernel command line

Initramfs networking is handled separately from your normal netplan or /etc/network/interfaces configuration; it's driven by the kernel command line and a much older mechanism. Check /etc/initramfs-tools/initramfs.conf for the DEVICE setting. Leaving it blank tries every interface, which is fine for a single-NIC box.

For DHCP, add ip=dhcp to the kernel command line via GRUB:

sudo sed -i 's/GRUB_CMDLINE_LINUX="/GRUB_CMDLINE_LINUX="ip=dhcp /' /etc/default/grub
sudo update-grub

For a static address, which is generally the better choice on a server, use the kernel's ip= syntax: client-ip::gateway-ip:netmask:hostname:interface:none:

GRUB_CMDLINE_LINUX="ip=203.0.113.10::203.0.113.1:255.255.255.0::eth0:none"

Confirm the interface name with ip a beforehand. Cloud images frequently use ens3 or enp1s0 rather than eth0, and initramfs won't rename it for you.

Make sure the network driver for your NIC is actually built into the initramfs. If it's a virtio NIC under KVM this is usually automatic, but if MODULES=most isn't set in initramfs.conf and your image is trimmed down, add the specific driver module to /etc/initramfs-tools/modules.

Rebuild, reboot with the console open, unlock

sudo update-initramfs -u -k all

Confirm dropbear and its keys actually made it into the image before you reboot:

lsinitramfs /boot/initrd.img-$(uname -r) | grep -i dropbear

Now, with your out-of-band console open and watching, reboot:

sudo reboot

Once the console shows the machine waiting at the crypt prompt, connect from another terminal:

ssh -p 2222 [email protected]

Verify the host key fingerprint matches what you noted down earlier before typing anything sensitive. Once connected, unlock the disk:

cryptroot-unlock

Enter the LUKS passphrase. The session will report success and the SSH connection will drop as the initramfs hands off to the real root filesystem and dropbear disappears along with it. The rest of boot proceeds normally.

Skip the typing with a forced command

If you'd rather not remember the exact command every time, restrict the authorised key to run it automatically:

command="/bin/cryptroot-unlock",no-port-forwarding,no-x11-forwarding,no-agent-forwarding ssh-ed25519 AAAA... you@laptop

This still requires an interactive session for the passphrase prompt, so don't add no-pty. It does mean a plain ssh -p 2222 root@host drops you straight into the unlock prompt rather than a shell.

That's a reasonable default, and it slightly reduces what an attacker with your key (but not the passphrase) can poke at.

The ways this goes wrong at 3am

Unattended-upgrades or a scheduled reboot on a box you're not actively watching is the classic way to discover that your initramfs networking config was wrong. Test a full reboot cycle deliberately, with console access open, before trusting this on a machine that reboots unattended.

Regenerating dropbear's host keys after any reinstall or update-initramfs cleanup will change the fingerprint. Unlike everyday known_hosts warnings, a changed fingerprint here shouldn't be shrugged off as routine: treat it as suspicious, and re-verify out of band if it happens unexpectedly.

Finally, remember this doesn't fully replace TPM-backed auto-unlock; it complements it for a different threat model. If someone needs the machine to survive unattended reboots without a human ever typing a passphrase, look at TPM2 enrolment or a network-bound approach like Tang/Clevis or Mandos instead.

Dropbear-in-initramfs is for the case where you specifically want a person, somewhere, to remain the one thing standing between a reboot and a decrypted disk.

← All articles