Articles / Linux

  • linux
  • self-hosting
  • power-cuts
  • ups
  • systemd
  • reliability

Why Your Home Server Dies When the Power Blips Twice

One blip is survivable. The second one, thirty seconds or a couple of minutes later, is the one that corrupts the database or leaves the box stuck in a boot loop. The server was not unlucky; it was caught halfway through starting up.

If you live on a long overhead line, you will have seen the pattern: lights flicker, come back, then go off again shortly after. It is rarely coincidence, and the cause is worth a minute of your time.

Why the power blips twice

Overhead rural lines are protected by auto-reclosers. A branch touches the line, a fault current flows, and the recloser opens to clear it. Most such faults are transient (wind, a bird, a branch), so the recloser closes again after a short delay and tries.

If the fault is still there, it opens again, and may repeat the attempt after a longer delay before locking out. I will not quote exact timings: they vary by network operator and by device settings. The shape is what matters: a short outage, a brief return, then another outage.

So "twice" is the protection working as designed. Your server needs to cope with a return of power that is only a loan.

What the first blip sets up

Say the first outage is shorter than your UPS or PSU hold-up time. Nothing happens, which is fine. Say it is longer: the machine is off, power returns, and the BIOS option "Restore on AC Power Loss" (or similar) boots it straight away.

Now the machine is at its most fragile. In the next minute or so it may be doing all of these:

  • replaying the filesystem journal after an unclean stop;
  • spinning up several disks at once (the highest inrush draw of its life);
  • starting a RAID or ZFS resilver check;
  • starting the database, which runs its own crash recovery;
  • waiting for a router or ONT that is itself still booting.

The second blip lands on any of those. A journal replay interrupted by a power cut is usually survivable. A crash recovery interrupted, repeatedly, on a flaky disk is how people end up restoring from backup.

Quick detour: why spin-up matters

Hang on, why do disks matter so much here? A spinning drive draws far more current starting the motor than it does reading. Server boards stagger spin-up for exactly this reason; many home builds do not.

On a marginal supply, or a UPS that has just been drained by the first outage, that surge can brown out the very machine you are trying to rescue. The boot loop then looks like a software fault. It is arithmetic about watts.

The UPS makes it worse if it is configured naively

A UPS that has run down and shut the server off will usually restore its output once mains returns, sometimes with only a few percent of battery. The server starts, the mains drops again, and the UPS now has almost nothing to give.

Many units can be told to hold the output off until the battery has recharged to some percentage. Look for a "minimum battery charge to restart" setting in the UPS menu or its management tool. It is the single most useful thing to change.

Make boot wait for stable mains

You want the server to start only once power has been good for a while. Two layers help.

  1. In firmware, set the AC-loss behaviour to "last state" or "stay off" if you can tolerate a manual start, or add a start delay if the board offers one.
  2. In the OS, hold heavy services until the UPS reports mains present and charged.

For the second layer, if you use NUT, a systemd drop-in can gate a service. This is a sketch: the UPS name myups and the 80 percent threshold are examples, so adjust them.

# /etc/systemd/system/postgresql.service.d/wait-for-power.conf
[Service]
ExecStartPre=/bin/sh -c 'until [ "$(upsc myups ups.status 2>/dev/null)" = "OL" ] \
  && [ "$(upsc myups battery.charge 2>/dev/null)" -ge 80 ]; do sleep 10; done'
TimeoutStartSec=15min

Note the TimeoutStartSec: without it, systemd gives up long before a slow recharge finishes. Also, ups.status can hold several flags (for example OL CHRG), so the exact-match test above is deliberately strict; loosen it with a glob match if your UPS reports charging flags.

Check it actually works

Do not trust the config; pull the plug. With a UPS, simulate the pattern properly:

  1. cut mains until the UPS shuts the server down;
  2. restore mains for twenty seconds;
  3. cut it again, then restore it for good.

Watch what the server does after each return. If it boots into the second cut, your hold-off is not working. Afterwards, check journalctl -b -1 for the interrupted boot and look at your database logs for recovery messages.

Things that still bite

  • The router or ONT may take longer to come up than the server, so anything needing DNS or a network mount at boot fails. Order those services after network-online.target, and make them retry.
  • A write cache on a cheap SSD without power-loss protection can lose recent writes however clean your boot logic is.
  • Backups matter more than any of this; a gated boot reduces damage, it does not remove it.

The pattern to remember: a power return is not the end of the outage, only a gap in it. Design the boot sequence for the gap.