Power Cuts Turn Home Labs into Distributed Systems

The power goes out, the UPS beeps, and suddenly your home lab has become a distributed system. Machines disappear at different times, disks flush different amounts of data, network equipment reboots in the wrong order, and one service cheerfully assumes the others are still alive.

A power cut is a partial failure with terrible timing. It is not merely a restart. It is a small, physical version of the failures we spend pages of documentation trying to describe: lost messages, stale state, interrupted writes and components that disagree about what happened.

Everything fails at a different moment

A wall socket gives you one apparently simple event: power is gone. Your lab experiences several events instead. The router may stay alive for a few minutes. The storage server may shut down cleanly. A Raspberry Pi may lose power immediately. The Wi-Fi access point may reboot before the machines that need it.

That ordering matters. A service can be running while its database is unavailable. A node can have a network address while its clock is wrong. A container can restart successfully while the volume it expects is still being checked.

This is the first useful mental model: your home lab is a collection of failure domains, even if every box sits in the same cupboard.

Power supplies fail or run out of battery at different times.
Network paths vanish independently of the hosts using them.
Storage may acknowledge some writes and lose others.
Services restart in an order you did not plan.

UPS protection buys time, not correctness

A UPS is excellent at turning an abrupt failure into a controlled shutdown. It cannot make an unclean write durable, repair a corrupt database, or decide which replica is authoritative. It gives software time to behave properly.

The useful setup is therefore not just "put the server on a UPS". The UPS needs to tell something when its battery is low, and that something needs to begin a shutdown before the battery is empty.

For a small Linux host, a monitoring daemon can usually trigger a shutdown through the UPS' USB or network interface. The exact package varies, but the design should answer three questions:

How much runtime remains?
Which machines shut down first?
What happens if the UPS daemon itself cannot reach the UPS?

Do not shut down every machine at the first flicker. A short outage may be harmless, and repeated power cycling is its own form of wear. Use a delay and a minimum-runtime threshold, then test the policy by pulling the plug with non-critical equipment attached.

Clean shutdown is a protocol

Stopping processes in the right order is less glamorous than choosing a storage engine, but it is where reliability usually lives. Applications need to stop accepting work, finish or cancel in-flight operations, flush state, and close their dependencies.

Shutdown should move from the edge towards the centre. Stop schedulers and public entry points first. Then stop workers. Finally stop databases, queues and storage services. Reversing that order during boot gives dependencies time to become ready before clients use them.

A practical shutdown sequence might look like this:

Disable new jobs and external traffic.
Let active requests finish, subject to a deadline.
Stop consumers so queues stop changing.
Flush application state and close databases.
Unmount or power off storage only after dependants have stopped.

Systemd can express parts of this dependency graph, but only if the services describe their relationships. A service that merely happens to start after PostgreSQL is not necessarily ordered correctly. It may need the database to be fully ready, not just have a process with a PID.

Recovery is where the real bugs appear

Most home lab services survive a reboot. The interesting failures appear when they recover with half a transaction, a stale lock, or a message that was sent but not acknowledged.

Imagine a backup job copying a large file when power disappears. On restart, the destination might contain a complete file, a truncated file, or a file with the expected name but incomplete contents. If the next run trusts the filename, it has converted an interrupted operation into apparently valid state.

Use temporary names and an explicit completion marker. Write to backup.img.partial, flush as appropriate, and rename it to backup.img only when the operation is complete. The consumer should only consider the final name authoritative.

tmp="backup.img.partial"
rm -f -- "$tmp"
make-backup >"$tmp" && sync -f "$tmp" && mv -- "$tmp" backup.img

This does not solve every durability problem. The exact guarantees depend on the filesystem and storage hardware, and a directory update may need its own flush for especially careful designs. The important part is that recovery can distinguish finished work from an interrupted attempt.

Queues need idempotent consumers

Power cuts make message delivery ambiguous. A worker may process a job, lose power before acknowledging it, and receive the same job again after reboot. Or it may acknowledge first and lose the result. You cannot generally recover the missing moment by staring harder at the logs.

Design the operation so repeating it is safe. This is idempotency, and it is far more useful in a home lab than pretending exactly-once processing exists everywhere.

For example, a photo-import job should identify the source file by a stable digest or source identifier. Before creating a new record, it checks whether that identifier has already been processed. The database should enforce the uniqueness rule too, because an application check alone can race.

For effects outside the database, use an idempotency key where possible. A notification, DNS update or remote API call may still need reconciliation after a restart. Store enough state to retry safely, and make the retry visible rather than silently inventing a second result.

Split brain can happen in a cupboard

Distributed systems do not require distant data centres. Two local machines can disagree just as effectively when one loses power and the other keeps serving traffic.

Suppose a primary node disappears. A monitoring script promotes a secondary. The primary then returns with old data and starts serving before it has learnt about the promotion. You now have two writers, a situation commonly called split brain.

Quick detour: this is why a spare machine is not automatically a high-availability system. Redundancy without authority is two opportunities to write conflicting history.

For a modest lab, the safest answer is often deliberately boring. Keep one writer, stop services when their dependencies are uncertain, and require a human decision before promotion. If you do automate failover, define fencing: the old writer must be prevented from serving before the new writer takes over.

Time is another dependency

After a long outage, machines may come back with different clocks. Network time is unavailable until the network is working, and the network may depend on services that are waiting for correct time. Certificates, scheduled jobs, leases and log ordering can all become confusing.

Use a hardware-backed or battery-backed clock where practical, but still treat time synchronisation as a recovery step. Services should tolerate a clock correction during startup and avoid using wall-clock timestamps as unique event ordering.

For expiry and durations, use a monotonic clock when the platform and language provide one. For audit records, record wall-clock time, but do not assume that two machines' timestamps form a perfect sequence.

Test recovery while the lab is boring

The best time to discover that a service cannot recover is before you need it. A monthly test can be small: stop one host, unplug another, restart a database during a write, and check that the documented recovery steps still work.

Keep a short recovery note beside the equipment. It should name the dependencies, the safe boot order, the commands to inspect logs, and the point at which you stop automating and make a human decision.

Also check the uninteresting things:

Are UPS batteries still healthy?
Does the router return with the expected DHCP leases?
Do encrypted volumes require a key that is stored on the failed host?
Can you restore a service without an internet connection?
Do monitoring alerts work when the monitoring host is down?

One final wrinkle: recovery itself can overload the lab. Every service may restart at once, retry every failed dependency, and begin a full scan while backups and monitoring do the same. Add backoff, staggered startup and bounded concurrency. A system that survives the outage but collapses during recovery has merely delayed the incident.

Power cuts are useful teachers because they remove the comforting fiction that a system has one state. There is only what each component knows, what it has durably written, and what the other components are prepared to believe. Design around that, and the next blackout becomes a controlled restart instead of an archaeological dig through corrupted state.