Articles / Linux

964 words

  • ups
  • nut
  • linux
  • self-hosting
  • power-cuts
  • sysadmin

Your UPS Can Only Shut the Server Down If It Can Talk to It

A UPS plugged into the wall and nothing else is a delay, not protection. When the mains drops, the box beeps, the server carries on happily, and twenty minutes later the battery runs flat and the machine loses power exactly as it would have without the UPS. You have bought yourself a longer countdown to the same hard crash.

On a rural supply this matters more than most places. Long outages from a fault down the lane, plus a steady drizzle of one-second blips, are normal. Here is how to make the server actually hear what the UPS is saying, using NUT (Network UPS Tools) on Linux.

The goal: shut down cleanly before the battery dies

The UPS needs a data connection (usually USB) to the server. A daemon reads the UPS state, and when the battery is nearly empty it tells the OS to shut down, then tells the UPS to cut its outlets so everything restarts when the mains returns.

Check your UPS has a data port before buying anything else. Many cheap units have a USB socket that is for firmware updates or a vendor tool and is not usable by NUT. The NUT hardware compatibility list is the place to check.

Install and configure NUT

On Debian-ish systems the package is nut. There are four files that matter, all under /etc/nut/.

  1. nut.conf: set MODE=standalone for one server with one UPS.
  2. ups.conf: describes the UPS and which driver talks to it.
  3. upsd.users: a local user for the monitor to authenticate as.
  4. upsmon.conf: what to monitor and what to run at shutdown.

The first three are short:

# /etc/nut/ups.conf
[rack]
    driver = usbhid-ups
    port = auto
    desc = "UPS under the stairs"

# /etc/nut/upsd.users
[upsmon]
    password = use-something-random
    upsmon primary

Older NUT versions spell primary as master. If your config check complains, that is the likely cause.

# /etc/nut/upsmon.conf
MONITOR rack@localhost 1 upsmon use-something-random primary
SHUTDOWNCMD "/sbin/shutdown -h +0"

Restart the NUT services, then ask the UPS what it thinks:

upsc rack@localhost
upsc rack@localhost ups.status
upsc rack@localhost battery.charge

If upsc prints a long list of variables, the link works. ups.status of OL means online. OB is on battery. LB is low battery, and that last one is the flag that triggers the shutdown.

Where "low battery" actually comes from

Quick detour, because this is the bit that bites. The LB flag is not magic: the UPS (or the driver) sets it when charge or estimated runtime crosses a threshold, and the variables battery.charge.low and battery.runtime.low hold those thresholds. Many consumer units ship with values like 10 to 20 per cent charge.

That sounds fine until you remember that a lead-acid battery that is three years old does not behave like the datasheet. Its reported percentage can fall off a cliff under load. Ten per cent of a tired battery may be thirty seconds, not five minutes.

You can raise the threshold in ups.conf without touching the UPS firmware:

[rack]
    driver = usbhid-ups
    port = auto
    override.battery.charge.low = 40

Pick the number by measurement, not by guessing. Which brings us to testing.

Measure your real runtime

Run the server at its normal load, pull the UPS plug from the wall, and log what happens. Do this with the shutdown command temporarily disabled, or you will just trigger it.

while sleep 10; do
  echo "$(date +%T) $(upsc rack@localhost ups.status) $(upsc rack@localhost battery.charge) $(upsc rack@localhost battery.runtime)"
done

Note how long it takes to go from OB to the charge you set as low. Then work out how long your server needs to shut down: stopping databases and flushing disks is not instant. Your low-battery point should leave at least double that as margin. Repeat once a year, as the battery ages.

Do not shut down on every blip

Short interruptions are the norm in the countryside. By default upsmon only starts the shutdown when the UPS reports LB (or on a forced shutdown), so a two-second dip does not take your server down. That is good. But it also means a long outage with a UPS that never raises LB runs the battery to nothing.

If you would rather shut down after a fixed time on battery, upssched can run a timer: start it on ONBATT, cancel it on ONLINE, and shut down when it fires. That is a belt-and-braces extra, useful when you do not trust the UPS's own estimate.

Why the UPS cuts power after the server halts

Hang on, this looks odd. After the OS has halted, NUT tells the UPS to switch off its outlets. Why would you want that?

Because a halted server on a UPS that stays on never reboots. When the mains returns nothing triggers a boot, since the machine never lost power. Cutting the outlets forces a real power loss, and with the BIOS or firmware set to "power on after AC loss", the server comes back by itself. The UPS then recharges while the server is up, so your next outage starts from a decent state.

Set that firmware option now. It is the step people forget, and the first time you notice is when you are away from home.

Test it for real, once

The proper test is a forced shutdown, which does exactly what it says and takes the machine down:

sudo upsmon -c fsd

Do it at a time you can stand it, and confirm three things:

  • the server halts cleanly and the filesystems are marked clean on the next boot;
  • the UPS cuts its outlets after the OS is down;
  • the server powers itself back on.

If any one of those fails, you have found out on a Sunday afternoon rather than during a storm.

Keep the network alive too

One more rural quirk. If your router or ONT is not on the UPS, the server will be shutting down in a dark network, and any alert it sends goes nowhere. Put the router on the same UPS and use the NOTIFYCMD hook in upsmon.conf to send a message on ONBATT. You will know the power is out before the neighbours do.

← All articles