Survive a power cut

A time server’s whole job is to be there. Ours had been up for days, Stratum 1, 199 ns — and it would not have survived a power outage. Two independent bugs, neither visible from any amount of systemctl status.

Pull the plug. It’s the only test that finds these.

Failure 1: never pin the gpsd baud rate

Several guides tell you to reconfigure the GPS module to a higher baud rate and then pin gpsd to match:

GPSD_OPTIONS="-n -s 115200"   # ← don't

Here’s what happens. Most u-blox modules hold their config in volatile RAM. Cut the power and the module comes back at its factory 9600. gpsd, pinned to 115200, opens the port, talks to a device that isn’t listening, and reports nothing. Not a degraded fix. No GPS at all. Your Stratum 1 server silently becomes a Stratum 3 client of the internet — and stays that way until a human notices.

Let gpsd auto-probe:

GPSD_OPTIONS="-n"

It sweeps the standard baud rates, finds the module wherever it landed, and comes back on its own.

And the baud change bought nothing anyway

We measured it: −1 ns difference in PPS offset between 9600 and 115200. Which makes sense — the precision lives in the pulse, not the sentences. NMEA at 9600 has plenty of time to tell you which second it is. Faster serial changes nothing and costs you your power-cut resilience.

Failure 2: your monitoring is load-bearing

gpsd ships with socket activation. It starts when something connects to port 2947. And chrony never connects to port 2947 — it reads gpsd’s shared memory.

So on a fresh boot, nothing starts gpsd. No gpsd, no NMEA, no refclock SHM 0, no second-numbering for the PPS pulses.

Ours appeared to work. It worked because the dashboard — our web status page — polls gpsd on 2947, and the dashboard starts at boot. The monitoring was socket-activating the thing it was monitoring. Stop the dashboard “to reduce load on the clock” and you’d have stopped the clock.

$ systemctl is-enabled gpsd.service
disabled          ← this is the bug

sudo systemctl enable gpsd.service

Check gpsd.service, not gpsd.socket.

What a healthy recovery looks like

With both fixed, and a few internet NTP servers left in chrony.conf as a backstop, we cut the mains and watched:

t+	state
0 s	power restored, boot
~35 s	chrony up, Stratum 3 — leaning on internet NTP. Serving time.
~90 s	GPS fix acquired, NMEA flowing
165 s	PPS trusted, internet sources demoted, Stratum 1

No human involved. That middle window — serving slightly-worse time instead of no time — is why you keep the upstream servers configured even on a GPS clock.