Skip to content

What this is (and isn't)

Transcript

Here’s a thing that cost us a week. It might be costing you right now, and you’d have no way to know.

We built a GPS-disciplined time server on a Raspberry Pi 4. Stratum 1 — a satellite pulse, once a second, straight into a GPIO pin. Then we did what every guide says to do next. Install a realtime kernel. Isolate the interrupt. Pin it to a quiet core.

The realtime kernel made it three times worse. Not slightly worse. Three times. 2134 ns of jitter became 6947 ns.

Here is why. PREEMPT_RT buys its determinism by turning interrupt handlers into schedulable threads. For almost every driver that’s a good trade. But the PPS driver takes its timestamp inside its handler. So threading it doesn’t defer some work — it defers the act of looking at the clock. We didn’t make the system more predictable. We put a scheduler between the pulse and the clock.

The fix is four lines. One flag: IRQF_NO_THREAD. Keep that one handler in hard-IRQ context, and let everything else stay preemptible. Our error went from 2468 ns to 199 ns.

As far as we can tell, nobody has applied this. Which means anyone running GPS timing on a realtime kernel is quietly eating microseconds of error and has no reason to suspect it — because nothing looks broken. chrony still says Stratum 1. The dashboard still says locked. The number is just silently worse.

So that’s the talk. Measure your clock. Don’t trust the guide. And if you take one thing from us, take the patch.

This is not a build guide.

Guides for building a GPS-disciplined Stratum 1 NTP server on a Raspberry Pi already exist. geerlingguy/time-pi and josh-blake/pixie are both good. Go read them. Come back when your jitter is bad.

This site is what happened when we followed that advice on a Raspberry Pi 4 and measured everything: most of it is wrong on this board, one piece of it is wrong on every board, and the single change that helped most was a one-line kernel patch nobody has written down.

PREEMPT_RT made it 3× worse

The realtime kernel — the marquee upgrade — tripled our PPS jitter (2134 ns → 6947 ns). It force-threads interrupt handlers, and the PPS driver takes its timestamp inside the handler. We put a scheduler between the electrical edge and the clock. → Why

You cannot pin the PPS interrupt

On a Pi 4, GPIO interrupts are demuxed through pinctrl-bcm2835 and refuse an smp_affinity. The “isolate the PPS IRQ on its own core” advice is unachievable here — and the only way to enable it is the very thing that costs you the accuracy. → Why

PTP is impossible on a Pi 4

ethtool -T eth0PTP Hardware Clock: none. There is no hardware timestamping. Software PTP is just a worse NTP. Don’t chase it. → Why

Your dashboard is taxing your clock

Ours cost 36% more PPS jitter — by forking chronyc four times a second onto the one core the PPS interrupt is welded to. The instrument was bending the measurement. → Why

Almost none of the things we expected.

ChangeRMS offset
Baseline823 ns
chrony: median filter + prefer on the PPS refclock440 ns
PREEMPT_RT (unpatched)2468 nsworse
PREEMPT_RT + our IRQF_NO_THREAD patch199 ns

Baud rate, SBAS, CPU isolation, IRQ pinning: noise, or actively harmful. The full numbers and methodology are in the measurements.

If you take nothing else from this site, take the kernel patch. Every person running GPIO-based PPS on a PREEMPT_RT kernel is, right now, silently eating microseconds of jitter and has no idea. It’s four lines. It’s upstreamable. It’s the reason our RMS offset is 199 ns instead of 2468 ns.

The escapement is the part of a mechanical clock that takes continuous energy and chops it into discrete, regular ticks. It’s the single component that decides whether a clock is precise or worthless. That is exactly what a PPS interrupt handler does: it takes an electrical edge and turns it into one discrete timestamp. Our entire finding is that all the precision in the system lives in that one handler — and that the realtime kernel was putting a scheduler in front of it.

We didn’t fix a time server. We fixed the escapement.

The cuckoo is the other half. The bird’s whole job is to pop out and announce the hour; the PPS pulse’s whole job is to pop out and announce the second. The bird is the pulse. And, well — everything you were told about this turned out to be a bit cuckoo.