The observer effect

We built a status dashboard for the time server. Then somebody asked the obvious question nobody asks: is the dashboard hurting the clock?

It was. By 36%.

The measurement

A/B/A, sixty-two seconds of raw ppstest per round, with the middle round as the control and the third to prove it wasn’t drift:

Round	Dashboard	PPS jitter (σ)	peak-to-peak
1	on	1912 ns	10252 ns
2	off	1304 ns	6914 ns
3	on	2179 ns	11683 ns

Round 3 reproduces round 1. It’s real.

Why

Two facts, individually harmless, catastrophic together:

The collector was forking chronyc four times a second — once each for tracking, sources, sourcestats, clients. Process creation is one of the most expensive things you can ask a scheduler to do.
That churn landed on CPU 0 — the one core the PPS interrupt is welded to and cannot be moved off.

The monitoring tool was standing on the neck of the thing it monitors. And it was invisible: every metric looked fine, chrony still said Stratum 1, the dashboard still said “locked”. The number was just quietly worse.

The fix (no timing code was touched)

1. Batch chronyc into one process. It reads commands from stdin, so one fork serves all three:

printf 'tracking\nsources\nsourcestats\n' | chronyc -c

Tell the outputs apart by field count: 14 = tracking, 10 = sources, 8 = sourcestats.

2. Get off CPU 0.

[Service]
CPUAffinity=1

The result

	Dashboard off	Dashboard on	Penalty
Before	1304 ns	1912 / 2179 ns	+36%
After	1437 ns	1169 / 1450 ns	none — within noise

The box running its entire production stack is now quieter than it was sitting idle before the fix.

The general rule

On a machine where one core is load-bearing for precision, every other service you run is a tenant on the other cores, whether it knows it or not. And process creation is the loudest neighbour there is.

A monitoring tool must not perturb what it measures. If you have never checked whether yours does, you do not know that it doesn’t.