diff --git a/content/blog/ft8-second-by-second.md b/content/blog/ft8-second-by-second.md index f2fa80c..5bec170 100644 --- a/content/blog/ft8-second-by-second.md +++ b/content/blog/ft8-second-by-second.md @@ -1,6 +1,6 @@ +++ title = "An FT8 transmission, second by second" -date = 2026-04-27 +date = 2026-04-26 +++ diff --git a/content/blog/gps-ntp-pi.md b/content/blog/gps-ntp-pi.md index 0697aa2..8aac730 100644 --- a/content/blog/gps-ntp-pi.md +++ b/content/blog/gps-ntp-pi.md @@ -4,23 +4,23 @@ date = 2026-04-27 draft = false +++ -Internet NTP is fine. Polling pool.ntp.org over a residential link gets you within a few milliseconds of UTC, which is enough to keep TLS certificates from looking like they're from the future and enough to make `tail -f` across two machines look mostly sensible. It is not enough to do anything interesting with timestamps. Below the millisecond floor, the network jitter eats everything: the kernel queueing the packet, the NIC waking up, the path through your ISP's gear, the upstream NTP server's own scheduling latency. You can stack more peers, you can pick lower-latency ones, you cannot make the network honest. +Internet NTP is fine, in the same way that supermarket sushi is fine. Polling pool.ntp.org over a residential link gets you within a few milliseconds of UTC, which is enough to keep TLS certificates from looking like they're from the future and enough to make `tail -f` across two machines look mostly sensible. It is not enough to do anything interesting with timestamps. Below the millisecond floor, the network jitter eats everything: the kernel queueing the packet, the NIC waking up, the path through your ISP's gear, the upstream NTP server's own scheduling latency. You can stack more peers, you can pick lower-latency ones, you cannot make the network honest. The network does not want to be honest. The network is, broadly speaking, a series of nested apologies wearing a TCP/IP costume. -The fix is to skip the network entirely. A GPS receiver with a 1PPS output can pin the local clock to within a few hundred nanoseconds of UTC for the cost of a $50 hat, an antenna with a view of the sky, and an afternoon of fighting Raspberry Pi UART config. That puts you a few orders of magnitude better than you can ever get over the internet, and lets you serve time to the rest of your network at stratum 1. +The fix is to skip the network entirely. A GPS receiver with a 1PPS output can pin the local clock to within a few hundred nanoseconds of UTC for the cost of a $50 hat, an antenna with a view of the sky, and an afternoon of fighting Raspberry Pi UART config, which costs nothing in money and a great deal in dignity. That puts you a few orders of magnitude better than you can ever get over the internet, and lets you serve time to the rest of your network at stratum 1, which is a thing nobody in your house will care about and you will mention at every available opportunity anyway. ## The two halves of GPS time -Every GPS receiver does two related but separate things. The first is decode the navigation message and produce a stream of NMEA sentences over a serial port: `$GPRMC` once a second with a UTC timestamp, position, speed, the works. The second is emit a hardware pulse, exactly once a second, on a dedicated pin, whose rising edge is aligned with the start of each UTC second to within roughly 10 nanoseconds. +Every GPS receiver does two related but separate things, and pretends they are one thing. The first is decode the navigation message and produce a stream of NMEA sentences over a serial port: `$GPRMC` once a second with a UTC timestamp, position, speed, the works. The second is emit a hardware pulse, exactly once a second, on a dedicated pin, whose rising edge is aligned with the start of each UTC second to within roughly 10 nanoseconds. -Those are very different signals dressed up to look like one. NMEA tells you what second it is. The pulse tells you when that second started. NMEA arrives over a UART running at 9600 baud, which means the timestamp characters dribble in over tens of milliseconds and the OS hands them to userspace whenever it feels like it. The pulse is a clean voltage edge that the kernel can timestamp inside an interrupt handler. Use the NMEA stream for the integer second, use the pulse for the fractional part, and the result is a clock with the slow accuracy of GPS and the fast precision of a hardware edge. +Those are very different signals dressed up to look like one, like a labrador in a bowtie. NMEA tells you what second it is. The pulse tells you when that second started. NMEA arrives over a UART running at 9600 baud, which means the timestamp characters dribble in over tens of milliseconds and the OS hands them to userspace whenever it feels like it, which is roughly the temporal precision of a postcard. The pulse is a clean voltage edge that the kernel can timestamp inside an interrupt handler. Use the NMEA stream for the integer second, use the pulse for the fractional part, and the result is a clock with the slow accuracy of GPS and the fast precision of a hardware edge. The labrador, somehow, becomes a Swiss watch. -This is exactly the split chrony's `refclock` configuration is designed around: a slow source for coarse time, a `PPS` source locked to it for fine time, and chrony's filter combining the two. +That split is exactly what chrony's `refclock` configuration is designed around: a slow source for coarse time, a `PPS` source locked to it for fine time, and chrony's filter combining the two and pretending it was never any other way. ## The Uputronics hat -The [Uputronics GPS Expansion Board](https://store.uputronics.com/index.php?route=product/product&product_id=81) is a GPIO header hat built around a u-blox MAX-M8Q. It exposes the receiver's UART on the Pi's `/dev/ttyAMA0` (the "real" PL011 UART, not the software-emulated `mini-uart`) and routes the 1PPS line to GPIO 18. There's an SMA jack for an external active antenna, a coin-cell holder for the backup battery so the receiver doesn't have to cold-start every reboot, and an LED that blinks once per second when it has a fix, which is the most satisfying status indicator in the entire stack. +The [Uputronics GPS Expansion Board](https://store.uputronics.com/index.php?route=product/product&product_id=81) is a GPIO header hat built around a u-blox MAX-M8Q. It exposes the receiver's UART on the Pi's `/dev/ttyAMA0` (the "real" PL011 UART, not the software-emulated `mini-uart`, which is real in the sense that a stage prop is real) and routes the 1PPS line to GPIO 18. There's an SMA jack for an external active antenna, a coin-cell holder for the backup battery so the receiver doesn't have to cold-start every reboot, and an LED that blinks once per second when it has a fix, which is the most satisfying status indicator in the entire stack and the one piece of debugging gear that has never lied to me. -A few alternatives exist (Adafruit's Ultimate GPS HAT, the Sparkfun XA1110 breakout with a wired pulse) and the rest of the writeup applies to any of them with minor pin changes. The Uputronics is the one I have, and the MAX-M8Q is a nice receiver: 72 channel, GPS + GLONASS + Galileo + BeiDou, fix in under 30 seconds from cold and a couple of seconds warm. +A few alternatives exist (Adafruit's Ultimate GPS HAT, the Sparkfun XA1110 breakout with a wired pulse) and the rest of the writeup applies to any of them with minor pin changes. The Uputronics is the one I have, and the MAX-M8Q is a nice receiver: 72 channel, GPS + GLONASS + Galileo + BeiDou, fix in under 30 seconds from cold and a couple of seconds warm. It will also, if you put the antenna anywhere near a window, find more satellites than you knew were up there, which is one of those quietly humbling reminders that the sky is full of expensive American, Russian, European, and Chinese hardware all screaming the time at you for free. ## Wiring up the Pi @@ -29,17 +29,17 @@ Three packages cover everything below. `pps-tools` brings in `ppstest` for sanit sudo apt update sudo apt install pps-tools gpsd chrony -chrony's built-in refclock drivers are `PHC`, `PPS`, `SHM`, and `SOCK`. There's no NMEA parser inside chrony itself, which is why gpsd is in the picture: gpsd does the parsing and hands the result to chrony through shared memory or a Unix socket. +chrony's built-in refclock drivers are `PHC`, `PPS`, `SHM`, and `SOCK`. There's no NMEA parser inside chrony itself, presumably because the chrony authors got one good look at NMEA 0183 and decided life was too short. gpsd does the parsing and hands the result to chrony through shared memory or a Unix socket. -On a Pi 4 or 5, the kernel default is to give `/dev/ttyAMA0` to the Bluetooth radio and shove the serial console out the mini-UART, which is the wrong choice for everything. Three lines in `/boot/firmware/config.txt` fix it: +On a Pi 4 or 5, the kernel default is to give `/dev/ttyAMA0` to the Bluetooth radio and shove the serial console out the mini-UART, which is the wrong choice for absolutely everything you might want to do, but is the right choice if you are a Raspberry Pi engineer who has never had to talk to a GPS at four in the morning. Three lines in `/boot/firmware/config.txt` fix it: enable_uart=1 dtoverlay=disable-bt dtoverlay=pps-gpio,gpiopin=18 -The first turns on the PL011 and points it at the GPIO pins the hat uses. The second tells the kernel to stop fighting you for the good UART by parking the Bluetooth modem on the mini-UART (or you can leave Bluetooth disabled, depending on whether you ever want it back). The third loads the [PPS-GPIO kernel driver](https://www.kernel.org/doc/html/latest/driver-api/pps.html) and tells it to watch GPIO 18 for the pulse and timestamp every rising edge. +The first turns on the PL011 and points it at the GPIO pins the hat uses. The second tells the kernel to stop fighting you for the good UART by parking the Bluetooth modem on the mini-UART, or you can leave Bluetooth disabled entirely, which is what I did because I have never once wanted a Raspberry Pi to do Bluetooth. The third loads the [PPS-GPIO kernel driver](https://www.kernel.org/doc/html/latest/driver-api/pps.html) and tells it to watch GPIO 18 for the pulse and timestamp every rising edge, which is the actual technically interesting bit and gets one line of config because Linux is occasionally generous. -You also want to evict the serial console from `/dev/ttyAMA0`, because if `getty` is sitting on the port reading login banners, the GPS will not be amused. There are two pieces to disable: the kernel console handoff in `cmdline.txt` and the `serial-getty@ttyAMA0` systemd service that gets enabled on the fly when the kernel passes that console argument. Remove the token from `/boot/firmware/cmdline.txt`: +You also want to evict the serial console from `/dev/ttyAMA0`, because if `getty` is sitting on the port reading login banners at the receiver, the GPS will simply file all of that under "not NMEA, do not care" and the receiver will go on muttering its time-of-day to nobody in particular. There are two pieces to disable: the kernel console handoff in `cmdline.txt` and the `serial-getty@ttyAMA0` systemd service that gets enabled on the fly when the kernel passes that console argument, because systemd believes in defense in depth, where the depth is mostly directed at you. Remove the token from `/boot/firmware/cmdline.txt`: sudo sed -i 's/console=serial0,115200 //' /boot/firmware/cmdline.txt @@ -47,7 +47,7 @@ Then disable the getty service so it doesn't come back and grab the UART before sudo systemctl disable --now serial-getty@ttyAMA0 -Skipping the second step is the most common reason gpsd shows up in the journal complaining `SER: /dev/ttyAMA0 already opened by another process`. The cmdline change alone only takes effect on the next boot, and even then `systemd-getty-generator` may have already wired up a unit that survives the cmdline edit. +Skipping the second step is the most common reason gpsd shows up in the journal complaining `SER: /dev/ttyAMA0 already opened by another process`, which is the polite Unix way of saying "you have two daemons in love with the same UART and one of them needs to go." The cmdline change alone only takes effect on the next boot, and even then `systemd-getty-generator` may have already wired up a unit that survives the cmdline edit, because systemd holds opinions and is willing to die on hills you didn't know existed. You also need to give chrony's `_chrony` user and gpsd's `gpsd` user access to the PPS device. The kernel creates `/dev/pps0` owned by root with no group access, so a one-line udev rule plus group memberships handle it: @@ -65,7 +65,7 @@ After a reboot, `/dev/pps0` shows up and `sudo ppstest /dev/pps0` should print o source 0 - assert 1769472001.000000123, sequence: 1 source 0 - assert 1769472002.000000118, sequence: 2 -If the assert times don't tick once per second, your wiring is wrong, your antenna doesn't have sky, or the receiver hasn't acquired yet. The hat's LED is the fastest way to tell which. +If the assert times don't tick once per second, your wiring is wrong, your antenna doesn't have sky, or the receiver hasn't acquired yet, in roughly that order of likelihood. The hat's LED is the fastest way to tell which, and is why I have very strong feelings about hardware that includes a status indicator and very tepid feelings about hardware that does not. ## Pointing gpsd at the receiver @@ -78,7 +78,7 @@ Edit `/etc/default/gpsd`: DEVICES="/dev/ttyAMA0 /dev/pps0" GPSD_OPTIONS="-n" -`-n` tells gpsd to start polling the receiver immediately instead of waiting for a client to connect, which matters because chrony talks to gpsd through SHM rather than the gpsd socket protocol, so without `-n` gpsd will sit idle and chrony will see nothing. Listing `/dev/pps0` alongside the UART tells gpsd to pair the kernel PPS source with the receiver and publish PPS timestamps into SHM segment 1. +`-n` tells gpsd to start polling the receiver immediately instead of waiting for a client to connect, which matters because chrony talks to gpsd through SHM rather than the gpsd socket protocol, so without `-n` gpsd will sit idle, chrony will see nothing, and you will spend a happy hour reading manpages before remembering that gpsd is, at heart, a daemon with abandonment issues. Listing `/dev/pps0` alongside the UART tells gpsd to pair the kernel PPS source with the receiver and publish PPS timestamps into SHM segment 1. sudo systemctl enable --now gpsd @@ -87,9 +87,9 @@ Before starting gpsd you can sanity-check the raw NMEA by reading the UART direc sudo stty -F /dev/ttyAMA0 9600 raw -echo sudo timeout 3 cat /dev/ttyAMA0 -You should see a stream of `$GPRMC`, `$GPGGA`, `$GPGSV` sentences. If the timestamp field in `$GPRMC` is empty and the status flag reads `V` (void), the receiver doesn't have a fix yet. Wait for the hat's PPS LED to start blinking and try again. Once gpsd is running it owns `/dev/ttyAMA0`, so don't leave a `cat` running against it. +You should see a stream of `$GPRMC`, `$GPGGA`, `$GPGSV` sentences, which look like CSV designed by someone who had heard CSV described once at a party. If the timestamp field in `$GPRMC` is empty and the status flag reads `V` (void), the receiver doesn't have a fix yet. Wait for the hat's PPS LED to start blinking and try again. Once gpsd is running it owns `/dev/ttyAMA0`, so don't leave a `cat` running against it unless you enjoy the sound of two processes fighting over a serial port. -After gpsd is up, the verification chrony cares about happens through chrony itself. Once the chrony config in the next section is in place, `chronyc sources -v` will show the `NMEA` and `PPS` refclocks transitioning out of the `?` state and Reach climbing off zero within a couple of polling intervals. That, plus `ppstest /dev/pps0` for the pulse and a brief `cat /dev/ttyAMA0` for the NMEA stream, is all the diagnostic surface this setup needs. +After gpsd is up, the verification chrony cares about happens through chrony itself. Once the chrony config below is in place, `chronyc sources -v` will show the `NMEA` and `PPS` refclocks transitioning out of the `?` state and Reach climbing off zero within a couple of polling intervals. That, plus `ppstest /dev/pps0` for the pulse and a brief `cat /dev/ttyAMA0` for the NMEA stream, is all the diagnostic surface the setup needs, which is a refreshing change from anything else in modern Linux. ## The chrony config @@ -114,9 +114,9 @@ The interesting part of `/etc/chrony/chrony.conf`: # Don't let the kernel step the clock more than 100 ms after startup. makestep 1.0 3 -The `noselect` on the NMEA source is deliberate. Its job is not to discipline the clock, only to tag each PPS pulse with the right second. The `lock NMEA` on the PPS line is what actually wires the two halves together: chrony refuses to use a PPS pulse unless a coarse source has independently agreed on which second it represents, which keeps a stuck or rolled-over receiver from quietly skewing your clock by a full second. +The `noselect` on the NMEA source is deliberate. Its job is not to discipline the clock, only to tag each PPS pulse with the right second, like the supporting actor whose entire role is making the lead's punchlines land. The `lock NMEA` on the PPS line is what actually wires the two halves together: chrony refuses to use a PPS pulse unless a coarse source has independently agreed on which second it represents, which keeps a stuck or rolled-over receiver from quietly skewing your clock by a full second and ruining everyone's evening. -The `prefer` on PPS tells chrony that when it's available and locked, nothing else gets a vote. The internet pool is there so that if the antenna falls over or the receiver loses its mind, you degrade gracefully to ordinary NTP instead of free-running on the Pi's crystal, which is a quartz oscillator with the long-term stability of a wet noodle. +The `prefer` on PPS tells chrony that when it's available and locked, nothing else gets a vote, which is the only sane policy in a household where someone has gone to the trouble of mounting a GPS antenna. The internet pool is there so that if the antenna falls over or the receiver loses its mind, you degrade gracefully to ordinary NTP instead of free-running on the Pi's crystal, which is a quartz oscillator with the long-term stability of a wet noodle and the temperature sensitivity of a small mammal. ## Verifying it actually works @@ -136,22 +136,22 @@ The `prefer` on PPS tells chrony that when it's available and locked, nothing el Update interval : 16.0 sec Leap status : Normal -The numbers to look at are RMS offset (sub-microsecond is great, single-digit microseconds is fine) and root dispersion (which is chrony's own estimate of how far off it might be). Stratum 1 with reference ID `PPS` means chrony thinks the local clock is being disciplined by a hardware reference, which is exactly the claim being made. +The numbers to look at are RMS offset (sub-microsecond is great, single-digit microseconds is fine, anything bigger means you have something interesting to investigate and a free evening you weren't planning on) and root dispersion (chrony's own estimate of how far off it might be, which is chrony being unusually candid for an NTP daemon). Stratum 1 with reference ID `PPS` means chrony thinks the local clock is being disciplined by a hardware reference, which is exactly the claim being made and the bit you can mention at parties. -`chronyc sources -v` shows the individual refclocks and any internet peers, with the `*` marker on whichever source is currently being used to discipline the clock. If `PPS` isn't starred after a few minutes of runtime, something's wrong: either the PPS pulse isn't reaching the kernel (`sudo ppstest /dev/pps0` will tell you), or gpsd isn't publishing into SHM segment 1 (check `/dev/pps0` permissions and that gpsd's `DEVICES` line includes it), or the NMEA segment hasn't locked so PPS refuses to run. +`chronyc sources -v` shows the individual refclocks and any internet peers, with the `*` marker on whichever source is currently being used to discipline the clock. If `PPS` isn't starred after a few minutes of runtime, something's wrong: either the PPS pulse isn't reaching the kernel (`sudo ppstest /dev/pps0` will tell you), or gpsd isn't publishing into SHM segment 1 (check `/dev/pps0` permissions and that gpsd's `DEVICES` line includes it), or the NMEA segment hasn't locked and PPS is refusing to run on principle. The principle is correct. Side with chrony. ## Tightening it down -The setup above lands somewhere around 300 ns RMS on a stock Pi 4 or 5, which is already absurd by any practical standard. Two cheap changes pull it under 150 ns. +The setup above lands somewhere around 300 ns RMS on a stock Pi 4 or 5, which is already absurd by any practical standard and would have astonished an entire generation of horologists. Two cheap changes pull it under 150 ns, because of course they do, and because if you have come this far you are going to do them. -The first is to take gpsd out of the PPS path. Right now PPS travels from the kernel `/dev/pps0` device into gpsd's PPS thread, into SHM segment 1, into chrony. Each handoff is a context switch with scheduler jitter attached. chrony has its own `PPS` refclock driver that opens `/dev/pps0` directly and reads the kernel's interrupt-timestamped pulses without anything in between. gpsd is still useful for parsing NMEA, so it stays in the picture for the coarse source. Replace the SHM 1 line with a direct PPS line: +The first is to take gpsd out of the PPS path. Right now the pulse travels from the kernel `/dev/pps0` device into gpsd's PPS thread, into SHM segment 1, into chrony, which is three trips through userspace for what was supposed to be a single voltage edge. Each handoff is a context switch with scheduler jitter attached, like passing a wine glass between increasingly drunk dinner guests. chrony has its own `PPS` refclock driver that opens `/dev/pps0` directly and reads the kernel's interrupt-timestamped pulses without anything in between. gpsd is still useful for parsing NMEA, so it stays in the picture for the coarse source. Replace the SHM 1 line with a direct PPS line: refclock SHM 0 refid NMEA precision 1e-1 offset 0.0 delay 0.2 noselect refclock PPS /dev/pps0 refid PPS lock NMEA prefer `sudo systemctl restart chrony` and watch `chronyc sources -v`. The estimated error on the PPS line drops by roughly half on its own. -The second is the CPU governor. Raspberry Pi OS defaults to `ondemand`, which lets the cores idle at 600 MHz and ramp up only when something needs them. Interrupts arriving during a low-clock idle take longer to enter the handler, and the first few microseconds of every pulse are timestamped at the slow clock before the core scales up. Pinning to `performance` flattens that: +The second is the CPU governor. Raspberry Pi OS defaults to `ondemand`, which lets the cores idle at 600 MHz and ramp up only when something needs them, which is excellent for laptops and rubbish for time servers. Interrupts arriving during a low-clock idle take longer to enter the handler, and the first few microseconds of every pulse are timestamped at the slow clock before the core wakes up and remembers it has a job. Pinning to `performance` flattens that: echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor @@ -172,14 +172,14 @@ To make it stick across reboots without dragging in `cpufrequtils` (which isn't EOF sudo systemctl enable cpu-performance.service -The cost is roughly 1 watt of additional idle power and a Pi that runs a few degrees warmer. +The cost is roughly 1 watt of additional idle power and a Pi that runs a few degrees warmer, which you will not notice and your electric bill will not notice and nobody anywhere will ever notice. -After both changes, `chronyc sources -v` on my own box reports the PPS source at +/- 107 ns estimated error and individual samples scattered around -100 to +200 ns of the locked-on clock. The remaining noise is the Pi's interrupt latency itself: with a stock kernel, occasionally something high-priority sits on a core long enough to delay the PPS interrupt by a microsecond or two, and that long tail dominates. Killing it requires a `PREEMPT_RT` kernel (real but small additional gain, significant complexity) or hardware timestamping (a different project entirely, involving NICs with PTP support or dedicated timecards). For a closet stratum-1, 100 ns is the floor where the law of diminishing returns starts looking less like a curve and more like a wall. +After both changes, `chronyc sources -v` on my own box reports the PPS source at +/- 107 ns estimated error and individual samples scattered around -100 to +200 ns of the locked-on clock. The remaining noise is the Pi's interrupt latency itself: with a stock kernel, occasionally something high-priority sits on a core long enough to delay the PPS interrupt by a microsecond or two, and that long tail dominates everything. Killing it requires a `PREEMPT_RT` kernel (real but small additional gain, significant complexity, and an entirely new set of things that can go wrong at four in the morning) or hardware timestamping (a different project entirely, involving NICs with PTP support or dedicated timecards, and a credit card you are willing to be unkind to). For a closet stratum-1, 100 ns is the floor where the law of diminishing returns stops being a curve and becomes a wall, against which you are politely invited to bash your head. ## What you actually get -Once it's running, `ntpdate -q` from another machine on the LAN reports offsets in the tens-of-microseconds range, dominated by the LAN switch and the receiving machine's own clock-handling rather than anything on the Pi. The Pi itself is within a microsecond of UTC on the long-term average, which is a thousand times tighter than what the internet pool can give you. +Once it's running, `ntpdate -q` from another machine on the LAN reports offsets in the tens-of-microseconds range, dominated by the LAN switch and the receiving machine's own clock-handling rather than anything on the Pi. The Pi itself is within a microsecond of UTC on the long-term average, which is a thousand times tighter than what the internet pool can give you and several thousand times more than anything on your network can usefully consume. -It is also wildly more time accuracy than any reasonable amateur radio application needs. FT8 wants you within a couple hundred milliseconds of the slot boundary, and meteor scatter modes want maybe 10 ms. Pulse-per-second timestamps on SDR captures, distributed measurement runs, and logging across multiple boxes are where the precision actually pays off, along with the simple satisfaction of running a stratum-1 server in a closet for the price of a nice dinner. +It is also wildly more time accuracy than any reasonable amateur radio application needs. FT8 wants you within a couple hundred milliseconds of the slot boundary, and meteor scatter modes want maybe 10 ms. The Pi will deliver you nanoseconds. Pulse-per-second timestamps on SDR captures, distributed measurement runs, and logging across multiple boxes are where the precision actually pays off, along with the simple, dignified, slightly unhinged satisfaction of running a stratum-1 server in a closet for the price of a nice dinner. -The closet part matters too. The Pi pulls about 3 watts running chrony and the GPS receiver, which is less than the wall wart that powers it on standby. It has been running on my network continuously since I built it, which is the highest praise I can give a piece of infrastructure: I forget it exists, and the clocks are right. +The closet part matters too. The Pi pulls about 3 watts running chrony and the GPS receiver, which is less than the wall wart that powers it on standby and dramatically less than your router. It has been running on my network continuously since I built it, which is the highest praise I can give a piece of infrastructure: I forget it exists, and the clocks are right.