102 lines
6.7 KiB
Markdown
102 lines
6.7 KiB
Markdown
+++
|
||
title = "When Postgres isn't the right choice"
|
||
date = 2026-04-18
|
||
draft = true
|
||
+++
|
||
|
||
I've been working on For the [Microwave Propagation](https://prop.w5isp.com) site I've been working on
|
||
|
||
┌────────────────────────────────────────┬───────────┬───────────┐
|
||
│ Phase │ Wall time │ % of fh=0 │
|
||
├────────────────────────────────────────┼───────────┼───────────┤
|
||
│ HRRR fetch (f00) │ 27s │ 5% │
|
||
├────────────────────────────────────────┼───────────┼───────────┤
|
||
│ Store HRRR profiles (JSONB upsert) │ 37s │ 7% │
|
||
├────────────────────────────────────────┼───────────┼───────────┤
|
||
│ Native duct fetch (wgrib2) │ 104s │ 19% │
|
||
├────────────────────────────────────────┼───────────┼───────────┤
|
||
│ NEXRAD fetch + merge (f00 only) │ 17s │ 3% │
|
||
├────────────────────────────────────────┼───────────┼───────────┤
|
||
│ Commercial-link merge (f00 only) │ 95s │ 17% │
|
||
├────────────────────────────────────────┼───────────┼───────────┤
|
||
│ Score grid + upsert propagation_scores │ 281s │ 50% │
|
||
├────────────────────────────────────────┼───────────┼───────────┤
|
||
│ Total fh=0 │ 562s │ 100% │
|
||
└────────────────────────────────────────┴───────────┴───────────┘
|
||
|
||
The propagation map at prop.w5isp.com ingests HRRR weather every hour and re-scores the entire CONUS grid across five
|
||
amateur-radio microwave bands. That is 95,000 grid points × 5 bands × 19 forecast hours = ~9 million score values per chain. For
|
||
a long time those lived in a Postgres table called propagation_scores. They don't anymore, and the map went from sluggish to
|
||
instantaneous. This is the story of why.
|
||
|
||
The shape mismatch
|
||
|
||
Each hourly run did this:
|
||
|
||
DELETE FROM propagation_scores WHERE valid_time = $1;
|
||
INSERT INTO propagation_scores (lat, lon, band_mhz, valid_time, score, factors) ...
|
||
|
||
475,000 rows per forecast hour, funneled through four indexes (PK UUID, uniqueness tuple, valid_time, (band_mhz, valid_time)),
|
||
plus a JSONB factors blob for the analysis hour. Scoring + upsert took about 4m 40s per forecast hour. Across 19 hours that's
|
||
more than an entire hourly cron cycle, so we had to back off to every-three-hours just to keep up.
|
||
|
||
The problem wasn't Postgres — Postgres was doing exactly what we asked. The problem was that the shape we were asking for was
|
||
wrong for how the data gets read.
|
||
|
||
The map is a canvas heatmap. It wants one dense (lat, lon) → score array for a single (band, valid_time). We were storing it as
|
||
half a million row tuples with MVCC bookkeeping, toast pages, and a B-tree index per column we'd never query by. Every write
|
||
rematerialized metadata the reader throws away.
|
||
|
||
Three stopgaps that weren't enough
|
||
|
||
Before rewriting the store, we tuned what was there:
|
||
|
||
1. DELETE + INSERT … with no ON CONFLICT. The chain worker rewrites a full (valid_time, all bands) slice every hour, so conflict
|
||
detection was pure waste. Skipping it shaved a chunk off.
|
||
2. Skip factors on forecast hours. factors is a ~200-byte JSONB per row describing the ten scoring components (rain, humidity,
|
||
refractivity, etc.). We only show it on the analysis hour (f00) when the user clicks a cell. Passing factors: nil for f01–f18
|
||
skips a JSONB encode plus a toast write for roughly half the volume.
|
||
3. UNLOGGED table. Writes bypass the WAL entirely. On unclean shutdown the table truncates — fine, because PropagationGridWorker
|
||
rebuilds it from HRRR every hour. Durability was never the point; this was a cache with extra steps.
|
||
|
||
These helped, but the per-row overhead of a row-oriented store against a dense numeric grid is a ceiling you can't tune past. The
|
||
phase was still the wall-clock dominator.
|
||
|
||
|
||
The binary file
|
||
|
||
Each (band_mhz, valid_time) grid now lands in one file at /data/scores/{band}/{iso}.ntms:
|
||
|
||
magic : 4 bytes "NTMS"
|
||
version : 1 byte 0x01
|
||
band_mhz : 4 bytes uint32 LE
|
||
valid_time : 8 bytes int64 unix seconds LE
|
||
lat_min : 4 bytes float32 LE
|
||
lon_min : 4 bytes float32 LE
|
||
step_deg : 4 bytes float32 LE
|
||
n_rows : 2 bytes uint16 LE
|
||
n_cols : 2 bytes uint16 LE
|
||
scores : n_rows × n_cols bytes (uint8, 0–100; 255 = no-data)
|
||
|
||
33-byte header, dense uint8 array. A full CONUS grid serializes to ~93 KB per band per hour. A cell at (lat, lon) is at byte
|
||
offset row × n_cols + col, so a point lookup is arithmetic plus a single File.read/1. No index, no query planner, no lock
|
||
manager.
|
||
|
||
Writes go through the temp-then-rename pattern:
|
||
|
||
tmp = path <> ".tmp." <> unique_suffix()
|
||
File.write!(tmp, binary, [:binary])
|
||
File.rename!(tmp, path)
|
||
|
||
With Postgres off the hot path, the full f00–f18 chain dropped from ~170 min to ~45–60 min, so we ran the cron up to hourly. The
|
||
15-minute prune cron reaped expired files with rm — instant.
|
||
|
||
The f00 factor breakdown
|
||
|
||
One thing binary files don't handle well is the analysis-hour factor breakdown (the 10-component panel you see when you click a
|
||
cell). Those factors are heterogeneous Elixir maps, not a numeric grid.
|
||
|
||
The solution was a second file format, ProfilesFile: one compressed ETF file per valid_time at
|
||
/data/scores/profiles/{iso}.etf.gz, storing the enriched HRRR grid keyed by {lat, lon}. On a click, we reload the relevant cell's
|
||
profile and rescore on demand. Same atomic-rename write pattern, different payload — ETF where heterogeneity matters, raw binary
|
||
where density does.
|