w5isp.com/content/blog/postgres1.md

+++ title = "When Postgres isn't the right choice" date = 2026-04-18 draft = true +++

I've been working on For the Microwave Propagation site I've been working on

┌────────────────────────────────────────┬───────────┬───────────┐ │ Phase │ Wall time │ % of fh=0 │ ├────────────────────────────────────────┼───────────┼───────────┤ │ HRRR fetch (f00) │ 27s │ 5% │ ├────────────────────────────────────────┼───────────┼───────────┤ │ Store HRRR profiles (JSONB upsert) │ 37s │ 7% │ ├────────────────────────────────────────┼───────────┼───────────┤ │ Native duct fetch (wgrib2) │ 104s │ 19% │ ├────────────────────────────────────────┼───────────┼───────────┤ │ NEXRAD fetch + merge (f00 only) │ 17s │ 3% │ ├────────────────────────────────────────┼───────────┼───────────┤ │ Commercial-link merge (f00 only) │ 95s │ 17% │ ├────────────────────────────────────────┼───────────┼───────────┤ │ Score grid + upsert propagation_scores │ 281s │ 50% │ ├────────────────────────────────────────┼───────────┼───────────┤ │ Total fh=0 │ 562s │ 100% │ └────────────────────────────────────────┴───────────┴───────────┘

The propagation map at prop.w5isp.com ingests HRRR weather every hour and re-scores the entire CONUS grid across five amateur-radio microwave bands. That is 95,000 grid points × 5 bands × 19 forecast hours = ~9 million score values per chain. For a long time those lived in a Postgres table called propagation_scores. They don't anymore, and the map went from sluggish to instantaneous. This is the story of why.

The shape mismatch

Each hourly run did this:

DELETE FROM propagation_scores WHERE valid_time = $1; INSERT INTO propagation_scores (lat, lon, band_mhz, valid_time, score, factors) ...

475,000 rows per forecast hour, funneled through four indexes (PK UUID, uniqueness tuple, valid_time, (band_mhz, valid_time)), plus a JSONB factors blob for the analysis hour. Scoring + upsert took about 4m 40s per forecast hour. Across 19 hours that's more than an entire hourly cron cycle, so we had to back off to every-three-hours just to keep up.

The problem wasn't Postgres — Postgres was doing exactly what we asked. The problem was that the shape we were asking for was wrong for how the data gets read.

The map is a canvas heatmap. It wants one dense (lat, lon) → score array for a single (band, valid_time). We were storing it as half a million row tuples with MVCC bookkeeping, toast pages, and a B-tree index per column we'd never query by. Every write rematerialized metadata the reader throws away.

Three stopgaps that weren't enough

Before rewriting the store, we tuned what was there:

1. DELETE + INSERT … with no ON CONFLICT. The chain worker rewrites a full (valid_time, all bands) slice every hour, so conflict detection was pure waste. Skipping it shaved a chunk off.
2. Skip factors on forecast hours. factors is a ~200-byte JSONB per row describing the ten scoring components (rain, humidity, refractivity, etc.). We only show it on the analysis hour (f00) when the user clicks a cell. Passing factors: nil for f01–f18 skips a JSONB encode plus a toast write for roughly half the volume.
3. UNLOGGED table. Writes bypass the WAL entirely. On unclean shutdown the table truncates — fine, because PropagationGridWorker rebuilds it from HRRR every hour. Durability was never the point; this was a cache with extra steps.

These helped, but the per-row overhead of a row-oriented store against a dense numeric grid is a ceiling you can't tune past. The phase was still the wall-clock dominator.

The binary file

Each (band_mhz, valid_time) grid now lands in one file at /data/scores/{band}/{iso}.ntms:

magic : 4 bytes "NTMS" version : 1 byte 0x01 band_mhz : 4 bytes uint32 LE valid_time : 8 bytes int64 unix seconds LE lat_min : 4 bytes float32 LE lon_min : 4 bytes float32 LE step_deg : 4 bytes float32 LE n_rows : 2 bytes uint16 LE n_cols : 2 bytes uint16 LE scores : n_rows × n_cols bytes (uint8, 0–100; 255 = no-data)

33-byte header, dense uint8 array. A full CONUS grid serializes to ~93 KB per band per hour. A cell at (lat, lon) is at byte offset row × n_cols + col, so a point lookup is arithmetic plus a single File.read/1. No index, no query planner, no lock manager.

Writes go through the temp-then-rename pattern:

tmp = path <> ".tmp." <> unique_suffix() File.write!(tmp, binary, [:binary]) File.rename!(tmp, path)

With Postgres off the hot path, the full f00–f18 chain dropped from ~170 min to ~45–60 min, so we ran the cron up to hourly. The 15-minute prune cron reaped expired files with rm — instant.

The f00 factor breakdown

One thing binary files don't handle well is the analysis-hour factor breakdown (the 10-component panel you see when you click a cell). Those factors are heterogeneous Elixir maps, not a numeric grid.

The solution was a second file format, ProfilesFile: one compressed ETF file per valid_time at /data/scores/profiles/{iso}.etf.gz, storing the enriched HRRR grid keyed by {lat, lon}. On a click, we reload the relevant cell's profile and rescore on demand. Same atomic-rename write pattern, different payload — ETF where heterogeneity matters, raw binary where density does.