prop/docs/plans/2026-04-14-duckdb-scores-storage.md

DuckDB / Flat-File Propagation Scores Storage Plan

Goal: Replace Postgres-backed propagation_scores with per-(band_mhz, valid_time) on-disk files (DuckDB or raw binary), so the hot write path becomes a single file write instead of 475k Postgres rows through indexes + WAL.

Status: Speculative architectural alternative. Do not execute until the already-shipped Postgres wins (DELETE+COPY, skip-factors, UNLOGGED) have been measured in prod. If those three drop the scoring+upsert phase below ~60s per forecast hour, this rewrite is likely unnecessary.

---

Why consider this

Each forecast-hour write to propagation_scores is:

• 475k rows (95k grid points × 5 bands)
• Passes through 4 indexes (PK UUID, unique tuple, valid_time, (band_mhz, valid_time))
• Even with DELETE+COPY and UNLOGGED, Postgres overhead — toast pages, page locks, vacuum state, MVCC tuple housekeeping — adds per-row cost that a flat array layout avoids entirely
• Shapes a dense numeric grid into a row-oriented representation the map never needs

The map render path also doesn't want individual rows. It wants a dense (lat, lon) → score array for a single (band, valid_time). SQL is the wrong shape.

Approach options

Option A — Parquet files, read via DuckDB

One Parquet file per (band_mhz, valid_time):

/data/scores/10000/2026-04-14T18-00-00Z.parquet
/data/scores/24000/2026-04-14T18-00-00Z.parquet
...

• Columns: lat, lon, score (plus factors only on f00 files)
• ~95k rows × 3 numeric columns + JSONB on f00 ≈ 1-2 MB per f00 file, ~400 KB per fh≥1 file
• Written once per forecast hour via Elixir → DuckDB WAL-less writer (COPY ... TO 'file.parquet') or a Rust/Elixir native Parquet encoder
• Read on the map side via duckdbex (Elixir NIF wrapper) or a Rust sidecar

Pros:

• Columnar compression: a single Parquet file per hour is ~300-500 KB compressed
• DuckDB can query across files: SELECT score FROM 'scores/10000/*.parquet' WHERE lat BETWEEN ... AND ...
• Files are portable — easy to debug, copy, inspect with duckdb-cli
• Pruning is rm — instant

Cons:

• Add a new dependency (duckdbex or a Rust NIF)
• Concurrent writers across multiple pods need file coordination (atomic rename, or a single writer pod)
• factors for f00 still needs somewhere to live — either embedded in the Parquet or a separate lookup file

Option B — Raw binary mmap files

One file per (band_mhz, valid_time), laid out as:

header: 16 bytes (magic + lat_min, lon_min, step, n_rows, n_cols)
scores: n_rows × n_cols × 1 byte (score 0-100)

• ~95k × 1 byte ≈ 95 KB per file uncompressed
• Written in Elixir via File.write/2 on a binary blob
• Read on the map side via :ets.new(:mmap) / File.read/1 cached per (band, vt)

Pros:

• Simplest possible layout
• Writes are ~20 ms per file
• No dependencies at all
• Matches the canvas heatmap's consumption shape exactly

Cons:

• Point-detail (single-cell lookup) needs an index by (lat, lon) — easy with the grid step but less ergonomic than SQL
• factors needs a separate store (another binary file, or keep it in Postgres)
• No ad-hoc query capability (can't ask "what was the score at 35.0/-97.0 two hours ago" without loading the file)

Option C — DuckDB on persistent disk

Single .duckdb file under /data/scores.duckdb:

• One table with the same schema as propagation_scores
• DuckDB is columnar, compressed, and serializes writes via a single database file
• Writes via INSERT INTO scores VALUES ... batched
• Reads via direct SQL from Elixir

Pros:

• Least migration friction — keeps a SQL-ish API
• Columnar storage naturally compresses score columns
• DuckDB writes are dramatically faster than Postgres for bulk inserts (no WAL, no per-row overhead)

Cons:

• DuckDB is single-writer. Multi-pod clusters need to funnel writes through one pod (the propagation grid worker pod — which is exactly the worker-pod-isolation idea discussed earlier in this session)
• File corruption on crash: DuckDB has improved crash recovery but it's not Postgres-grade
• Requires duckdbex NIF

Recommendation

If the already-shipped Postgres wins aren't enough, start with Option B (raw binary files). It's the lowest-risk, highest-impact change because:

1. The map's canvas heatmap already wants a dense score array — a binary file is a closer shape match than SQL rows
2. No new dependencies — File.write/2 and File.read/1 are enough
3. Concurrent write safety is a solved problem: write to a temp file, File.rename/2 atomically
4. Reverting is easy: keep propagation_scores populated from the flat file asynchronously during a migration period

The factors storage question is the only thing that forces picking between Option A and Option B:

• If we're fine with no factors for forecast hours (already the case after the skip-factors change), factors for f00 stay in Postgres and the hot path is file-only. Option B works as-is.
• If we want factors for all hours, Option A (Parquet) is cleaner because factors can ride on the same file.

Architectural prerequisites

Before this is worth doing:

1. Separate worker pod. Multi-writer coordination on a single file is painful; a single "propagation" pod that owns all writes is much simpler. The isolation idea from earlier in this session dovetails here.
2. Shared volume or single-node. Files need to live on a PVC shared between the worker and web pods, or the web pods need a cached-read path that pulls the file from the worker pod via HTTP. The existing /data NFS mount is already shared across all pods, which makes /data/scores free.
3. Baseline perf measurements from the already-shipped Postgres changes. Don't do this if the map already feels fast.

Read-path changes

Map render (bulk)

Current:

Propagation.scores_at(band_mhz, valid_time, bounds)
# → list of %{lat, lon, score} maps from Postgres

New:

Propagation.ScoresFile.load(band_mhz, valid_time)
# → %{lat_min, lon_min, step, rows, cols, data: <<binary>>}

The JS hook receives the binary directly (base64 or bytestring in an assign) and draws the canvas without an intermediate list traversal.

Point detail (single-cell)

Current:

Propagation.point_detail(band, lat, lon, valid_time)
# → %{score, factors, ...}

New:

Propagation.ScoresFile.point(band_mhz, valid_time, lat, lon)
# → %{score}
Propagation.FactorsStore.for_point(lat, lon, valid_time)
# → %{factors} (still Postgres-backed, only f00)

Scorer diff

Current: reads latest f00 rows with factors from Postgres. Unchanged — factors stay in Postgres.

ScoreCache

Current: ETS cache of (band, vt) → list-of-scores. Unchanged — can wrap the file-load path the same way.

Write-path changes

PropagationGridWorker.run_chain_step/2 currently calls:

scores = compute_scores(grid_data, valid_time, forecast_hour)
Propagation.replace_scores(scores, valid_time)

Replace with:

scores = compute_scores(grid_data, valid_time, forecast_hour)
Propagation.ScoresFile.write!(band_mhz, valid_time, scores)
# Still write factors to Postgres for f00 only
if forecast_hour == 0, do: Propagation.replace_factors(factor_rows, valid_time)

Expected timings: ~10-50 ms per band per fh vs 4-5 seconds per band per fh today. The scoring CPU cost stays the same; only the storage phase collapses.

Pruning

Current: 15-minute cron running DELETE WHERE valid_time < now - 2h.

New: 15-minute cron running find /data/scores -mmin +120 -delete (or Elixir equivalent).

Zero vacuum pressure, zero WAL, zero index maintenance.

Migration path (if this gets greenlit)

1. Commit 1: Introduce Propagation.ScoresFile module. Dual-write: replace_scores still writes Postgres as today, but also writes the flat file. Map render still reads from Postgres. Verify file shape matches what the JS side wants.

2. Commit 2: Switch map-render read path to load from the flat file. Keep Postgres dual-write as a fallback. Measure wall-time improvement.

3. Commit 3: Stop writing to Postgres. Drop the propagation_scores table (or keep it empty for rollback). Point detail for forecast hours reads score from file; factors for f00 still hits a Postgres table (or a separate file).

Each commit is independently reversible.

Risk / pushback

• Added complexity of a second storage system. Today: Postgres only. After: Postgres for QSOs + ASOS + weather + factors, plus a flat-file store for grid scores. The cognitive overhead is real.
• Shared-volume assumptions couple web and worker pods to the same PVC. Worth revisiting the separate-worker-pod idea first; if that's not on the table, the flat-file approach is harder because every web pod needs a consistent view of the files.
• Queryability loss. Today, ad-hoc SQL against propagation_scores works for debugging. After: need to boot a Python/DuckDB shell to query files.
• Opportunity cost. If the Postgres tuning already lands us in under 90 min per chain, this rewrite is 2-3 weeks of work for perhaps a 2× improvement on a phase that's already fast enough. The plan survives regardless — we can always pick it up if scale demands it (finer grid resolution, more forecast hours, more bands).

When to revisit

Trigger conditions for picking this up:

• A single forecast-hour chain step still exceeds 5 min after all three Postgres wins are deployed
• A new feature requires 100m grid resolution (~8× more points → 4M rows per fh), which will blow through the Postgres wins immediately
• The Postgres pool becomes a bottleneck for other workers because propagation queue writes keep it saturated

Until one of those trips, the existing propagation_scores table + the shipped tuning is the right place to be.