# DuckDB / Flat-File Propagation Scores Storage Plan **Goal:** Replace Postgres-backed `propagation_scores` with per-`(band_mhz, valid_time)` on-disk files (DuckDB or raw binary), so the hot write path becomes a single file write instead of 475k Postgres rows through indexes + WAL. **Status:** Speculative architectural alternative. Do not execute until the already-shipped Postgres wins (DELETE+COPY, skip-factors, UNLOGGED) have been measured in prod. If those three drop the scoring+upsert phase below ~60s per forecast hour, this rewrite is likely unnecessary. --- ## Why consider this Each forecast-hour write to `propagation_scores` is: - 475k rows (95k grid points × 5 bands) - Passes through 4 indexes (PK UUID, unique tuple, valid_time, (band_mhz, valid_time)) - Even with DELETE+COPY and UNLOGGED, Postgres overhead — toast pages, page locks, vacuum state, MVCC tuple housekeeping — adds per-row cost that a flat array layout avoids entirely - Shapes a dense numeric grid into a row-oriented representation the map never needs The map render path also doesn't want individual rows. It wants a dense `(lat, lon) → score` array for a single `(band, valid_time)`. SQL is the wrong shape. ## Approach options ### Option A — Parquet files, read via DuckDB One Parquet file per `(band_mhz, valid_time)`: ``` /data/scores/10000/2026-04-14T18-00-00Z.parquet /data/scores/24000/2026-04-14T18-00-00Z.parquet ... ``` - Columns: `lat`, `lon`, `score` (plus `factors` only on f00 files) - ~95k rows × 3 numeric columns + JSONB on f00 ≈ 1-2 MB per f00 file, ~400 KB per fh≥1 file - Written once per forecast hour via Elixir → DuckDB WAL-less writer (`COPY ... TO 'file.parquet'`) or a Rust/Elixir native Parquet encoder - Read on the map side via [`duckdbex`](https://hex.pm/packages/duckdbex) (Elixir NIF wrapper) or a Rust sidecar Pros: - Columnar compression: a single Parquet file per hour is ~300-500 KB compressed - DuckDB can query across files: `SELECT score FROM 'scores/10000/*.parquet' WHERE lat BETWEEN ... AND ...` - Files are portable — easy to debug, copy, inspect with `duckdb-cli` - Pruning is `rm` — instant Cons: - Add a new dependency (`duckdbex` or a Rust NIF) - Concurrent writers across multiple pods need file coordination (atomic rename, or a single writer pod) - `factors` for f00 still needs somewhere to live — either embedded in the Parquet or a separate lookup file ### Option B — Raw binary mmap files One file per `(band_mhz, valid_time)`, laid out as: ``` header: 16 bytes (magic + lat_min, lon_min, step, n_rows, n_cols) scores: n_rows × n_cols × 1 byte (score 0-100) ``` - ~95k × 1 byte ≈ 95 KB per file uncompressed - Written in Elixir via `File.write/2` on a binary blob - Read on the map side via `:ets.new(:mmap)` / `File.read/1` cached per `(band, vt)` Pros: - Simplest possible layout - Writes are ~20 ms per file - No dependencies at all - Matches the canvas heatmap's consumption shape exactly Cons: - Point-detail (single-cell lookup) needs an index by (lat, lon) — easy with the grid step but less ergonomic than SQL - `factors` needs a separate store (another binary file, or keep it in Postgres) - No ad-hoc query capability (can't ask "what was the score at 35.0/-97.0 two hours ago" without loading the file) ### Option C — DuckDB on persistent disk Single `.duckdb` file under `/data/scores.duckdb`: - One table with the same schema as `propagation_scores` - DuckDB is columnar, compressed, and serializes writes via a single database file - Writes via `INSERT INTO scores VALUES ...` batched - Reads via direct SQL from Elixir Pros: - Least migration friction — keeps a SQL-ish API - Columnar storage naturally compresses `score` columns - DuckDB writes are dramatically faster than Postgres for bulk inserts (no WAL, no per-row overhead) Cons: - DuckDB is single-writer. Multi-pod clusters need to funnel writes through one pod (the propagation grid worker pod — which is exactly the worker-pod-isolation idea discussed earlier in this session) - File corruption on crash: DuckDB has improved crash recovery but it's not Postgres-grade - Requires `duckdbex` NIF ## Recommendation If the already-shipped Postgres wins aren't enough, start with **Option B** (raw binary files). It's the lowest-risk, highest-impact change because: 1. The map's canvas heatmap *already* wants a dense score array — a binary file is a closer shape match than SQL rows 2. No new dependencies — `File.write/2` and `File.read/1` are enough 3. Concurrent write safety is a solved problem: write to a temp file, `File.rename/2` atomically 4. Reverting is easy: keep `propagation_scores` populated from the flat file asynchronously during a migration period The `factors` storage question is the only thing that forces picking between Option A and Option B: - If we're fine with **no factors for forecast hours** (already the case after the skip-factors change), factors for f00 stay in Postgres and the hot path is file-only. Option B works as-is. - If we want factors for all hours, Option A (Parquet) is cleaner because factors can ride on the same file. ## Architectural prerequisites Before this is worth doing: 1. **Separate worker pod**. Multi-writer coordination on a single file is painful; a single "propagation" pod that owns all writes is much simpler. The isolation idea from earlier in this session dovetails here. 2. **Shared volume or single-node**. Files need to live on a PVC shared between the worker and web pods, or the web pods need a cached-read path that pulls the file from the worker pod via HTTP. The existing `/data` NFS mount is already shared across all pods, which makes `/data/scores` free. 3. **Baseline perf measurements** from the already-shipped Postgres changes. Don't do this if the map already feels fast. ## Read-path changes ### Map render (bulk) Current: ```elixir Propagation.scores_at(band_mhz, valid_time, bounds) # → list of %{lat, lon, score} maps from Postgres ``` New: ```elixir Propagation.ScoresFile.load(band_mhz, valid_time) # → %{lat_min, lon_min, step, rows, cols, data: <>} ``` The JS hook receives the binary directly (base64 or bytestring in an assign) and draws the canvas without an intermediate list traversal. ### Point detail (single-cell) Current: ```elixir Propagation.point_detail(band, lat, lon, valid_time) # → %{score, factors, ...} ``` New: ```elixir Propagation.ScoresFile.point(band_mhz, valid_time, lat, lon) # → %{score} Propagation.FactorsStore.for_point(lat, lon, valid_time) # → %{factors} (still Postgres-backed, only f00) ``` ### Scorer diff Current: reads latest f00 rows with factors from Postgres. Unchanged — factors stay in Postgres. ### ScoreCache Current: ETS cache of `(band, vt) → list-of-scores`. Unchanged — can wrap the file-load path the same way. ## Write-path changes `PropagationGridWorker.run_chain_step/2` currently calls: ```elixir scores = compute_scores(grid_data, valid_time, forecast_hour) Propagation.replace_scores(scores, valid_time) ``` Replace with: ```elixir scores = compute_scores(grid_data, valid_time, forecast_hour) Propagation.ScoresFile.write!(band_mhz, valid_time, scores) # Still write factors to Postgres for f00 only if forecast_hour == 0, do: Propagation.replace_factors(factor_rows, valid_time) ``` Expected timings: ~10-50 ms per band per fh vs 4-5 seconds per band per fh today. The scoring CPU cost stays the same; only the storage phase collapses. ## Pruning Current: 15-minute cron running `DELETE WHERE valid_time < now - 2h`. New: 15-minute cron running `find /data/scores -mmin +120 -delete` (or Elixir equivalent). Zero vacuum pressure, zero WAL, zero index maintenance. ## Migration path (if this gets greenlit) 1. **Commit 1**: Introduce `Propagation.ScoresFile` module. Dual-write: `replace_scores` still writes Postgres as today, but also writes the flat file. Map render still reads from Postgres. Verify file shape matches what the JS side wants. 2. **Commit 2**: Switch map-render read path to load from the flat file. Keep Postgres dual-write as a fallback. Measure wall-time improvement. 3. **Commit 3**: Stop writing to Postgres. Drop the `propagation_scores` table (or keep it empty for rollback). Point detail for forecast hours reads score from file; factors for f00 still hits a Postgres table (or a separate file). Each commit is independently reversible. ## Risk / pushback - **Added complexity** of a second storage system. Today: Postgres only. After: Postgres for QSOs + ASOS + weather + factors, plus a flat-file store for grid scores. The cognitive overhead is real. - **Shared-volume assumptions** couple web and worker pods to the same PVC. Worth revisiting the separate-worker-pod idea first; if that's not on the table, the flat-file approach is harder because every web pod needs a consistent view of the files. - **Queryability loss**. Today, ad-hoc SQL against `propagation_scores` works for debugging. After: need to boot a Python/DuckDB shell to query files. - **Opportunity cost**. If the Postgres tuning already lands us in under 90 min per chain, this rewrite is 2-3 weeks of work for perhaps a 2× improvement on a phase that's already fast enough. The plan survives regardless — we can always pick it up if scale demands it (finer grid resolution, more forecast hours, more bands). ## When to revisit Trigger conditions for picking this up: - A single forecast-hour chain step still exceeds 5 min after all three Postgres wins are deployed - A new feature requires 100m grid resolution (~8× more points → 4M rows per fh), which will blow through the Postgres wins immediately - The Postgres pool becomes a bottleneck for other workers because `propagation` queue writes keep it saturated Until one of those trips, the existing `propagation_scores` table + the shipped tuning is the right place to be.