diff --git a/docs/runbook_propagation_pipeline.md b/docs/runbook_propagation_pipeline.md index d4e6acbc..e6490d8e 100644 --- a/docs/runbook_propagation_pipeline.md +++ b/docs/runbook_propagation_pipeline.md @@ -18,8 +18,10 @@ prop-grid-rs (worker nodes) Postgres (10.0.15.30) prop pods (nod 1. Rust completes `{band, valid_time}` compute and writes the `.prop` file via atomic rename. (Readers also accept legacy `.ntms` files with `NTMS` magic during a rolling deploy.) -2. Rust emits `NOTIFY propagation_ready ''`. -3. `NotifyListener` on each Elixir pod receives it, prunes +2. Rust emits `NOTIFY propagation_ready '|'` + (pipe-delimited: run_time first, valid_time second — needed for + `retain_scores_window` which uses run_time as the retention anchor). +3. `NotifyListener` on each Elixir pod splits the payload, prunes ScoreCache entries outside the active forecast window, and broadcasts `propagation_updated` over PubSub. 4. LiveView clients see the broadcast and re-request scores. The @@ -91,6 +93,19 @@ to drain. All fetch/decode/score helpers moved to `rust/prop_grid_rs/src/pipeline.rs`; the Elixir module is retained only as the cron entry point, not as a fallback compute path. +## On-disk file formats + +Score files live under `/data/scores//.`: + +| Extension | Source | Writer | Description | +|-----------|--------|--------|-------------| +| `.prop` | HRRR (f00–f48) | `prop-grid-rs` (Rust) | Primary CONUS scores, 0.125° grid, 92k cells per band | +| `.hrdps.prop` | HRDPS (Canadian) | `hrdps_grid_worker` (Elixir) | Canadian coverage, merged at read with HRRR > HRDPS priority | +| `.gefs.prop` | GEFS (f024–f168) | `gefs_fetch_worker` (Elixir) | Extended-range ensemble mean, merged at read with HRRR > HRDPS > GEFS priority | +| `.ntms` | Legacy | (deprecated) | Old format, still accepted by readers during rolling deploys | + +All files use atomic rename on write (write to `.tmp.`, then `rename(2)`). + ## Monitoring signals - `avg_over_time(beam_memory_total_bytes[1h])` on prop pods — steady diff --git a/prediction.md b/prediction.md index 6bc131fa..b806324a 100644 --- a/prediction.md +++ b/prediction.md @@ -1,5 +1,7 @@ # Propagation Prediction System Design +> **Status (2026-08-01): All three phases are implemented.** The Rust pipeline (`prop-grid-rs`) fetches HRRR f00–f48 hourly, scores all 22 bands across the full CONUS grid, and writes per-valid_time `.prop` files. The map timeline shows the next 48 hours of real NWP-forecast scores (not persistence). `Propagation.point_forecast/3` provides the per-point sparkline. See `docs/runbook_propagation_pipeline.md` for operations; `scripts/validate_forecast.py` measures forecast skill by lead time. + ## The Opportunity No tool exists for microwave operators to answer: **"When should I get on the air in the next 24 hours?"**