Propagation pipeline runbook

The hourly propagation grid flows across two runtimes and a shared NFS volume. This doc names every failure mode at that seam and says how the system recovers.

prop-grid-rs (worker nodes)      Postgres (10.0.15.30)            prop pods (node1/2/3)
 ├─ fetch GRIB2 (NOMADS)         ├─ grid_tasks (SKIP LOCKED)     ├─ ScoreCache (ETS, capped LRU)
 ├─ decode                       ├─ LISTEN/NOTIFY channel        ├─ NotifyListener
 ├─ score 22 bands × 92k         │    "propagation_ready"        ├─ Lazy disk read on miss
 ├─ write /data/scores/.../.prop │                               └─ PubSub → LiveView
 └─ NOTIFY propagation_ready ────┘

Primary refresh path (happy case)

Rust completes {band, valid_time} compute and writes the .prop file via atomic rename. (Readers also accept legacy .ntms files with NTMS magic during a rolling deploy.)
Rust emits NOTIFY propagation_ready '<run_time_iso>|<valid_time_iso>' (pipe-delimited: run_time first, valid_time second — needed for retain_scores_window which uses run_time as the retention anchor).
NotifyListener on each Elixir pod splits the payload, prunes ScoreCache entries outside the active forecast window, and broadcasts propagation_updated over PubSub.
LiveView clients see the broadcast and re-request scores. The first request per {band, valid_time} lazy-fills ScoreCache from the on-disk .prop file (~50–100 ms); subsequent requests hit ETS until the LRU evicts.

Latency budget end-to-end: < 2 s on the warm path; < 200 ms added on the first lazy fill.

Failure modes

FM1 — NOTIFY dropped on reconnect

Symptom. {band, valid_time} files land on disk but LiveView clients don't auto-refresh until the user interacts.

Why. Postgres queues LISTEN notifications per-connection and discards any that are unread on disconnect. A DB restart, a network blip, or the Postgrex.Notifications connection hitting its idle timeout all drop whatever queued notifies were in flight.

Recovery. Lazy fill: the next /map request reads the on-disk .prop file directly via Propagation.scores_at_fresh/3. Stale ETS entries get evicted by NotifyListener's prune_outside_window on the next NOTIFY that does land.

FM2 — NFS stale read racing Rust write

Symptom. ScoresFile.read/2 returns {:error, reason} when the reader opens the file exactly between Rust's rename(2) of the .tmp.<uniq> file and the fsync landing on the NFS server.

Why. The rename is atomic on the same NFSv4 filesystem, but the NFS client cache can briefly serve a stale directory listing or a no-longer-existing path.

Recovery. LiveView callers retry on the next interaction; the read settles after NFS consistency converges (sub-second).

FM3 — Cluster partition

Symptom. Pods on one side of a cluster partition don't see the other side's propagation_updated PubSub messages.

Why. Phoenix.PubSub fans messages over the BEAM cluster mesh. A libcluster partition stops the message reaching partitioned peers.

Recovery. Each pod reads scores from the shared NFS mount on demand, so partitions only delay LiveView refresh — they never cause divergent answers.

FM4 — Rust worker dead

Symptom. grid_tasks rows with status = 'queued' accumulate; no new .prop files land; /map serves the last successful run's scores.

Why. Deployment bug, Rust panic, OOM on a worker node, or disk-full on /data.

Recovery. Manual. kubectl -n prop logs deploy/prop-grid-rs for cause. The Elixir side is read-only against the grid — it does not regenerate scores from within the BEAM since the extraction in 65693ed. lib/microwaveprop/workers/propagation_grid_worker.ex still exists as a seed worker: the hourly cron fires it with empty args and it inserts 19 grid_tasks rows (f00 + f01..f18) for Rust to drain. All fetch/decode/score helpers moved to rust/prop_grid_rs/src/pipeline.rs; the Elixir module is retained only as the cron entry point, not as a fallback compute path.

On-disk file formats

Score files live under /data/scores/<band_mhz>/<valid_time_iso>.<ext>:

Extension	Source	Writer	Description
`.prop`	HRRR (f00–f48)	`prop-grid-rs` (Rust)	Primary CONUS scores, 0.125° grid, 92k cells per band
`.hrdps.prop`	HRDPS (Canadian)	`hrdps_grid_worker` (Elixir)	Canadian coverage, merged at read with HRRR > HRDPS priority
`.gefs.prop`	GEFS (f024–f168)	`gefs_fetch_worker` (Elixir)	Extended-range ensemble mean, merged at read with HRRR > HRDPS > GEFS priority
`.ntms`	Legacy	(deprecated)	Old format, still accepted by readers during rolling deploys

All files use atomic rename on write (write to .tmp.<uuid>, then rename(2)).

Monitoring signals

avg_over_time(beam_memory_total_bytes[1h]) on prop pods — steady climb above 1 GiB means the ScoreCache LRU cap isn't holding (see ScoreCache.max_entries/0).
microwaveprop.propagation.scores_at.cache.count{hit=...} — if the miss rate is consistently above ~30 % the LRU cap may be too small for the active session count.
grid_tasks queue depth in Prometheus — should drain to 0 between hourly cron ticks.

Test coverage

test/microwaveprop/propagation/score_cache_test.exs covers the LRU eviction path.
Partition recovery (FM3) is untested in CI — requires multi-node cluster and libcluster orchestration.

5.5 KiB Raw Blame History Unescape Escape