prop/docs/runbook_propagation_pipeline.md

Propagation pipeline runbook

The hourly propagation grid flows across two runtimes and a shared NFS volume. This doc names every failure mode at that seam and says how the system recovers.

prop-grid-rs (worker nodes)      Postgres (10.0.15.30)            prop pods (node1/2/3)
 ├─ fetch GRIB2 (NOMADS)         ├─ grid_tasks (SKIP LOCKED)     ├─ ScoreCache (ETS, capped LRU)
 ├─ decode                       ├─ LISTEN/NOTIFY channel        ├─ NotifyListener
 ├─ score 22 bands × 92k         │    "propagation_ready"        ├─ Lazy disk read on miss
 ├─ write /data/scores/.../.prop │                               └─ PubSub → LiveView
 └─ NOTIFY propagation_ready ────┘

Primary refresh path (happy case)

1. Rust completes {band, valid_time} compute and writes the .prop file via atomic rename. (Readers also accept legacy .ntms files with NTMS magic during a rolling deploy.)
2. Rust emits NOTIFY propagation_ready '<iso valid_time>'.
3. NotifyListener on each Elixir pod receives it, prunes ScoreCache entries outside the active forecast window, and broadcasts propagation_updated over PubSub.
4. LiveView clients see the broadcast and re-request scores. The first request per {band, valid_time} lazy-fills ScoreCache from the on-disk .prop file (~50–100 ms); subsequent requests hit ETS until the LRU evicts.

Latency budget end-to-end: < 2 s on the warm path; < 200 ms added on the first lazy fill.

Failure modes

FM1 — NOTIFY dropped on reconnect

Symptom. {band, valid_time} files land on disk but LiveView clients don't auto-refresh until the user interacts.

Why. Postgres queues LISTEN notifications per-connection and discards any that are unread on disconnect. A DB restart, a network blip, or the Postgrex.Notifications connection hitting its idle timeout all drop whatever queued notifies were in flight.

Recovery. Lazy fill: the next /map request reads the on-disk .prop file directly via Propagation.scores_at_fresh/3. Stale ETS entries get evicted by NotifyListener's prune_outside_window on the next NOTIFY that does land.

FM2 — NFS stale read racing Rust write

Symptom. ScoresFile.read/2 returns {:error, reason} when the reader opens the file exactly between Rust's rename(2) of the .tmp.<uniq> file and the fsync landing on the NFS server.

Why. The rename is atomic on the same NFSv4 filesystem, but the NFS client cache can briefly serve a stale directory listing or a no-longer-existing path.

Recovery. LiveView callers retry on the next interaction; the read settles after NFS consistency converges (sub-second).

FM3 — Cluster partition

Symptom. Pods on one side of a cluster partition don't see the other side's propagation_updated PubSub messages.

Why. Phoenix.PubSub fans messages over the BEAM cluster mesh. A libcluster partition stops the message reaching partitioned peers.

Recovery. Each pod reads scores from the shared NFS mount on demand, so partitions only delay LiveView refresh — they never cause divergent answers.

FM4 — Rust worker dead

Symptom. grid_tasks rows with status = 'queued' accumulate; no new .prop files land; /map serves the last successful run's scores.

Why. Deployment bug, Rust panic, OOM on a worker node, or disk-full on /data.

Recovery. Manual. kubectl -n prop logs deploy/prop-grid-rs for cause. The Elixir side is read-only against the grid — it does not regenerate scores from within the BEAM since the extraction in 65693ed. lib/microwaveprop/workers/propagation_grid_worker.ex still exists as a seed worker: the hourly cron fires it with empty args and it inserts 19 grid_tasks rows (f00 + f01..f18) for Rust to drain. All fetch/decode/score helpers moved to rust/prop_grid_rs/src/pipeline.rs; the Elixir module is retained only as the cron entry point, not as a fallback compute path.

Monitoring signals

• avg_over_time(beam_memory_total_bytes[1h]) on prop pods — steady climb above 1 GiB means the ScoreCache LRU cap isn't holding (see ScoreCache.max_entries/0).
• microwaveprop.propagation.scores_at.cache.count{hit=...} — if the miss rate is consistently above ~30 % the LRU cap may be too small for the active session count.
• grid_tasks queue depth in Prometheus — should drain to 0 between hourly cron ticks.

Test coverage

• test/microwaveprop/propagation/score_cache_test.exs covers the LRU eviction path.
• Partition recovery (FM3) is untested in CI — requires multi-node cluster and libcluster orchestration.