# Propagation pipeline runbook

The hourly propagation grid flows across two runtimes and a shared
NFS volume. This doc names every failure mode at that seam and says
how the system recovers.

```
prop-grid-rs (worker nodes)      Postgres (talos5 — 10.0.15.30)  prop pods (node1/2/3)
 ├─ fetch GRIB2 (NOMADS)         ├─ grid_tasks (SKIP LOCKED)     ├─ ScoreCache (ETS)
 ├─ decode                       ├─ LISTEN/NOTIFY channel        ├─ NotifyListener
 ├─ score 22 bands × 92k         │    "propagation_ready"        ├─ ScoreCacheReconciler (60s)
 ├─ write /data/scores/.../.prop │                               └─ PubSub → LiveView
 └─ NOTIFY propagation_ready ────┘
```

## Primary refresh path (happy case)

1. Rust completes `{band, valid_time}` compute and writes the `.prop`
   file via atomic rename. (Readers also accept legacy `.ntms` files
   with `NTMS` magic during a rolling deploy.)
2. Rust emits `NOTIFY propagation_ready '<iso valid_time>'`.
3. `NotifyListener` on each Elixir pod receives it and calls
   `Propagation.warm_cache_and_broadcast/2` which reads the file once
   and PubSub-broadcasts the payload to every peer.
4. Each pod's `ScoreCache` receives the message and writes into its
   local ETS table. `/map` clients are notified via the
   `"propagation:updated"` topic.

Latency budget end-to-end: < 2s.

## Failure modes

### FM1 — NOTIFY dropped on reconnect

**Symptom.** `{band, valid_time}` files land on disk but ScoreCache
stays cold until the first lazy miss.

**Why.** Postgres queues LISTEN notifications per-connection and
discards any that are unread on disconnect. A DB restart, a network
blip, or the Postgrex.Notifications connection hitting its idle
timeout all drop whatever queued notifies were in flight.

**Recovery.** `ScoreCacheReconciler` sweeps `/data/scores` every 60s
per-pod and warms missing `{band, valid_time}` pairs from disk into
local ETS. Each pod reconciles independently — no cluster messaging
required.

### FM2 — NotifyListener not running locally

**Symptom.** One pod is cold while peers are warm. `/map` load time
on that pod is ~150ms slower than average until the reconciler runs.

**Why.** The listener isn't currently part of the supervision tree
(see `lib/microwaveprop/application.ex`). When it is re-added, early
startup or Postgrex.Notifications boot failure can leave it
unsubscribed for several seconds while Rust writes land.

**Recovery.** Same as FM1 — reconciler catches up within one tick.

### FM3 — NFS stale read racing Rust write

**Symptom.** `ScoresFile.read/2` returns `{:error, reason}` when
the reader opens the file exactly between Rust's `rename(2)` of the
`.tmp.<uniq>` file and the `fsync` landing on the NFS server.

**Why.** The rename is atomic on the same NFSv4 filesystem, but the
NFS client cache can briefly serve a stale directory listing or a
no-longer-existing path.

**Recovery.** Reconciler calls `ScoresFile.read/2` directly and
pattern-matches `{:error, _} -> :error`, logging at debug and
skipping the key. The next tick re-reads the file after NFS
consistency settles. Lazy-miss path
(`Propagation.scores_at_fetch/3`) has historically surfaced this as
a LiveView error — the reconciler warms the cache before that miss
happens.

### FM4 — Cluster partition splits ScoreCache broadcast

**Symptom.** Pods on one side of a cluster partition have up-to-date
cache; pods on the other side don't, even though both can read NFS
and Postgres.

**Why.** `ScoreCache.broadcast_put/3` fans the payload out over
PubSub. A libcluster partition stops the message reaching partitioned
peers.

**Recovery.** Each pod's reconciler reads from NFS directly, so it
converges regardless of PubSub reachability. Partition does not
cause divergent scores — only transient latency until the next
sweep.

### FM5 — Rust worker dead

**Symptom.** `grid_tasks` rows with `status = 'queued'` accumulate;
no new `.prop` files land; `/map` serves the last successful run's
scores.

**Why.** Deployment bug, Rust panic, OOM on a worker node, or
disk-full on `/data`.

**Recovery.** Manual. `kubectl -n prop logs deploy/prop-grid-rs` for
cause. The Elixir side is read-only against the grid — it does not
regenerate scores from within the BEAM since the extraction in
`65693ed`. `lib/microwaveprop/workers/propagation_grid_worker.ex`
still exists as a seed worker: the hourly cron fires it with empty
args and it inserts 19 `grid_tasks` rows (f00 + f01..f18) for Rust
to drain. All fetch/decode/score helpers moved to
`rust/prop_grid_rs/src/pipeline.rs`; the Elixir module is retained
only as the cron entry point, not as a fallback compute path.

## Monitoring signals

- `avg_over_time(beam_memory_total_bytes[1h])` on prop pods — steady
  climb above 1 GiB means the cache isn't pruning (see
  `ScoreCache.prune_older_than/1` in `NotifyListener`).
- Reconciler log line `"ScoreCacheReconciler: warmed N"` — if N is
  consistently non-zero every minute, NOTIFY is dropping and
  NotifyListener wants investigation.
- `grid_tasks` queue depth in Prometheus — should drain to 0 between
  hourly cron ticks.

## Test coverage

- `test/microwaveprop/propagation/score_cache_reconciler_test.exs`
  covers FM1 and FM3 (missing-file skip).
- Partition recovery (FM4) is untested in CI — requires multi-node
  cluster and libcluster orchestration.