Graham McIntire
4f82bd691e
ScoreCache held 437 entries × ~1.94 MiB each (~850 MiB per pod) of
{band_mhz, valid_time} grids — data already on NFS as compact .prop
files. NotifyListener and ScoreCacheReconciler both eagerly materialised
every band into ETS on each Rust completion / 60s sweep, so 5 hot
replicas wasted ~4.2 GiB of redundant cache and OOMed at 6 GiB.
Bound the cache to 32 entries with eviction by oldest valid_time, drop
the eager warm loops, and let LiveView callers lazy-fill via the
existing ScoresFile read path. ScoreCacheReconciler had no remaining
purpose and is deleted; runbook + prom_ex counters updated to match.
Steady-state cache footprint ~60 MiB per pod instead of ~850 MiB.
110 lines
4.6 KiB
Markdown
110 lines
4.6 KiB
Markdown
# Propagation pipeline runbook
|
||
|
||
The hourly propagation grid flows across two runtimes and a shared
|
||
NFS volume. This doc names every failure mode at that seam and says
|
||
how the system recovers.
|
||
|
||
```
|
||
prop-grid-rs (worker nodes) Postgres (10.0.15.30) prop pods (node1/2/3)
|
||
├─ fetch GRIB2 (NOMADS) ├─ grid_tasks (SKIP LOCKED) ├─ ScoreCache (ETS, capped LRU)
|
||
├─ decode ├─ LISTEN/NOTIFY channel ├─ NotifyListener
|
||
├─ score 22 bands × 92k │ "propagation_ready" ├─ Lazy disk read on miss
|
||
├─ write /data/scores/.../.prop │ └─ PubSub → LiveView
|
||
└─ NOTIFY propagation_ready ────┘
|
||
```
|
||
|
||
## Primary refresh path (happy case)
|
||
|
||
1. Rust completes `{band, valid_time}` compute and writes the `.prop`
|
||
file via atomic rename. (Readers also accept legacy `.ntms` files
|
||
with `NTMS` magic during a rolling deploy.)
|
||
2. Rust emits `NOTIFY propagation_ready '<iso valid_time>'`.
|
||
3. `NotifyListener` on each Elixir pod receives it, prunes
|
||
ScoreCache entries outside the active forecast window, and
|
||
broadcasts `propagation_updated` over PubSub.
|
||
4. LiveView clients see the broadcast and re-request scores. The
|
||
first request per `{band, valid_time}` lazy-fills `ScoreCache`
|
||
from the on-disk `.prop` file (~50–100 ms); subsequent requests
|
||
hit ETS until the LRU evicts.
|
||
|
||
Latency budget end-to-end: < 2 s on the warm path; < 200 ms added
|
||
on the first lazy fill.
|
||
|
||
## Failure modes
|
||
|
||
### FM1 — NOTIFY dropped on reconnect
|
||
|
||
**Symptom.** `{band, valid_time}` files land on disk but
|
||
LiveView clients don't auto-refresh until the user interacts.
|
||
|
||
**Why.** Postgres queues LISTEN notifications per-connection and
|
||
discards any that are unread on disconnect. A DB restart, a network
|
||
blip, or the Postgrex.Notifications connection hitting its idle
|
||
timeout all drop whatever queued notifies were in flight.
|
||
|
||
**Recovery.** Lazy fill: the next `/map` request reads the on-disk
|
||
`.prop` file directly via `Propagation.scores_at_fresh/3`. Stale
|
||
ETS entries get evicted by NotifyListener's `prune_outside_window`
|
||
on the next NOTIFY that does land.
|
||
|
||
### FM2 — NFS stale read racing Rust write
|
||
|
||
**Symptom.** `ScoresFile.read/2` returns `{:error, reason}` when
|
||
the reader opens the file exactly between Rust's `rename(2)` of the
|
||
`.tmp.<uniq>` file and the `fsync` landing on the NFS server.
|
||
|
||
**Why.** The rename is atomic on the same NFSv4 filesystem, but the
|
||
NFS client cache can briefly serve a stale directory listing or a
|
||
no-longer-existing path.
|
||
|
||
**Recovery.** LiveView callers retry on the next interaction; the
|
||
read settles after NFS consistency converges (sub-second).
|
||
|
||
### FM3 — Cluster partition
|
||
|
||
**Symptom.** Pods on one side of a cluster partition don't see
|
||
the other side's `propagation_updated` PubSub messages.
|
||
|
||
**Why.** Phoenix.PubSub fans messages over the BEAM cluster mesh.
|
||
A libcluster partition stops the message reaching partitioned peers.
|
||
|
||
**Recovery.** Each pod reads scores from the shared NFS mount on
|
||
demand, so partitions only delay LiveView refresh — they never cause
|
||
divergent answers.
|
||
|
||
### FM4 — Rust worker dead
|
||
|
||
**Symptom.** `grid_tasks` rows with `status = 'queued'` accumulate;
|
||
no new `.prop` files land; `/map` serves the last successful run's
|
||
scores.
|
||
|
||
**Why.** Deployment bug, Rust panic, OOM on a worker node, or
|
||
disk-full on `/data`.
|
||
|
||
**Recovery.** Manual. `kubectl -n prop logs deploy/prop-grid-rs` for
|
||
cause. The Elixir side is read-only against the grid — it does not
|
||
regenerate scores from within the BEAM since the extraction in
|
||
`65693ed`. `lib/microwaveprop/workers/propagation_grid_worker.ex`
|
||
still exists as a seed worker: the hourly cron fires it with empty
|
||
args and it inserts 19 `grid_tasks` rows (f00 + f01..f18) for Rust
|
||
to drain. All fetch/decode/score helpers moved to
|
||
`rust/prop_grid_rs/src/pipeline.rs`; the Elixir module is retained
|
||
only as the cron entry point, not as a fallback compute path.
|
||
|
||
## Monitoring signals
|
||
|
||
- `avg_over_time(beam_memory_total_bytes[1h])` on prop pods — steady
|
||
climb above 1 GiB means the ScoreCache LRU cap isn't holding (see
|
||
`ScoreCache.max_entries/0`).
|
||
- `microwaveprop.propagation.scores_at.cache.count{hit=...}` — if the
|
||
miss rate is consistently above ~30 % the LRU cap may be too small
|
||
for the active session count.
|
||
- `grid_tasks` queue depth in Prometheus — should drain to 0 between
|
||
hourly cron ticks.
|
||
|
||
## Test coverage
|
||
|
||
- `test/microwaveprop/propagation/score_cache_test.exs` covers the
|
||
LRU eviction path.
|
||
- Partition recovery (FM3) is untested in CI — requires multi-node
|
||
cluster and libcluster orchestration.
|