fix(review): address code-reviewer findings

Fixes flagged by the code-reviewer agent's pass over the session's
commits (cc9220b..7b78a25):

- Propagation.warm_cache_and_broadcast/2 now uses ScoresFile.read/2
  directly and returns {:ok, :ok} | {:error, :enoent | :invalid_format}.
  Previously it called ScoresFile.read_bounds/3 which silently returns
  [] on missing/corrupt files, poisoning ScoreCache with an empty grid
  that the reconciler couldn't heal.
- NotifyListener.warm_band/2 and ScoreCacheReconciler.warm_one/2 now
  pattern-match {:error, reason} and skip (log, return :error) instead
  of caching empty. rescue clauses kept as defense-in-depth for
  unexpected faults in PubSub.broadcast / ETS writes.
- ScoresFile.extract_points/2 promoted to @doc public — callers that
  need to distinguish missing file from empty grid can feed read/2
  payloads here themselves.
- Weather.build_grid_cache_row/4: replaced || fallbacks with prefer/3
  helper (Map.fetch) so a legitimate persisted ducting_detected: false
  is not clobbered by a derived-from-sounding true.
- Weather.hrrr_data_fully_present?/1 @spec tightened from map() to
  Contact.t() | field-constrained map, plus is_nil(qso_timestamp)
  guard so callers with partial contacts get a clean false rather
  than a HrrrClient.nearest_hrrr_hour/1 crash.
- AdminTaskWorker.native_derive bulk-UPDATE: chunk reduced 2000 → 500
  and wrapped in try/rescue with per-row fallback on Postgrex errors
  so a single bad row doesn't kill the remaining 1999 in its chunk.
- Runbook FM3 rewritten to match the actual code path (no rescue;
  explicit {:error, _} pattern match). FM5 clarifies the surviving
  PropagationGridWorker is a cron-fired seed worker, not a fallback
  compute path.
- Dialyzer: strict flags added (:error_handling, :unknown,
  :unmatched_returns, :extra_return, :missing_return). Baseline
  emitted 130 warnings, mostly discarded Task.start/PubSub/Logger
  returns; a follow-up will tighten those and backfill @specs.

New tests:
- ScoreCacheReconciler GenServer lifecycle: run_on_start true/false,
  interval_ms rescheduling, info-level log line.
- Weather.hrrr_data_fully_present?/1: nil qso_timestamp returns false.
- Weather.build_grid_cache_rows/2: explicit ducting_detected: false
  on profile beats derived true from sounding params.
- RadarFrameWorker: pins the NexradClient "NEXRAD n0q HTTP <code>"
  error string contract so permanent_error?/1 classification doesn't
  silently regress if the client's error format changes.

2026-04-21 10:09:46 -05:00

5.2 KiB

Raw Blame History

Propagation pipeline runbook

The hourly propagation grid flows across two runtimes and a shared NFS volume. This doc names every failure mode at that seam and says how the system recovers.

prop-grid-rs (talos5)            Postgres (10.0.15.30)           prop pods (node1/2/3)
 ├─ fetch GRIB2 (NOMADS)         ├─ grid_tasks (SKIP LOCKED)     ├─ ScoreCache (ETS)
 ├─ decode                       ├─ LISTEN/NOTIFY channel        ├─ NotifyListener
 ├─ score 22 bands × 92k         │    "propagation_ready"        ├─ ScoreCacheReconciler (60s)
 ├─ write /data/scores/.../.ntms │                               └─ PubSub → LiveView
 └─ NOTIFY propagation_ready ────┘

Primary refresh path (happy case)

Rust completes {band, valid_time} compute and writes the .ntms file via atomic rename.
Rust emits NOTIFY propagation_ready '<iso valid_time>'.
NotifyListener on each Elixir pod receives it and calls Propagation.warm_cache_and_broadcast/2 which reads the file once and PubSub-broadcasts the payload to every peer.
Each pod's ScoreCache receives the message and writes into its local ETS table. /map clients are notified via the "propagation:updated" topic.

Latency budget end-to-end: < 2s.

Failure modes

FM1 — NOTIFY dropped on reconnect

Symptom. {band, valid_time} files land on disk but ScoreCache stays cold until the first lazy miss.

Why. Postgres queues LISTEN notifications per-connection and discards any that are unread on disconnect. A DB restart, a network blip, or the Postgrex.Notifications connection hitting its idle timeout all drop whatever queued notifies were in flight.

Recovery. ScoreCacheReconciler sweeps /data/scores every 60s per-pod and warms missing {band, valid_time} pairs from disk into local ETS. Each pod reconciles independently — no cluster messaging required.

FM2 — NotifyListener not running locally

Symptom. One pod is cold while peers are warm. /map load time on that pod is ~150ms slower than average until the reconciler runs.

Why. The listener isn't currently part of the supervision tree (see lib/microwaveprop/application.ex). When it is re-added, early startup or Postgrex.Notifications boot failure can leave it unsubscribed for several seconds while Rust writes land.

Recovery. Same as FM1 — reconciler catches up within one tick.

FM3 — NFS stale read racing Rust write

Symptom. ScoresFile.read/2 returns {:error, reason} when the reader opens the file exactly between Rust's rename(2) of the .tmp.<uniq> file and the fsync landing on the NFS server.

Why. The rename is atomic on the same NFSv4 filesystem, but the NFS client cache can briefly serve a stale directory listing or a no-longer-existing path.

Recovery. Reconciler calls ScoresFile.read/2 directly and pattern-matches {:error, _} -> :error, logging at debug and skipping the key. The next tick re-reads the file after NFS consistency settles. Lazy-miss path (Propagation.scores_at_fetch/3) has historically surfaced this as a LiveView error — the reconciler warms the cache before that miss happens.

FM4 — Cluster partition splits ScoreCache broadcast

Symptom. Pods on one side of a cluster partition have up-to-date cache; pods on the other side don't, even though both can read NFS and Postgres.

Why. ScoreCache.broadcast_put/3 fans the payload out over PubSub. A libcluster partition stops the message reaching partitioned peers.

Recovery. Each pod's reconciler reads from NFS directly, so it converges regardless of PubSub reachability. Partition does not cause divergent scores — only transient latency until the next sweep.

FM5 — Rust worker dead

Symptom. grid_tasks rows with status = 'queued' accumulate; no new .ntms files land; /map serves the last successful run's scores.

Why. Deployment bug, Rust panic, OOM on talos5 (unlikely with 32 GB headroom), or disk-full on /data.

Recovery. Manual. kubectl -n prop logs deploy/prop-grid-rs for cause. The Elixir side is read-only against the grid — it does not regenerate scores from within the BEAM since the extraction in 65693ed. lib/microwaveprop/workers/propagation_grid_worker.ex still exists as a seed worker: the hourly cron fires it with empty args and it inserts 19 grid_tasks rows (f00 + f01..f18) for Rust to drain. All fetch/decode/score helpers moved to rust/prop_grid_rs/src/pipeline.rs; the Elixir module is retained only as the cron entry point, not as a fallback compute path.

Monitoring signals

avg_over_time(beam_memory_total_bytes[1h]) on prop pods — steady climb above 1 GiB means the cache isn't pruning (see ScoreCache.prune_older_than/1 in NotifyListener).
Reconciler log line "ScoreCacheReconciler: warmed N" — if N is consistently non-zero every minute, NOTIFY is dropping and NotifyListener wants investigation.
grid_tasks queue depth in Prometheus — should drain to 0 between hourly cron ticks.

Test coverage

test/microwaveprop/propagation/score_cache_reconciler_test.exs covers FM1 and FM3 (missing-file skip).
Partition recovery (FM4) is untested in CI — requires multi-node cluster and libcluster orchestration.

5.2 KiB Raw Blame History Unescape Escape