prop/rust/prop_grid_rs
Graham McIntire ee7809e17b
hardening: Rust worker circuit-breaker + delete wall-clock sleep from beacon test
**Rust workers exit after 30 consecutive claim_next failures.**
Both `prop-grid-rs` and `hrrr-point-worker` ran a `while` loop that
slept 10s and retried forever on `claim_next` errors. If Postgres
went unreachable, pods would stay alive but useless — no circuit
breaker, no liveness probe, just an infinite stream of error logs.
Now we increment a consecutive-failure counter and exit with an
error once it hits 30 (~5 minutes of DB unavailability). k8s
restart policy then cycles the pod, which both triggers an
operator alert and gives us a fresh sqlx pool on the way back up.
A successful claim (Some or None) resets the counter so one
flaky query doesn't poison it.

**BeaconMonitorsTest: drop the Process.sleep(1100).** The test
relied on wall-clock gaps to disambiguate `inserted_at` values on
the `:utc_datetime` (second-precision) timestamps — one full
second of sleep per run, and on shared CI it could still flake
when both inserts landed in the same second. Replaced with an
explicit `Repo.update_all` that back-dates the first row by one
second. 1.1s faster, deterministic.
2026-04-21 18:02:46 -05:00
..
src hardening: Rust worker circuit-breaker + delete wall-clock sleep from beacon test 2026-04-21 18:02:46 -05:00
tests feat(grid-rs): Rust worker for HRRR f01..f18 propagation chain 2026-04-19 15:42:49 -05:00
.gitignore feat(grid-rs): Rust worker for HRRR f01..f18 propagation chain 2026-04-19 15:42:49 -05:00
Cargo.lock feat(rust): OTLP trace export to cluster OTel Collector 2026-04-20 11:59:49 -05:00
Cargo.toml feat(rust): OTLP trace export to cluster OTel Collector 2026-04-20 11:59:49 -05:00
Dockerfile fix(grid-rs): build wgrib2 in-tree instead of pulling from prop:latest 2026-04-20 08:33:53 -05:00