Graham McIntire
ee7809e17b
**Rust workers exit after 30 consecutive claim_next failures.** Both `prop-grid-rs` and `hrrr-point-worker` ran a `while` loop that slept 10s and retried forever on `claim_next` errors. If Postgres went unreachable, pods would stay alive but useless — no circuit breaker, no liveness probe, just an infinite stream of error logs. Now we increment a consecutive-failure counter and exit with an error once it hits 30 (~5 minutes of DB unavailability). k8s restart policy then cycles the pod, which both triggers an operator alert and gives us a fresh sqlx pool on the way back up. A successful claim (Some or None) resets the counter so one flaky query doesn't poison it. **BeaconMonitorsTest: drop the Process.sleep(1100).** The test relied on wall-clock gaps to disambiguate `inserted_at` values on the `:utc_datetime` (second-precision) timestamps — one full second of sleep per run, and on shared CI it could still flake when both inserts landed in the same second. Replaced with an explicit `Repo.update_all` that back-dates the first row by one second. 1.1s faster, deterministic. |
||
|---|---|---|
| .. | ||
| fixtures | ||
| microwaveprop | ||
| microwaveprop_web | ||
| mix/tasks | ||
| support | ||
| test_helper.exs | ||