Graham McIntire
024db2a5b4
Per-pod IEM rate limiter was 700ms (~1.4 req/sec); four pods put the fleet at ~5.7 req/sec, above the observed 429 threshold. Retry storm (Req exponential backoff up to 32s) piled concurrent sleeps on top of live workers and tripped Bandit's acceptor + Oban's leader heartbeat. - Raise default interval to 1500ms (~2.7 req/sec fleet). - Hot pod /live timeout 3s → 5s (acceptor stalls under retry sleeps). - prop-backfill exec probe period 30s → 120s, timeout 15s → 30s; the release CLI round-trips through distributed Erlang and 15s wasn't enough under load, restart-looping every ~10 min. |
||
|---|---|---|
| .. | ||
| deployment-backfill.yaml | ||
| deployment-grid-rs.yaml | ||
| deployment-hrrr-point-rs.yaml | ||
| deployment.yaml | ||
| flux.yaml | ||
| kustomization.yaml | ||
| metrics-service.yaml | ||
| namespace.yaml | ||
| rbac.yaml | ||
| service.yaml | ||