Per-pod IEM rate limiter was 700ms (~1.4 req/sec); four pods put the fleet at ~5.7 req/sec, above the observed 429 threshold. Retry storm (Req exponential backoff up to 32s) piled concurrent sleeps on top of live workers and tripped Bandit's acceptor + Oban's leader heartbeat. - Raise default interval to 1500ms (~2.7 req/sec fleet). - Hot pod /live timeout 3s → 5s (acceptor stalls under retry sleeps). - prop-backfill exec probe period 30s → 120s, timeout 15s → 30s; the release CLI round-trips through distributed Erlang and 15s wasn't enough under load, restart-looping every ~10 min. |
||
|---|---|---|
| .. | ||
| microwaveprop | ||
| microwaveprop_web | ||
| mix/tasks | ||
| microwaveprop.ex | ||
| microwaveprop_web.ex | ||