Graham McIntire
024db2a5b4
Per-pod IEM rate limiter was 700ms (~1.4 req/sec); four pods put the fleet at ~5.7 req/sec, above the observed 429 threshold. Retry storm (Req exponential backoff up to 32s) piled concurrent sleeps on top of live workers and tripped Bandit's acceptor + Oban's leader heartbeat. - Raise default interval to 1500ms (~2.7 req/sec fleet). - Hot pod /live timeout 3s → 5s (acceptor stalls under retry sleeps). - prop-backfill exec probe period 30s → 120s, timeout 15s → 30s; the release CLI round-trips through distributed Erlang and 15s wasn't enough under load, restart-looping every ~10 min.
75 lines
2.9 KiB
Elixir
75 lines
2.9 KiB
Elixir
defmodule Microwaveprop.Weather.IemRateLimiter do
|
|
@moduledoc """
|
|
Per-pod serial gate for Iowa Environmental Mesonet HTTP calls.
|
|
|
|
IEM rate-limits bulk historical queries per source IP across all
|
|
in-flight requests, not per-concurrency. Simply lowering the
|
|
`:weather` Oban queue to one slot per pod still produced a steady
|
|
stream of HTTP 429s because workers were issuing back-to-back
|
|
requests inside each job (ASOS spanning multiple days, RAOB
|
|
batches). This limiter enforces a minimum gap between successive
|
|
calls — callers block in `acquire/0` until their reserved slot
|
|
arrives.
|
|
|
|
Configured via `config :microwaveprop, :iem_rate_limiter_interval_ms`
|
|
(defaults to 1500ms ≈ 0.67 req/sec per pod; four pods give ~2.7
|
|
req/sec cluster-wide). Previously set at 700ms, which put four pods
|
|
at ~5.7 req/sec fleet-wide — above IEM's observed 429 threshold —
|
|
and the resulting retry storm (Req exponential backoff up to 32 s)
|
|
piled concurrent HTTP sleeps on top of live workers until Bandit's
|
|
acceptor and Oban's leader heartbeat both started tripping. The
|
|
limiter is per-pod, so cluster-synchronous bursts are still possible
|
|
when every pod reserves `now + interval` simultaneously; moving to
|
|
a global limiter is out of scope for this tuning pass.
|
|
In tests, pass `interval_ms: 0` to disable.
|
|
"""
|
|
|
|
use GenServer
|
|
|
|
@default_interval_ms 1500
|
|
|
|
@spec start_link(keyword()) :: GenServer.on_start()
|
|
def start_link(opts \\ []) do
|
|
GenServer.start_link(__MODULE__, opts, name: Keyword.get(opts, :name, __MODULE__))
|
|
end
|
|
|
|
@doc """
|
|
Reserve the next available slot and block until it arrives. Callers
|
|
MUST invoke this before every IEM HTTP request. Returns `:ok` once
|
|
the caller is allowed to proceed.
|
|
"""
|
|
@spec acquire(GenServer.server()) :: :ok
|
|
def acquire(server \\ __MODULE__) do
|
|
if registered?(server) do
|
|
reserved_ms = GenServer.call(server, :reserve, :infinity)
|
|
delay = reserved_ms - System.system_time(:millisecond)
|
|
if delay > 0, do: Process.sleep(delay)
|
|
:ok
|
|
else
|
|
# Limiter not started (e.g. dev Mix tasks running in isolation).
|
|
# Fail open rather than deadlock the caller.
|
|
:ok
|
|
end
|
|
end
|
|
|
|
defp registered?(pid) when is_pid(pid), do: Process.alive?(pid)
|
|
defp registered?(name) when is_atom(name), do: Process.whereis(name) != nil
|
|
defp registered?(_), do: true
|
|
|
|
@impl true
|
|
def init(opts) do
|
|
interval_ms = Keyword.get(opts, :interval_ms, @default_interval_ms)
|
|
{:ok, %{interval_ms: interval_ms, next_ms: System.system_time(:millisecond)}}
|
|
end
|
|
|
|
@impl true
|
|
def handle_call(:reserve, _from, %{interval_ms: 0} = state) do
|
|
{:reply, System.system_time(:millisecond), state}
|
|
end
|
|
|
|
def handle_call(:reserve, _from, %{interval_ms: interval, next_ms: next_ms} = state) do
|
|
now = System.system_time(:millisecond)
|
|
reserved = max(now, next_ms)
|
|
{:reply, reserved, %{state | next_ms: reserved + interval}}
|
|
end
|
|
end
|