defmodule Microwaveprop.Weather.IemRateLimiter do @moduledoc """ Per-pod serial gate for Iowa Environmental Mesonet HTTP calls. IEM rate-limits bulk historical queries per source IP across all in-flight requests, not per-concurrency. Simply lowering the `:weather` Oban queue to one slot per pod still produced a steady stream of HTTP 429s because workers were issuing back-to-back requests inside each job (ASOS spanning multiple days, RAOB batches). This limiter enforces a minimum gap between successive calls — callers block in `acquire/0` until their reserved slot arrives. Configured via `config :microwaveprop, :iem_rate_limiter_interval_ms` (defaults to 1500ms ≈ 0.67 req/sec per pod; four pods give ~2.7 req/sec cluster-wide). Previously set at 700ms, which put four pods at ~5.7 req/sec fleet-wide — above IEM's observed 429 threshold — and the resulting retry storm (Req exponential backoff up to 32 s) piled concurrent HTTP sleeps on top of live workers until Bandit's acceptor and Oban's leader heartbeat both started tripping. The limiter is per-pod, so cluster-synchronous bursts are still possible when every pod reserves `now + interval` simultaneously; moving to a global limiter is out of scope for this tuning pass. In tests, pass `interval_ms: 0` to disable. """ use GenServer @default_interval_ms 1500 @spec start_link(keyword()) :: GenServer.on_start() def start_link(opts \\ []) do GenServer.start_link(__MODULE__, opts, name: Keyword.get(opts, :name, __MODULE__)) end @doc """ Reserve the next available slot and block until it arrives. Callers MUST invoke this before every IEM HTTP request. Returns `:ok` once the caller is allowed to proceed. """ @spec acquire(GenServer.server()) :: :ok def acquire(server \\ __MODULE__) do if registered?(server) do reserved_ms = GenServer.call(server, :reserve, :infinity) delay = reserved_ms - System.system_time(:millisecond) if delay > 0, do: Process.sleep(delay) :ok else # Limiter not started (e.g. dev Mix tasks running in isolation). # Fail open rather than deadlock the caller. :ok end end defp registered?(pid) when is_pid(pid), do: Process.alive?(pid) defp registered?(name) when is_atom(name), do: Process.whereis(name) != nil defp registered?(_), do: true @impl true def init(opts) do interval_ms = Keyword.get(opts, :interval_ms, @default_interval_ms) {:ok, %{interval_ms: interval_ms, next_ms: System.system_time(:millisecond)}} end @impl true def handle_call(:reserve, _from, %{interval_ms: 0} = state) do {:reply, System.system_time(:millisecond), state} end def handle_call(:reserve, _from, %{interval_ms: interval, next_ms: next_ms} = state) do now = System.system_time(:millisecond) reserved = max(now, next_ms) {:reply, reserved, %{state | next_ms: reserved + interval}} end end