Graham McIntire
23e4e3fef2
Two independent wins the latest prod telemetry pointed at: 1. Iowa Environmental Mesonet rate limiter. IEM throttles per source IP across all in-flight requests, so dropping the `:weather` Oban queue to 1/pod wasn't enough — workers were still issuing back-to- back requests inside each job and drawing a steady stream of HTTP 429s (1,396 retryable on the queue). A GenServer-based token bucket serialises acquire() with a 700ms min gap per pod; three pods give ~4 req/sec cluster-wide, well under the observed 429 threshold. Wrapped around every IemClient fetch. 2. Parallel HRRR surface + pressure grid fetch. The two products live in separate wrfsfcf / wrfprsf GRIB files, so their fetch → wgrib2 → decode pipelines are independent; running them sequentially was adding ~30s per forecast hour for no reason. Task.async the pair, merge at the end. Halves per-fh wall time and should unblock some of the missing-hour cases we saw on the 14:00Z chain.
68 lines
2.4 KiB
Elixir
68 lines
2.4 KiB
Elixir
defmodule Microwaveprop.Weather.IemRateLimiter do
|
|
@moduledoc """
|
|
Per-pod serial gate for Iowa Environmental Mesonet HTTP calls.
|
|
|
|
IEM rate-limits bulk historical queries per source IP across all
|
|
in-flight requests, not per-concurrency. Simply lowering the
|
|
`:weather` Oban queue to one slot per pod still produced a steady
|
|
stream of HTTP 429s because workers were issuing back-to-back
|
|
requests inside each job (ASOS spanning multiple days, RAOB
|
|
batches). This limiter enforces a minimum gap between successive
|
|
calls — callers block in `acquire/0` until their reserved slot
|
|
arrives.
|
|
|
|
Configured via `config :microwaveprop, :iem_rate_limiter_interval_ms`
|
|
(defaults to 700ms ≈ 1.4 req/sec per pod; three pods give ~4 req/sec
|
|
cluster-wide, well under the empirically observed 429 threshold).
|
|
In tests, pass `interval_ms: 0` to disable.
|
|
"""
|
|
|
|
use GenServer
|
|
|
|
@default_interval_ms 700
|
|
|
|
@spec start_link(keyword()) :: GenServer.on_start()
|
|
def start_link(opts \\ []) do
|
|
GenServer.start_link(__MODULE__, opts, name: Keyword.get(opts, :name, __MODULE__))
|
|
end
|
|
|
|
@doc """
|
|
Reserve the next available slot and block until it arrives. Callers
|
|
MUST invoke this before every IEM HTTP request. Returns `:ok` once
|
|
the caller is allowed to proceed.
|
|
"""
|
|
@spec acquire(GenServer.server()) :: :ok
|
|
def acquire(server \\ __MODULE__) do
|
|
if registered?(server) do
|
|
reserved_ms = GenServer.call(server, :reserve, :infinity)
|
|
delay = reserved_ms - System.system_time(:millisecond)
|
|
if delay > 0, do: Process.sleep(delay)
|
|
:ok
|
|
else
|
|
# Limiter not started (e.g. dev Mix tasks running in isolation).
|
|
# Fail open rather than deadlock the caller.
|
|
:ok
|
|
end
|
|
end
|
|
|
|
defp registered?(pid) when is_pid(pid), do: Process.alive?(pid)
|
|
defp registered?(name) when is_atom(name), do: Process.whereis(name) != nil
|
|
defp registered?(_), do: true
|
|
|
|
@impl true
|
|
def init(opts) do
|
|
interval_ms = Keyword.get(opts, :interval_ms, @default_interval_ms)
|
|
{:ok, %{interval_ms: interval_ms, next_ms: System.system_time(:millisecond)}}
|
|
end
|
|
|
|
@impl true
|
|
def handle_call(:reserve, _from, %{interval_ms: 0} = state) do
|
|
{:reply, System.system_time(:millisecond), state}
|
|
end
|
|
|
|
def handle_call(:reserve, _from, %{interval_ms: interval, next_ms: next_ms} = state) do
|
|
now = System.system_time(:millisecond)
|
|
reserved = max(now, next_ms)
|
|
{:reply, reserved, %{state | next_ms: reserved + interval}}
|
|
end
|
|
end
|