Drop era5_batch global_limit — tracker leaks across deploys

The Pro Smart engine's global_limit tracker is stored inside
oban_producers.meta and is not reliably released when a pod exits
between job start and completion. After today's rollout two pods died
holding tracked slots for running era5_batch jobs; the survivors refused
to start any new ones because the cluster-wide tracker was stuck at
4/4 — with zero actual jobs executing — and the whole backfill froze
for 2+ hours.

Keeping rate_limit (10/hr) is enough to stay polite to Copernicus CDS.
local_limit stays at 2 per pod, so effective cluster concurrency just
scales with replica count instead of being capped by a global value
whose bookkeeping can't survive a restart.
This commit is contained in:
Graham McIntire 2026-04-13 15:28:45 -05:00
parent 0d22d27a1e
commit 2c2b0170cc
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59

View file

@ -167,9 +167,11 @@ if config_env() == :prod do
terrain: 3,
iemre: 3,
era5: 2,
# No global_limit: the Pro tracker leaked phantom slots across deploys
# and stalled the whole backfill. The rate_limit is enough to protect
# CDS and lets the per-pod local_limit scale naturally with replicas.
era5_batch: [
local_limit: 2,
global_limit: 4,
rate_limit: [allowed: 10, period: {1, :hour}]
],
rtma: 2,