Drop era5_batch global_limit — tracker leaks across deploys
The Pro Smart engine's global_limit tracker is stored inside oban_producers.meta and is not reliably released when a pod exits between job start and completion. After today's rollout two pods died holding tracked slots for running era5_batch jobs; the survivors refused to start any new ones because the cluster-wide tracker was stuck at 4/4 — with zero actual jobs executing — and the whole backfill froze for 2+ hours. Keeping rate_limit (10/hr) is enough to stay polite to Copernicus CDS. local_limit stays at 2 per pod, so effective cluster concurrency just scales with replica count instead of being capped by a global value whose bookkeeping can't survive a restart.
This commit is contained in:
parent
0d22d27a1e
commit
2c2b0170cc
1 changed files with 3 additions and 1 deletions
|
|
@ -167,9 +167,11 @@ if config_env() == :prod do
|
|||
terrain: 3,
|
||||
iemre: 3,
|
||||
era5: 2,
|
||||
# No global_limit: the Pro tracker leaked phantom slots across deploys
|
||||
# and stalled the whole backfill. The rate_limit is enough to protect
|
||||
# CDS and lets the per-pod local_limit scale naturally with replicas.
|
||||
era5_batch: [
|
||||
local_limit: 2,
|
||||
global_limit: 4,
|
||||
rate_limit: [allowed: 10, period: {1, :hour}]
|
||||
],
|
||||
rtma: 2,
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue