Graham McIntire
2c2b0170cc
The Pro Smart engine's global_limit tracker is stored inside oban_producers.meta and is not reliably released when a pod exits between job start and completion. After today's rollout two pods died holding tracked slots for running era5_batch jobs; the survivors refused to start any new ones because the cluster-wide tracker was stuck at 4/4 — with zero actual jobs executing — and the whole backfill froze for 2+ hours. Keeping rate_limit (10/hr) is enough to stay polite to Copernicus CDS. local_limit stays at 2 per pod, so effective cluster concurrency just scales with replica count instead of being capped by a global value whose bookkeeping can't survive a restart. |
||
|---|---|---|
| .. | ||
| config.exs | ||
| dev.exs | ||
| prod.exs | ||
| runtime.exs | ||
| test.exs | ||