Move Era5MonthBatchWorker to its own queue

The slow CDS submit/poll/download cycle of the month-batch worker was
sharing the era5 queue with the cheap Era5FetchWorker router, and its
2 concurrent slots were permanently pinned by long-running batch jobs
while hundreds of router jobs starved. Give the batch worker its own
era5_batch queue (also 2 concurrent per pod) so the router has
dedicated capacity.
This commit is contained in:
Graham McIntire 2026-04-09 14:12:22 -05:00
parent b0cf2cdd80
commit 485676887a
2 changed files with 6 additions and 1 deletions

View file

@ -141,6 +141,7 @@ if config_env() == :prod do
terrain: 3,
iemre: 3,
era5: 2,
era5_batch: 2,
rtma: 2,
backfill_enqueue: 1
],

View file

@ -13,9 +13,13 @@ defmodule Microwaveprop.Workers.Era5MonthBatchWorker do
Uniqueness on the full arg set means Oban collapses duplicate enqueues
for the same tile-month multiple `Era5FetchWorker` requests for the
same region collapse into a single batch.
Runs on its own `:era5_batch` queue so its slow CDS poll cycles don't
starve the fast `Era5FetchWorker` router that also lives in the ERA5
namespace.
"""
use Oban.Worker,
queue: :era5,
queue: :era5_batch,
max_attempts: 3,
unique: [
period: :infinity,