Move Era5MonthBatchWorker to its own queue
The slow CDS submit/poll/download cycle of the month-batch worker was sharing the era5 queue with the cheap Era5FetchWorker router, and its 2 concurrent slots were permanently pinned by long-running batch jobs while hundreds of router jobs starved. Give the batch worker its own era5_batch queue (also 2 concurrent per pod) so the router has dedicated capacity.
This commit is contained in:
parent
b0cf2cdd80
commit
485676887a
2 changed files with 6 additions and 1 deletions
|
|
@ -141,6 +141,7 @@ if config_env() == :prod do
|
|||
terrain: 3,
|
||||
iemre: 3,
|
||||
era5: 2,
|
||||
era5_batch: 2,
|
||||
rtma: 2,
|
||||
backfill_enqueue: 1
|
||||
],
|
||||
|
|
|
|||
|
|
@ -13,9 +13,13 @@ defmodule Microwaveprop.Workers.Era5MonthBatchWorker do
|
|||
Uniqueness on the full arg set means Oban collapses duplicate enqueues
|
||||
for the same tile-month — multiple `Era5FetchWorker` requests for the
|
||||
same region collapse into a single batch.
|
||||
|
||||
Runs on its own `:era5_batch` queue so its slow CDS poll cycles don't
|
||||
starve the fast `Era5FetchWorker` router that also lives in the ERA5
|
||||
namespace.
|
||||
"""
|
||||
use Oban.Worker,
|
||||
queue: :era5,
|
||||
queue: :era5_batch,
|
||||
max_attempts: 3,
|
||||
unique: [
|
||||
period: :infinity,
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue