Move Era5MonthBatchWorker to its own queue
The slow CDS submit/poll/download cycle of the month-batch worker was sharing the era5 queue with the cheap Era5FetchWorker router, and its 2 concurrent slots were permanently pinned by long-running batch jobs while hundreds of router jobs starved. Give the batch worker its own era5_batch queue (also 2 concurrent per pod) so the router has dedicated capacity.
This commit is contained in:
parent
b0cf2cdd80
commit
485676887a
2 changed files with 6 additions and 1 deletions
|
|
@ -141,6 +141,7 @@ if config_env() == :prod do
|
||||||
terrain: 3,
|
terrain: 3,
|
||||||
iemre: 3,
|
iemre: 3,
|
||||||
era5: 2,
|
era5: 2,
|
||||||
|
era5_batch: 2,
|
||||||
rtma: 2,
|
rtma: 2,
|
||||||
backfill_enqueue: 1
|
backfill_enqueue: 1
|
||||||
],
|
],
|
||||||
|
|
|
||||||
|
|
@ -13,9 +13,13 @@ defmodule Microwaveprop.Workers.Era5MonthBatchWorker do
|
||||||
Uniqueness on the full arg set means Oban collapses duplicate enqueues
|
Uniqueness on the full arg set means Oban collapses duplicate enqueues
|
||||||
for the same tile-month — multiple `Era5FetchWorker` requests for the
|
for the same tile-month — multiple `Era5FetchWorker` requests for the
|
||||||
same region collapse into a single batch.
|
same region collapse into a single batch.
|
||||||
|
|
||||||
|
Runs on its own `:era5_batch` queue so its slow CDS poll cycles don't
|
||||||
|
starve the fast `Era5FetchWorker` router that also lives in the ERA5
|
||||||
|
namespace.
|
||||||
"""
|
"""
|
||||||
use Oban.Worker,
|
use Oban.Worker,
|
||||||
queue: :era5,
|
queue: :era5_batch,
|
||||||
max_attempts: 3,
|
max_attempts: 3,
|
||||||
unique: [
|
unique: [
|
||||||
period: :infinity,
|
period: :infinity,
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue