Switch Oban to the Pro Smart engine, rate-limit ERA5 CDS
Wire up the Oban Pro Smart engine in base + runtime config and add the Pro migration (oban_producers, oban_crons, oban_queues tables + the v1.6 partition/workflow index additions on oban_jobs). On the Smart engine the era5_batch queue now has a global_limit of 4 in-flight jobs across the cluster and a rate_limit of 10 jobs per hour so slow Copernicus CDS submit/poll cycles can't exhaust our daily request budget when a backfill kicks off.
This commit is contained in:
parent
0db5c2ae3e
commit
73d473a79b
3 changed files with 17 additions and 2 deletions
|
|
@ -43,6 +43,7 @@ config :microwaveprop, MicrowavepropWeb.Endpoint,
|
|||
live_view: [signing_salt: "Gdq36xze"]
|
||||
|
||||
config :microwaveprop, Oban,
|
||||
engine: Oban.Pro.Engines.Smart,
|
||||
repo: Microwaveprop.Repo,
|
||||
queues: [solar: 1, weather: 20, enqueue: 1, hrrr: 5, terrain: 4, commercial: 2, iemre: 10, propagation: 1],
|
||||
plugins: [
|
||||
|
|
|
|||
|
|
@ -129,8 +129,11 @@ if config_env() == :prod do
|
|||
],
|
||||
secret_key_base: secret_key_base
|
||||
|
||||
# Production Oban: live scoring, polling, and on-demand QSO enrichment
|
||||
# Production Oban: live scoring, polling, and on-demand QSO enrichment.
|
||||
# Runs on the Oban Pro Smart engine so we can use global_limit / rate_limit
|
||||
# to protect rate-limited upstreams (Copernicus CDS in particular).
|
||||
config :microwaveprop, Oban,
|
||||
engine: Oban.Pro.Engines.Smart,
|
||||
# Per-pod concurrency (×3 pods = effective cluster total)
|
||||
queues: [
|
||||
propagation: 1,
|
||||
|
|
@ -141,7 +144,11 @@ if config_env() == :prod do
|
|||
terrain: 3,
|
||||
iemre: 3,
|
||||
era5: 2,
|
||||
era5_batch: 2,
|
||||
era5_batch: [
|
||||
local_limit: 2,
|
||||
global_limit: 4,
|
||||
rate_limit: [allowed: 10, period: {1, :hour}]
|
||||
],
|
||||
rtma: 2,
|
||||
backfill_enqueue: 1
|
||||
],
|
||||
|
|
|
|||
7
priv/repo/migrations/20260409191733_add_oban_pro.exs
Normal file
7
priv/repo/migrations/20260409191733_add_oban_pro.exs
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
defmodule Microwaveprop.Repo.Migrations.AddObanPro do
|
||||
use Ecto.Migration
|
||||
|
||||
def up, do: Oban.Pro.Migration.up()
|
||||
|
||||
def down, do: Oban.Pro.Migration.down()
|
||||
end
|
||||
Loading…
Add table
Reference in a new issue