Switch Oban to the Pro Smart engine, rate-limit ERA5 CDS

Wire up the Oban Pro Smart engine in base + runtime config and add the
Pro migration (oban_producers, oban_crons, oban_queues tables + the
v1.6 partition/workflow index additions on oban_jobs).

On the Smart engine the era5_batch queue now has a global_limit of 4
in-flight jobs across the cluster and a rate_limit of 10 jobs per hour
so slow Copernicus CDS submit/poll cycles can't exhaust our daily
request budget when a backfill kicks off.
This commit is contained in:
Graham McIntire 2026-04-09 14:19:04 -05:00
parent 0db5c2ae3e
commit 73d473a79b
3 changed files with 17 additions and 2 deletions

View file

@ -43,6 +43,7 @@ config :microwaveprop, MicrowavepropWeb.Endpoint,
live_view: [signing_salt: "Gdq36xze"]
config :microwaveprop, Oban,
engine: Oban.Pro.Engines.Smart,
repo: Microwaveprop.Repo,
queues: [solar: 1, weather: 20, enqueue: 1, hrrr: 5, terrain: 4, commercial: 2, iemre: 10, propagation: 1],
plugins: [

View file

@ -129,8 +129,11 @@ if config_env() == :prod do
],
secret_key_base: secret_key_base
# Production Oban: live scoring, polling, and on-demand QSO enrichment
# Production Oban: live scoring, polling, and on-demand QSO enrichment.
# Runs on the Oban Pro Smart engine so we can use global_limit / rate_limit
# to protect rate-limited upstreams (Copernicus CDS in particular).
config :microwaveprop, Oban,
engine: Oban.Pro.Engines.Smart,
# Per-pod concurrency (×3 pods = effective cluster total)
queues: [
propagation: 1,
@ -141,7 +144,11 @@ if config_env() == :prod do
terrain: 3,
iemre: 3,
era5: 2,
era5_batch: 2,
era5_batch: [
local_limit: 2,
global_limit: 4,
rate_limit: [allowed: 10, period: {1, :hour}]
],
rtma: 2,
backfill_enqueue: 1
],

View file

@ -0,0 +1,7 @@
defmodule Microwaveprop.Repo.Migrations.AddObanPro do
use Ecto.Migration
def up, do: Oban.Pro.Migration.up()
def down, do: Oban.Pro.Migration.down()
end