fix(propagation): dedup PropagationGridWorker enqueues

FreshnessMonitor's comment claimed 'Oban unique constraint on
PropagationGridWorker prevents duplicates' — but the worker declared
no unique: option. During a long outage the monitor's 5-min stale-check
tick would pile up identical jobs (a 2h outage = 24 stacked jobs, each
doing the full f00-f18 HRRR chain).

Put the dedup on the worker (vs. the monitor's insert call) so anything
enqueuing it benefits — FreshnessMonitor, the hourly cron, or a manual
mix propagation_grid run.

unique: [period: 3600, states: [:available, :scheduled, :executing,
:retryable]] aligns with the hourly cron cadence. The seed job args
(%{}) collapse with themselves while chain-step jobs keep distinct
run_time/forecast_hour and remain enqueuable.
This commit is contained in:
Graham McIntire 2026-04-16 14:55:24 -05:00
parent 221d5516bc
commit bba860f28d
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
3 changed files with 22 additions and 10 deletions

View file

@ -54,9 +54,11 @@ defmodule Microwaveprop.Propagation.FreshnessMonitor do
end
defp enqueue_if_not_queued do
# Oban unique constraint on PropagationGridWorker prevents duplicates —
# if a job is already available/scheduled/executing/retryable within the
# last hour, insert is a no-op.
# PropagationGridWorker declares `unique:` over a 1-hour window on
# the seed args (`%{}`), so repeated 5-minute ticks during a long
# outage collapse into a single seed job — not one stacked chain
# per tick. `Oban.insert` returns `{:ok, job}` either way; the
# `conflict?` field on the returned job distinguishes the two.
Oban.insert(PropagationGridWorker.new(%{}))
end
end

View file

@ -24,7 +24,16 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
# (e.g., a rolling deploy that kills a mid-flight chain step)
# don't exhaust the chain's retry budget and discard the whole
# run. Legitimate scoring errors still give up after 5 attempts.
max_attempts: 5
max_attempts: 5,
# Deduplicate identical jobs across a 1-hour window. Uniqueness
# is over the full args set, so the seed (`%{}`) collapses with
# itself — FreshnessMonitor's 5-minute ticks during a long outage
# no longer stack 24 redundant f00-f18 chains — while chain steps
# with distinct `run_time` + `forecast_hour` args remain distinct
# from each other and from the seed. The period is the same as
# the hourly cron interval so successive hourly cron fires move
# past the window naturally.
unique: [period: 3600, states: [:available, :scheduled, :executing, :retryable]]
alias Microwaveprop.Commercial
alias Microwaveprop.Propagation

View file

@ -95,11 +95,12 @@ defmodule Microwaveprop.Propagation.FreshnessMonitorTest do
end)
end
test "repeated ticks on a persistently stale grid each enqueue another job" do
# Characterization note: the module comment claims an Oban unique
# constraint on PropagationGridWorker suppresses duplicates, but
# the worker currently declares no `unique:` option. Each tick
# therefore inserts a fresh job.
test "repeated ticks on a persistently stale grid are deduplicated into a single job" do
# PropagationGridWorker declares `unique:` on the seed args (`%{}`),
# so a stale-check tick that runs while an earlier seed is still
# available/scheduled/executing/retryable is a no-op. Without this,
# a long outage stacks one expensive HRRR f00-f18 chain per 5-min
# tick (e.g. a 2-hour outage = 24 redundant seed jobs).
ScoresFile.write!(@band_mhz, minutes_ago(180), @sample_scores)
Oban.Testing.with_testing_mode(:manual, fn ->
@ -107,7 +108,7 @@ defmodule Microwaveprop.Propagation.FreshnessMonitorTest do
FreshnessMonitor.handle_info(:check, %{})
FreshnessMonitor.handle_info(:check, %{})
assert 3 == length(all_enqueued(worker: PropagationGridWorker))
assert 1 == length(all_enqueued(worker: PropagationGridWorker))
end)
end