graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(workers): make weather/hrrr job uniqueness horizon indefinite

Browse source 
Both workers had unique: [period: 300] which only dedupes within a
5-minute window. Re-running the enrichment-enqueue script hours or
days later stacked identical jobs in the available queue — we found
~60k redundant weather + 3k redundant hrrr jobs in prod from the
March and April enqueue passes.

:infinity means Oban dedupes against every live job in the states
list regardless of age, so a second enqueue pass is a no-op as long
as the first pass's jobs haven't completed yet.
This commit is contained in:
Graham McIntire 2026-04-17 10:15:25 -05:00
parent 246f51575c
commit 847f2001dd
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
2 changed files with 2 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
lib/microwaveprop/workers/hrrr_fetch_worker.ex
View file

@ -3,7 +3,7 @@ defmodule Microwaveprop.Workers.HrrrFetchWorker do
use Oban.Worker,
queue: :hrrr,
max_attempts: 20,
unique: [period: 300, states: [:available, :scheduled, :executing, :retryable]]
unique: [period: :infinity, states: [:available, :scheduled, :executing, :retryable]]
alias Microwaveprop.Weather
alias Microwaveprop.Weather.HrrrClient

2
lib/microwaveprop/workers/weather_fetch_worker.ex
View file

@ -3,7 +3,7 @@ defmodule Microwaveprop.Workers.WeatherFetchWorker do
use Oban.Worker,
queue: :weather,
max_attempts: 20,
unique: [period: 300, states: [:available, :scheduled, :executing, :retryable]]
unique: [period: :infinity, states: [:available, :scheduled, :executing, :retryable]]
alias Microwaveprop.Repo
alias Microwaveprop.Weather