graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(prop/cron): fire HrdpsGridWorker 3x/hour to survive rolling deploys

Browse source 
The hourly :35 fire was reliably being missed in production: the prop
deployment is pinned to a single replica, and frequent ArgoCD deploys
restart the pod often enough that some fires land in the swap window.
Between 13:35Z and 17:25Z today the cron fired exactly zero times.

Switch to :05/:25/:45. Each fire calls seed_current_hour which is
idempotent on the (run_time, forecast_hour, kind, source) unique
index, so duplicate inserts in the same hour are no-ops at the DB
level and don't add cost on the happy path. Three shots per hour
gives two retry windows when a deploy kills the pod mid-cron-tick.
This commit is contained in:
Graham McIntire 2026-04-30 12:30:00 -05:00
parent 4901c83916
commit 47031e17f1
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
1 changed files with 9 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
config/runtime.exs
View file

@ -249,12 +249,15 @@ if config_env() == :prod do
{"5,20,35,50 * * * *", Microwaveprop.Workers.PropagationGridWorker}, {"5,20,35,50 * * * *", Microwaveprop.Workers.PropagationGridWorker},
# HRDPS Canadian propagation chain. ECCC publishes HRDPS f000 # HRDPS Canadian propagation chain. ECCC publishes HRDPS f000
# ~3-4h after each 00/06/12/18Z cycle. Worker seeds exactly one # ~3-4h after each 00/06/12/18Z cycle. Worker seeds exactly one
# forecast row per fire (current hour), so we run hourly to keep # forecast row per fire (current hour). Reseeds are idempotent
# the /weather Canadian cell within ~1h of clock time. Reseeds # through the (run_time, forecast_hour, kind, source) unique
# are idempotent through the (run_time, forecast_hour, kind, source) # index — duplicate fires for an already-seeded hour are no-ops
# unique index — duplicate fires for an already-seeded hour are # at the DB level, so running 3×/hour costs nothing on the happy
# no-ops at the DB level. # path and gives us 2 retry windows when a rolling deploy kills
{"35 * * * *", Microwaveprop.Workers.HrdpsGridWorker}, # the pod mid-cron-tick. The single-replica `prop` deployment
# combined with frequent ArgoCD deploys was reliably missing the
# hourly :35 fire — multiple shots per hour fixes that.
{"5,25,45 * * * *", Microwaveprop.Workers.HrdpsGridWorker},
# GEFS runs publish ~3-4h after the 00/06/12/18Z cycle; seed the # GEFS runs publish ~3-4h after the 00/06/12/18Z cycle; seed the
# Day 2-7 outlook 5h after each run to stay safely past NOMADS' # Day 2-7 outlook 5h after each run to stay safely past NOMADS'
# publication lag. Seeder enqueues f024-f168 at 6-hour cadence # publication lag. Seeder enqueues f024-f168 at 6-hour cadence