From 47031e17f1648d1b1e17e6a67d405697fc6f717f Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Thu, 30 Apr 2026 12:30:00 -0500 Subject: [PATCH] fix(prop/cron): fire HrdpsGridWorker 3x/hour to survive rolling deploys The hourly :35 fire was reliably being missed in production: the prop deployment is pinned to a single replica, and frequent ArgoCD deploys restart the pod often enough that some fires land in the swap window. Between 13:35Z and 17:25Z today the cron fired exactly zero times. Switch to :05/:25/:45. Each fire calls seed_current_hour which is idempotent on the (run_time, forecast_hour, kind, source) unique index, so duplicate inserts in the same hour are no-ops at the DB level and don't add cost on the happy path. Three shots per hour gives two retry windows when a deploy kills the pod mid-cron-tick. --- config/runtime.exs | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/config/runtime.exs b/config/runtime.exs index fc6ec5e7..c22a1833 100644 --- a/config/runtime.exs +++ b/config/runtime.exs @@ -249,12 +249,15 @@ if config_env() == :prod do {"5,20,35,50 * * * *", Microwaveprop.Workers.PropagationGridWorker}, # HRDPS Canadian propagation chain. ECCC publishes HRDPS f000 # ~3-4h after each 00/06/12/18Z cycle. Worker seeds exactly one - # forecast row per fire (current hour), so we run hourly to keep - # the /weather Canadian cell within ~1h of clock time. Reseeds - # are idempotent through the (run_time, forecast_hour, kind, source) - # unique index — duplicate fires for an already-seeded hour are - # no-ops at the DB level. - {"35 * * * *", Microwaveprop.Workers.HrdpsGridWorker}, + # forecast row per fire (current hour). Reseeds are idempotent + # through the (run_time, forecast_hour, kind, source) unique + # index — duplicate fires for an already-seeded hour are no-ops + # at the DB level, so running 3×/hour costs nothing on the happy + # path and gives us 2 retry windows when a rolling deploy kills + # the pod mid-cron-tick. The single-replica `prop` deployment + # combined with frequent ArgoCD deploys was reliably missing the + # hourly :35 fire — multiple shots per hour fixes that. + {"5,25,45 * * * *", Microwaveprop.Workers.HrdpsGridWorker}, # GEFS runs publish ~3-4h after the 00/06/12/18Z cycle; seed the # Day 2-7 outlook 5h after each run to stay safely past NOMADS' # publication lag. Seeder enqueues f024-f168 at 6-hour cadence