From c96bdca7b783ab51ada7d97659e5a2830fb7ff72 Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Tue, 21 Apr 2026 01:24:28 -0500 Subject: [PATCH] fix(backfill): cap cron run at 2000 contacts/tick to avoid 17 min discard Unlimited BackfillEnqueueWorker runs loaded all ~36k enrichable contacts and hit Oban's ~17 min execution timeout, discarding the parent job every cron tick. Enrichment was landing partially (jobs enqueued before the timeout) but the loop never completed and no success signal was emitted. A per-tick limit of 2000 keeps each run well under any timeout while still draining the backlog at 4000/hour across the */30 schedule. --- config/runtime.exs | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/config/runtime.exs b/config/runtime.exs index 75670863..58580e0e 100644 --- a/config/runtime.exs +++ b/config/runtime.exs @@ -208,8 +208,15 @@ if config_env() == :prod do # The :narr type is virtual: it targets contacts with hrrr_status = # :unavailable (pre-2014, missing from the HRRR archive) and dispatches # NarrFetchWorker against NCEI. See narr_jobs_for_contact/1. + # limit=2000 keeps each 30-min tick under the Oban job-execution + # timeout that discards unlimited runs after ~17 min. At 4000 + # contacts/hour the backlog of missing-data contacts drains + # steadily without any single job getting killed mid-way. {"*/30 * * * *", Microwaveprop.Workers.BackfillEnqueueWorker, - args: %{"types" => ["hrrr", "weather", "terrain", "iemre", "narr", "radar", "mechanism"]}}, + args: %{ + "limit" => 2000, + "types" => ["hrrr", "weather", "terrain", "iemre", "narr", "radar", "mechanism"] + }}, # Hourly safety net for pos1/pos2/distance_km. Normally every contact # gets positions at insert time via Radio.resolve_grids_and_insert/1, # but direct DB writes (manual fixes, bulk imports) can bypass that.