fix(backfill): cap cron run at 2000 contacts/tick to avoid 17 min discard

Unlimited BackfillEnqueueWorker runs loaded all ~36k enrichable
contacts and hit Oban's ~17 min execution timeout, discarding the
parent job every cron tick. Enrichment was landing partially
(jobs enqueued before the timeout) but the loop never completed
and no success signal was emitted. A per-tick limit of 2000 keeps
each run well under any timeout while still draining the backlog
at 4000/hour across the */30 schedule.
This commit is contained in:
Graham McIntire 2026-04-21 01:24:28 -05:00
parent e3d7f5b2c2
commit c96bdca7b7
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59

View file

@ -208,8 +208,15 @@ if config_env() == :prod do
# The :narr type is virtual: it targets contacts with hrrr_status =
# :unavailable (pre-2014, missing from the HRRR archive) and dispatches
# NarrFetchWorker against NCEI. See narr_jobs_for_contact/1.
# limit=2000 keeps each 30-min tick under the Oban job-execution
# timeout that discards unlimited runs after ~17 min. At 4000
# contacts/hour the backlog of missing-data contacts drains
# steadily without any single job getting killed mid-way.
{"*/30 * * * *", Microwaveprop.Workers.BackfillEnqueueWorker,
args: %{"types" => ["hrrr", "weather", "terrain", "iemre", "narr", "radar", "mechanism"]}},
args: %{
"limit" => 2000,
"types" => ["hrrr", "weather", "terrain", "iemre", "narr", "radar", "mechanism"]
}},
# Hourly safety net for pos1/pos2/distance_km. Normally every contact
# gets positions at insert time via Radio.resolve_grids_and_insert/1,
# but direct DB writes (manual fixes, bulk imports) can bypass that.