Unbreak propagation grid runs in prod

Oban's Lifeline plugin was rescuing PropagationGridWorker from
executing back to retryable every 10 minutes (runtime.exs) or
30 minutes (config.exs / dev.exs), but one full f00–f18 sweep
takes ~95 minutes and the worker's own timeout/1 is already
90 minutes. The rescue fired mid-run, the retry restarted at
f00, and after 3 rescues the job was discarded with errors: []
(Lifeline rescues don't write an error summary). prod's
propagation_scores never advanced beyond ~5 valid_times, so
the forecast timeline on /map only showed 3–4 hours instead of
the intended ~17.

Raise rescue_after to 120 minutes in all three configs — past
the worker's own 90-minute cap with headroom — and document
the contract so the next concurrency tweak doesn't regress it.

Also restore MrmsFetchWorker and AsosAdjustmentWorker to the
prod cron in runtime.exs: runtime.exs replaces config.exs's
plugin list wholesale and those two had been silently dropped
in the last rewrite. AsosAdjustmentWorker depends on the MRMS
precip cache, so both move together.
This commit is contained in:
Graham McIntire 2026-04-14 10:01:49 -05:00
parent 554bc9f339
commit 2abe66c0b4
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
3 changed files with 19 additions and 3 deletions

View file

@ -69,7 +69,8 @@ config :microwaveprop, Oban,
],
plugins: [
{Oban.Plugins.Pruner, max_age: 3600 * 24},
{Oban.Plugins.Lifeline, rescue_after: to_timeout(minute: 30)},
# See runtime.exs — must exceed PropagationGridWorker's 90-min timeout.
{Oban.Plugins.Lifeline, rescue_after: to_timeout(minute: 120)},
{Oban.Plugins.Cron,
crontab: [
{"0 8 * * *", Microwaveprop.Workers.SolarIndexWorker},

View file

@ -84,7 +84,8 @@ config :microwaveprop, Oban,
],
plugins: [
{Oban.Plugins.Pruner, max_age: 3600 * 24},
{Oban.Plugins.Lifeline, rescue_after: to_timeout(minute: 30)},
# See runtime.exs — must exceed PropagationGridWorker's 90-min timeout.
{Oban.Plugins.Lifeline, rescue_after: to_timeout(minute: 120)},
{Oban.Plugins.Cron,
crontab: [
{"5 */3 * * *", Microwaveprop.Workers.PropagationGridWorker},

View file

@ -197,7 +197,16 @@ if config_env() == :prod do
],
plugins: [
{Oban.Plugins.Pruner, max_age: 3600 * 24},
{Oban.Plugins.Lifeline, rescue_after: 10 * 60 * 1000},
# Lifeline rescues jobs stuck in `executing` after this window.
# MUST be larger than the longest `timeout/1` callback on any worker
# or Lifeline races the job's own deadline: PropagationGridWorker
# caps itself at 90 min and a real run is ~95 min of wall time, so
# 120 min gives enough headroom for the hourly sweep to finish
# before the safety net trips. Setting it shorter caused the grid
# job to be rescued mid-run and retried from f00, which discarded
# with empty errors after 3 rescues and left only 4-5 valid_times
# in propagation_scores.
{Oban.Plugins.Lifeline, rescue_after: to_timeout(minute: 120)},
{Oban.Plugins.Cron,
crontab: [
{"0 8 * * *", Microwaveprop.Workers.SolarIndexWorker},
@ -206,6 +215,11 @@ if config_env() == :prod do
# Hourly cron caused overlapping/failed runs because one full sweep
# exceeds a 60-minute window.
{"5 */3 * * *", Microwaveprop.Workers.PropagationGridWorker},
# MRMS refreshes the PrecipRate cache that AsosAdjustmentWorker
# overlays onto the score grid. Both must run together — an ASOS
# nudge without a fresh MRMS frame reverts to HRRR-only rain.
{"*/2 * * * *", Microwaveprop.Workers.MrmsFetchWorker},
{"*/10 * * * *", Microwaveprop.Workers.AsosAdjustmentWorker},
{"*/15 * * * *", Microwaveprop.Workers.PropagationPruneWorker},
{"*/30 * * * *", Microwaveprop.Workers.BackfillEnqueueWorker,
args: %{"limit" => 500, "types" => ["hrrr", "weather", "terrain", "iemre"]}},