prop/config
Graham McIntire 2abe66c0b4
Unbreak propagation grid runs in prod
Oban's Lifeline plugin was rescuing PropagationGridWorker from
executing back to retryable every 10 minutes (runtime.exs) or
30 minutes (config.exs / dev.exs), but one full f00–f18 sweep
takes ~95 minutes and the worker's own timeout/1 is already
90 minutes. The rescue fired mid-run, the retry restarted at
f00, and after 3 rescues the job was discarded with errors: []
(Lifeline rescues don't write an error summary). prod's
propagation_scores never advanced beyond ~5 valid_times, so
the forecast timeline on /map only showed 3–4 hours instead of
the intended ~17.

Raise rescue_after to 120 minutes in all three configs — past
the worker's own 90-minute cap with headroom — and document
the contract so the next concurrency tweak doesn't regress it.

Also restore MrmsFetchWorker and AsosAdjustmentWorker to the
prod cron in runtime.exs: runtime.exs replaces config.exs's
plugin list wholesale and those two had been silently dropped
in the last rewrite. AsosAdjustmentWorker depends on the MRMS
precip cache, so both move together.
2026-04-14 10:01:49 -05:00
..
config.exs Unbreak propagation grid runs in prod 2026-04-14 10:01:49 -05:00
dev.exs Unbreak propagation grid runs in prod 2026-04-14 10:01:49 -05:00
prod.exs Configure outbound SMTP email via Swoosh with TLS 2026-04-02 13:08:50 -05:00
runtime.exs Unbreak propagation grid runs in prod 2026-04-14 10:01:49 -05:00
test.exs Split ERA5 backfill into submit/poll workers with persistent CDS state 2026-04-13 16:26:26 -05:00