Oban's Lifeline plugin was rescuing PropagationGridWorker from executing back to retryable every 10 minutes (runtime.exs) or 30 minutes (config.exs / dev.exs), but one full f00–f18 sweep takes ~95 minutes and the worker's own timeout/1 is already 90 minutes. The rescue fired mid-run, the retry restarted at f00, and after 3 rescues the job was discarded with errors: [] (Lifeline rescues don't write an error summary). prod's propagation_scores never advanced beyond ~5 valid_times, so the forecast timeline on /map only showed 3–4 hours instead of the intended ~17. Raise rescue_after to 120 minutes in all three configs — past the worker's own 90-minute cap with headroom — and document the contract so the next concurrency tweak doesn't regress it. Also restore MrmsFetchWorker and AsosAdjustmentWorker to the prod cron in runtime.exs: runtime.exs replaces config.exs's plugin list wholesale and those two had been silently dropped in the last rewrite. AsosAdjustmentWorker depends on the MRMS precip cache, so both move together. |
||
|---|---|---|
| .. | ||
| config.exs | ||
| dev.exs | ||
| prod.exs | ||
| runtime.exs | ||
| test.exs | ||