Graham McIntire
f122eedfa8
Two wins on the hourly propagation pipeline: 1. Parallelize the chain. seed_chain was enqueuing only f00 and each step self-enqueued f00+1, serializing ~2.5 min × 19 forecast hours into a ~48 min chain. Fan out all 19 jobs at once — they're genuinely independent (different HRRR URLs, different output files) — and let queue concurrency (2 slots × 3 pods = 6 parallel workers) drop wall time to ~10 min. Side effect: one permanently-failing step no longer takes out the rest of the chain, so the rescue-chain logic I added last commit becomes unnecessary. Removed. The :final cleanup (retain_window + purge) used to run on the fh=18 transition; with parallel execution we don't know which step finishes last. Cleanup now relies on the existing PropagationPruneWorker 15-min cron for time-based pruning. Stale chain leftovers are already a minor concern with the 15-min prune; if file bloat becomes real, PruneWorker can gain retain_window later. 2. Hoist band-invariant factors. score_time_of_day, score_sky, score_wind, score_pressure depend on conditions only, not the band — but composite_score was recomputing them 17 times per point. Added Scorer.precompute_band_invariants/1, called once per point before the bands loop; composite_score uses the cached values when present and falls back to computing for any caller that doesn't pre-warm (test harness, path integrator). Saves ~30% of the scoring inner loop. |
||
|---|---|---|
| .. | ||
| microwaveprop | ||
| microwaveprop_web | ||
| mix/tasks | ||
| microwaveprop.ex | ||
| microwaveprop_web.ex | ||