Graham McIntire
f122eedfa8
Two wins on the hourly propagation pipeline: 1. Parallelize the chain. seed_chain was enqueuing only f00 and each step self-enqueued f00+1, serializing ~2.5 min × 19 forecast hours into a ~48 min chain. Fan out all 19 jobs at once — they're genuinely independent (different HRRR URLs, different output files) — and let queue concurrency (2 slots × 3 pods = 6 parallel workers) drop wall time to ~10 min. Side effect: one permanently-failing step no longer takes out the rest of the chain, so the rescue-chain logic I added last commit becomes unnecessary. Removed. The :final cleanup (retain_window + purge) used to run on the fh=18 transition; with parallel execution we don't know which step finishes last. Cleanup now relies on the existing PropagationPruneWorker 15-min cron for time-based pruning. Stale chain leftovers are already a minor concern with the 15-min prune; if file bloat becomes real, PruneWorker can gain retain_window later. 2. Hoist band-invariant factors. score_time_of_day, score_sky, score_wind, score_pressure depend on conditions only, not the band — but composite_score was recomputing them 17 times per point. Added Scorer.precompute_band_invariants/1, called once per point before the bands loop; composite_score uses the cached values when present and falls back to computing for any caller that doesn't pre-warm (test harness, path integrator). Saves ~30% of the scoring inner loop.
54 lines
2.2 KiB
Elixir
54 lines
2.2 KiB
Elixir
defmodule Microwaveprop.Workers.PropagationGridWorkerTest do
|
||
@moduledoc """
|
||
Tests the chain-orchestration behavior of PropagationGridWorker.
|
||
|
||
The worker processes forecast hours f00–f18 across the CONUS grid,
|
||
but a single full sweep takes ~2 hours of wall time — longer than a
|
||
typical pod restart window. To survive deploys, the worker processes
|
||
ONE forecast hour per `perform/1` call and enqueues the next hour as
|
||
a fresh Oban job. The tests here cover the dispatch + chain logic
|
||
without mocking the full HRRR / scoring stack.
|
||
"""
|
||
use Microwaveprop.DataCase, async: false
|
||
use Oban.Testing, repo: Microwaveprop.Repo
|
||
|
||
alias Microwaveprop.Workers.MrmsFetchWorker
|
||
alias Microwaveprop.Workers.PropagationGridWorker
|
||
alias Microwaveprop.Workers.PropagationPruneWorker
|
||
|
||
describe "queue priority" do
|
||
test "PropagationGridWorker runs at the highest priority on :propagation" do
|
||
assert PropagationGridWorker.__opts__()[:priority] == 0
|
||
end
|
||
|
||
test "MrmsFetchWorker and PropagationPruneWorker yield to the grid chain" do
|
||
# Same :propagation queue — must be lower priority so hourly chain
|
||
# steps jump ahead of a MRMS or pruner backlog.
|
||
assert MrmsFetchWorker.__opts__()[:priority] > 0
|
||
assert PropagationPruneWorker.__opts__()[:priority] > 0
|
||
end
|
||
end
|
||
|
||
describe "perform/1 — chain seeding (empty args)" do
|
||
test "enqueues all 19 forecast-hour jobs at once, all pointing at the same run_time" do
|
||
Oban.Testing.with_testing_mode(:manual, fn ->
|
||
assert :ok = PropagationGridWorker.perform(%Oban.Job{args: %{}})
|
||
|
||
jobs = all_enqueued(worker: PropagationGridWorker)
|
||
assert length(jobs) == 19
|
||
|
||
fhs = jobs |> Enum.map(& &1.args["forecast_hour"]) |> Enum.sort()
|
||
assert fhs == Enum.to_list(0..18)
|
||
|
||
# Every child job shares the same run_time and it's normalized
|
||
# to top-of-hour UTC.
|
||
run_times = jobs |> Enum.map(& &1.args["run_time"]) |> Enum.uniq()
|
||
assert [rt] = run_times
|
||
{:ok, dt, _} = DateTime.from_iso8601(rt)
|
||
assert dt.minute == 0
|
||
assert dt.second == 0
|
||
assert DateTime.before?(dt, DateTime.utc_now())
|
||
end)
|
||
end
|
||
end
|
||
end
|