Graham McIntire
0ed47db8b6
A full f00-f18 sweep takes ~170 min of wall time, longer than the ~90 min gap between deploys on this cluster. Over the last 7 days every run was killed mid-sweep and zero runs completed. The propagation_scores table only ever held the scraps from partial runs that landed before the pod died, which is why the map looked like it was showing "the current hour" (or nothing). The worker now processes exactly one forecast hour per perform/1 (~8-10 min) and enqueues the next hour as a fresh Oban job. A cron fire with empty args seeds the chain at f00; subsequent runs carry run_time + forecast_hour args. At f18 the chain stops and the pruner runs. Each step broadcasts propagation:updated immediately so new hours appear on the map as they land. Wall time per perform drops from ~170 min to ~10 min, so Lifeline's 2h rescue window is no longer a factor and a pod restart loses at most one forecast hour instead of the whole sweep. Retries and max_attempts now describe a single fh, not the whole chain. Also: bump the :propagation queue from 1 to 2 slots so PropagationPruneWorker can run alongside the chain job, drop the worker timeout/1 from 90 min to 20 min to match single-fh runs, and pull Lifeline rescue_after from 120 min to 45 min to keep the safety net above the step timeout. Adds a "Data from HH:MM UTC · Nh ago/now/+Nh" indicator above the pipeline chip in both the mobile and desktop sidebars so the user can tell which valid_time the map is showing when the bottom timeline isn't visible.
56 lines
2.1 KiB
Elixir
56 lines
2.1 KiB
Elixir
defmodule Microwaveprop.Workers.PropagationGridWorkerTest do
|
||
@moduledoc """
|
||
Tests the chain-orchestration behavior of PropagationGridWorker.
|
||
|
||
The worker processes forecast hours f00–f18 across the CONUS grid,
|
||
but a single full sweep takes ~2 hours of wall time — longer than a
|
||
typical pod restart window. To survive deploys, the worker processes
|
||
ONE forecast hour per `perform/1` call and enqueues the next hour as
|
||
a fresh Oban job. The tests here cover the dispatch + chain logic
|
||
without mocking the full HRRR / scoring stack.
|
||
"""
|
||
use Microwaveprop.DataCase, async: false
|
||
use Oban.Testing, repo: Microwaveprop.Repo
|
||
|
||
alias Microwaveprop.Workers.PropagationGridWorker
|
||
|
||
describe "perform/1 — chain seeding (empty args)" do
|
||
test "enqueues a single fh=0 chain step with a normalized run_time" do
|
||
Oban.Testing.with_testing_mode(:manual, fn ->
|
||
assert :ok = PropagationGridWorker.perform(%Oban.Job{args: %{}})
|
||
|
||
[child] = all_enqueued(worker: PropagationGridWorker)
|
||
assert child.args["forecast_hour"] == 0
|
||
assert is_binary(child.args["run_time"])
|
||
|
||
{:ok, run_time, _} = DateTime.from_iso8601(child.args["run_time"])
|
||
assert run_time.minute == 0
|
||
assert run_time.second == 0
|
||
assert DateTime.before?(run_time, DateTime.utc_now())
|
||
end)
|
||
end
|
||
end
|
||
|
||
describe "enqueue_next_step/2" do
|
||
test "enqueues fh+1 when fh < max" do
|
||
run_time = ~U[2026-04-14 16:00:00Z]
|
||
|
||
Oban.Testing.with_testing_mode(:manual, fn ->
|
||
assert {:ok, _} = PropagationGridWorker.enqueue_next_step(run_time, 0)
|
||
|
||
[child] = all_enqueued(worker: PropagationGridWorker)
|
||
assert child.args["forecast_hour"] == 1
|
||
assert child.args["run_time"] == DateTime.to_iso8601(run_time)
|
||
end)
|
||
end
|
||
|
||
test "does not enqueue anything after the final forecast hour" do
|
||
run_time = ~U[2026-04-14 16:00:00Z]
|
||
|
||
Oban.Testing.with_testing_mode(:manual, fn ->
|
||
assert :final = PropagationGridWorker.enqueue_next_step(run_time, 18)
|
||
assert [] = all_enqueued(worker: PropagationGridWorker)
|
||
end)
|
||
end
|
||
end
|
||
end
|