Graham McIntire
d5726a89d5
The HRDPS pipeline as written was unusable in production: each chain step took ~5 hours (not 30-90s as the dev-bench claim) because every `wgrib2 -lon` batch redundantly JPEG2000-decodes ~150 records. With 57 batches × 18 forecast hours, a single cycle would take days to drain. The /weather Canadian view only needs the current hour, so the cheapest fix is to stop seeding the rest of the chain entirely: * Rust grid: HRDPS_STEP=0.5 (was 0.125) drops point count 16× to ~3.5k. Coarser cells than HRRR, but /weather gets visible Canadian coverage inside one chain step (~20 min) instead of never. * GridTaskEnqueuer.seed_current_hour/2 inserts exactly one forecast row whose valid_time is closest to `now`. fh is clamped to 1..18 so the Rust F00Reserved branch never fires. * HrdpsGridWorker switches to seed_current_hour and drops the 6-hour-cycle cron in favor of HH:35 hourly fires. Reseeds are idempotent on the 4-column unique key. Also fixes a pre-existing credo "nested too deep" in ScalarFile.read_point by extracting lookup_in_chunk + hit_or_false helpers — happened to be on the read path that surfaces this work, so folding it into the same commit.
98 lines
3.3 KiB
Elixir
98 lines
3.3 KiB
Elixir
defmodule Microwaveprop.Workers.HrdpsGridWorker do
|
||
@moduledoc """
|
||
Cycle seed worker for the HRDPS Canadian propagation chain. Mirrors
|
||
`Microwaveprop.Workers.PropagationGridWorker` but seeds `grid_tasks`
|
||
rows with `source: "hrdps"` so the Rust worker routes them through
|
||
the HRDPS fetch path.
|
||
|
||
## Cadence
|
||
|
||
HRDPS runs 4×/day at 00/06/12/18Z and publishes f000 ~3-4h after each
|
||
cycle. This worker is intended to fire at HH+5 of each cycle hour
|
||
(05:35Z, 11:35Z, 17:35Z, 23:35Z) — see plan stage 6 for the cron entry.
|
||
|
||
## Activation
|
||
|
||
Active. The Rust prop-grid-rs worker handles `source: "hrdps"` rows via
|
||
`pipeline::run_chain_step_hrdps`, fetching from MSC Datamart, writing
|
||
scores to `<base>/<band>/<iso>.hrdps.prop` (sibling of the HRRR
|
||
`.prop` file for the same cycle). Cron entry lives in `runtime.exs`.
|
||
|
||
## Scope
|
||
|
||
Seeds exactly one forecast row per fire — the one whose valid_time
|
||
matches the current hour. The Canadian propagation/weather UI only
|
||
surfaces the present moment, so producing 18 forecast hours per cycle
|
||
was wasted wgrib2 work (~20 min/chain step at production point
|
||
density). If forecast hours come back into scope later, swap the
|
||
enqueuer call back to `GridTaskEnqueuer.seed_with_analysis/2`.
|
||
"""
|
||
|
||
use Oban.Worker,
|
||
queue: :hrdps,
|
||
priority: 0,
|
||
max_attempts: 5,
|
||
# Deduplicate seed jobs in a 6-hour window matching the cycle cadence.
|
||
unique: [period: 6 * 3600, states: [:available, :scheduled, :executing, :retryable]]
|
||
|
||
alias Microwaveprop.Instrument
|
||
alias Microwaveprop.Propagation.GridTaskEnqueuer
|
||
alias Microwaveprop.Weather.HrdpsClient
|
||
|
||
require Logger
|
||
|
||
@impl Oban.Worker
|
||
def perform(%Oban.Job{args: args}) when args == %{} do
|
||
Instrument.span([:hrdps, :grid_worker, :perform], %{}, fn ->
|
||
seed_chain()
|
||
end)
|
||
end
|
||
|
||
defp seed_chain do
|
||
# HRDPS shares grid_tasks with HRRR; the same reclaim sweep covers
|
||
# both because reclaim_stale_running/0 doesn't filter on source.
|
||
_ = GridTaskEnqueuer.reclaim_stale_running()
|
||
|
||
run_time = pick_run_time(DateTime.utc_now())
|
||
|
||
Logger.info("HrdpsGrid: seeding current-hour task run_time=#{run_time} source=hrdps")
|
||
|
||
case GridTaskEnqueuer.seed_current_hour(run_time, source: "hrdps") do
|
||
{:ok, count} ->
|
||
:telemetry.execute(
|
||
[:microwaveprop, :hrdps, :grid_worker, :seeded],
|
||
%{queued_tasks: count},
|
||
%{run_time: run_time}
|
||
)
|
||
|
||
:ok
|
||
|
||
{:error, reason} ->
|
||
:telemetry.execute(
|
||
[:microwaveprop, :hrdps, :grid_worker, :exception],
|
||
%{},
|
||
%{run_time: run_time, reason: inspect(reason)}
|
||
)
|
||
|
||
{:error, reason}
|
||
end
|
||
end
|
||
|
||
@doc """
|
||
Pick the freshest published HRDPS cycle. Probes `now-4h` (just past
|
||
typical publish latency) first; falls back to `now-10h` (one cycle
|
||
earlier) if the fresher cycle isn't yet on the datamart.
|
||
|
||
Public so tests can exercise selection logic without going through Oban.
|
||
"""
|
||
@spec pick_run_time(DateTime.t()) :: DateTime.t()
|
||
def pick_run_time(%DateTime{} = now) do
|
||
fresh = now |> DateTime.add(-4, :hour) |> HrdpsClient.nearest_hrdps_cycle() |> DateTime.truncate(:second)
|
||
|
||
if HrdpsClient.cycle_available?(fresh) do
|
||
fresh
|
||
else
|
||
now |> DateTime.add(-10, :hour) |> HrdpsClient.nearest_hrdps_cycle() |> DateTime.truncate(:second)
|
||
end
|
||
end
|
||
end
|