Graham McIntire
1da78a80e5
Stages 6, 7, 10 of the HRDPS plan, plus the Elixir read-side merge that makes Rust's `.hrdps.prop` writes show up on /map. Read side: - `ScoresFile.path_for_hrdps/2` + `read_hrdps/2` — companions to the HRRR equivalents. Same decode path; different filename. - `ScoresFile.read_bounds/3` now reads HRRR + HRDPS files and concatenates the cells. Files are disjoint by construction (Grid.hrdps_only_points excludes CONUS) so no de-dup needed. - `ScoresFile.read_point/4` checks `.prop`, then `.hrdps.prop`, then legacy `.ntms` — first hit wins. Cells outside their owning region read as no-data in the other file's grid, so the order doesn't matter for correctness. - `Propagation.warm_cache_and_broadcast/2` reads both files and merges before broadcasting to the cluster ScoreCache. A single missing file (HRDPS pre-cycle, HRRR briefly absent) is now OK — the cache warms with whichever side is available. Activation: - runtime.exs cron: HRDPS seed worker fires at HH:35 of each cycle hour (05:35Z, 11:35Z, 17:35Z, 23:35Z) — 5h after cycle, 30 min staggered from HRRR's */15 schedule. - HrdpsGridWorker moduledoc updated: no longer dormant; Rust prop-grid-rs handles source='hrdps' rows. - map_live North America bbox extended to (24.5, 60.0, -141.0, -52.0) for the out-of-coverage banner check; visitors anywhere in CONUS or the Canadian extent now stay banner-free. Cleanup: - mix hrdps.probe deleted — its risks are retired and the production HrdpsClient covers the same surface. What's still in motion (not blocking the Canadian /map): - Per-valid_time profile + scalar artifacts for HRDPS (would let /weather show Canadian temperature/refractivity layers). Rust pipeline currently skips those so the HRRR-written companion files aren't clobbered — separate stage when /weather UX is decided. - FreshnessMonitor HRDPS staleness tracking — HRDPS has its own cron and a different cadence than HRRR; HRRR-stale logic doesn't apply cleanly. Defer until prod tells us what alarms we actually want.
89 lines
2.9 KiB
Elixir
89 lines
2.9 KiB
Elixir
defmodule Microwaveprop.Workers.HrdpsGridWorker do
|
||
@moduledoc """
|
||
Cycle seed worker for the HRDPS Canadian propagation chain. Mirrors
|
||
`Microwaveprop.Workers.PropagationGridWorker` but seeds `grid_tasks`
|
||
rows with `source: "hrdps"` so the Rust worker routes them through
|
||
the HRDPS fetch path.
|
||
|
||
## Cadence
|
||
|
||
HRDPS runs 4×/day at 00/06/12/18Z and publishes f000 ~3-4h after each
|
||
cycle. This worker is intended to fire at HH+5 of each cycle hour
|
||
(05:35Z, 11:35Z, 17:35Z, 23:35Z) — see plan stage 6 for the cron entry.
|
||
|
||
## Activation
|
||
|
||
Active. The Rust prop-grid-rs worker handles `source: "hrdps"` rows via
|
||
`pipeline::run_chain_step_hrdps`, fetching from MSC Datamart, writing
|
||
scores to `<base>/<band>/<iso>.hrdps.prop` (sibling of the HRRR
|
||
`.prop` file for the same cycle). Cron entry lives in `runtime.exs`.
|
||
"""
|
||
|
||
use Oban.Worker,
|
||
queue: :hrdps,
|
||
priority: 0,
|
||
max_attempts: 5,
|
||
# Deduplicate seed jobs in a 6-hour window matching the cycle cadence.
|
||
unique: [period: 6 * 3600, states: [:available, :scheduled, :executing, :retryable]]
|
||
|
||
alias Microwaveprop.Instrument
|
||
alias Microwaveprop.Propagation.GridTaskEnqueuer
|
||
alias Microwaveprop.Weather.HrdpsClient
|
||
|
||
require Logger
|
||
|
||
@impl Oban.Worker
|
||
def perform(%Oban.Job{args: args}) when args == %{} do
|
||
Instrument.span([:hrdps, :grid_worker, :perform], %{}, fn ->
|
||
seed_chain()
|
||
end)
|
||
end
|
||
|
||
defp seed_chain do
|
||
# HRDPS shares grid_tasks with HRRR; the same reclaim sweep covers
|
||
# both because reclaim_stale_running/0 doesn't filter on source.
|
||
_ = GridTaskEnqueuer.reclaim_stale_running()
|
||
|
||
run_time = pick_run_time(DateTime.utc_now())
|
||
|
||
Logger.info("HrdpsGrid: seeding chain run_time=#{run_time} source=hrdps")
|
||
|
||
case GridTaskEnqueuer.seed_with_analysis(run_time, source: "hrdps") do
|
||
{:ok, count} ->
|
||
:telemetry.execute(
|
||
[:microwaveprop, :hrdps, :grid_worker, :seeded],
|
||
%{queued_tasks: count},
|
||
%{run_time: run_time}
|
||
)
|
||
|
||
:ok
|
||
|
||
{:error, reason} ->
|
||
:telemetry.execute(
|
||
[:microwaveprop, :hrdps, :grid_worker, :exception],
|
||
%{},
|
||
%{run_time: run_time, reason: inspect(reason)}
|
||
)
|
||
|
||
{:error, reason}
|
||
end
|
||
end
|
||
|
||
@doc """
|
||
Pick the freshest published HRDPS cycle. Probes `now-4h` (just past
|
||
typical publish latency) first; falls back to `now-10h` (one cycle
|
||
earlier) if the fresher cycle isn't yet on the datamart.
|
||
|
||
Public so tests can exercise selection logic without going through Oban.
|
||
"""
|
||
@spec pick_run_time(DateTime.t()) :: DateTime.t()
|
||
def pick_run_time(%DateTime{} = now) do
|
||
fresh = now |> DateTime.add(-4, :hour) |> HrdpsClient.nearest_hrdps_cycle() |> DateTime.truncate(:second)
|
||
|
||
if HrdpsClient.cycle_available?(fresh) do
|
||
fresh
|
||
else
|
||
now |> DateTime.add(-10, :hour) |> HrdpsClient.nearest_hrdps_cycle() |> DateTime.truncate(:second)
|
||
end
|
||
end
|
||
end
|