CLAUDE.md adds HRDPS to Key Data Sources and the Background Job Queues table with explicit "dormant until Rust ships" framing. The 2026-04-29 plan gets a Status snapshot at the top: which stages shipped (0-3 Elixir + 8) vs which are deferred to the Rust port sessions (4-7, 9). Activation path is one-paragraph: land Rust HRDPS, add cron entry, widen map bounds, extend FreshnessMonitor.
411 lines
24 KiB
Markdown
411 lines
24 KiB
Markdown
# HRDPS Canadian Propagation Grid Implementation Plan
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||
|
||
**Supersedes:** `docs/plans/2026-04-13-hrdps-canadian-prop-grid.md` — that plan was written before the probe wedge surfaced two architectural realities that change Stage 1's design: ECCC moved the datamart URL structure (the old `/model_hrdps/` root 404s now), and HRDPS publishes one variable per GRIB2 file rather than HRRR's bundled wrfsfcf/wrfprsf format. Major filenames, fetch shape, and idx assumptions are different.
|
||
|
||
**Goal:** Extend the propagation scoring grid into Canada at 0.125° (matching the existing CONUS sample density) sourced from HRDPS, so `/map` shows scores across all of North America. Hourly Canadian scores via the cycle's f00..f18 forecast hours; HRRR wins where it has coverage and HRDPS fills the rest.
|
||
|
||
**Architecture:** Build a parallel NWP ingestion pipeline (`HrdpsClient` → `HrdpsGridWorker` → `hrdps_profiles` table) that mirrors the HRRR shape. The single-variable-per-file difference is hidden inside `HrdpsClient`: it fetches the ~14 needed variable files concurrently, then runs `wgrib2 -merge` to produce a single multi-variable GRIB2 that downstream code can consume identically to a `wrfsfcf`/`wrfprsf` blob. HRDPS runs at 00/06/12/18Z and seeds f00..f18 (hourly) into the score chain, so Canadian scores refresh once per cycle but provide hourly forecast steps until the next cycle lands. Border stitching: a static `Grid.hrdps_only_points/0` mask carves Canada-only cells so HRRR and HRDPS never both write the same `(lat, lon)`.
|
||
|
||
**Tech Stack:** Elixir 1.15, Req, Postgres + Ecto, Oban, wgrib2 (already in image), Rust (for `prop-grid-rs` and `hrrr-point-rs` HRDPS branches)
|
||
|
||
**Prerequisite:** Stage 0 already done — the probe at `lib/mix/tasks/hrdps_probe.ex` retired the URL/projection/decode risks against the live datamart on 2026-04-29.
|
||
|
||
---
|
||
|
||
## Status snapshot (2026-04-29)
|
||
|
||
**Shipped — Elixir foundation complete:**
|
||
|
||
- ✅ Stage 0: Probe wedge (`mix hrdps.probe`)
|
||
- ✅ Stage 1: `HrdpsClient` with URL/variable catalog, concurrent fetch + byte-concat, `fetch_grid/3`, `cycle_available?/1` — `lib/microwaveprop/weather/hrdps_client.ex`
|
||
- ✅ Stage 2: `hrdps_profiles` partitioned table + `HrdpsProfile` schema
|
||
- ✅ Stage 3 (Elixir-side): `Grid.hrdps_only_points/0`, `Grid.point_source/1`, `grid_tasks.source` column, `GridTaskEnqueuer` source-aware seeding, `HrdpsGridWorker` seeder
|
||
- ✅ Stage 8: SRTM scope decision recorded — Option 1 (cap Canadian coverage at 60°N for v1; defer Arctic CDEM to a follow-up plan)
|
||
|
||
**Deferred — needs the Rust counterpart before it can ship:**
|
||
|
||
- ⏸ Stage 4: `prop-grid-rs` Rust HRDPS branch (~4 days). Until this ships, `HrdpsGridWorker` stays out of the cron — its module doc explicitly notes it is dormant scaffolding.
|
||
- ⏸ Stage 5: `hrrr-point-rs` Rust HRDPS branch (~1.5 days)
|
||
- ⏸ Stage 6: cycle scheduling cron entry — depends on Stage 4
|
||
- ⏸ Stage 7: map overlay bounds widening — depends on real data in `/data/scores`
|
||
- ⏸ Stage 9: FreshnessMonitor HRDPS staleness tracking — depends on real cycle data flowing
|
||
|
||
**Activation path once Rust ships:**
|
||
|
||
1. Land the Rust HRDPS branch + redeploy `prop-grid-rs` and `hrrr-point-rs`.
|
||
2. Add to `config/runtime.exs` cron: `{"35 5,11,17,23 * * *", Microwaveprop.Workers.HrdpsGridWorker}`.
|
||
3. Widen `MapLive.@initial_bounds` to `(23.5, 60.0, -141.0, -52.0)`.
|
||
4. Extend `FreshnessMonitor` with `hrdps_last_run_at`.
|
||
|
||
The hardest decisions (cycle cadence, border stitching, variable mapping, projection) are settled and encoded in code. The remaining work is mechanical port-and-wire-up.
|
||
|
||
---
|
||
|
||
## Stage 0: Probe wedge (DONE)
|
||
|
||
`mix hrdps.probe` proves the GRIB2 fetch + wgrib2 reprojection works end-to-end against the real ECCC datamart. Output verifies plausible Canadian temperatures at 5 cities. Throwaway code; will be deleted in Stage 10 once `HrdpsClient` covers the same surface area.
|
||
|
||
Findings that drive the rest of the plan:
|
||
|
||
- **URL:** `https://dd.weather.gc.ca/{YYYYMMDD}/WXO-DD/model_hrdps/continental/2.5km/{HH}/{FFF}/{YYYYMMDD}T{HH}Z_MSC_HRDPS_{VAR}_{LEVEL}_RLatLon0.0225_PT{FFF}H.grib2`
|
||
- **Variable naming:** MSC's `{VAR}_{LEVEL_TYPE}-{LEVEL_VALUE}` (e.g. `TMP_AGL-2m`, `TMP_ISBL_0850`) — different from NCEP's HRRR short-name format
|
||
- **One variable per file** (~3.5 MB each); no idx file / no byte-range partials
|
||
- **wgrib2 reprojects rotated lat/lon natively** via `-lon LON LAT` — no manual rotation math
|
||
- **Publish latency:** ~3-4h after each cycle (verified by cycle availability at probe time)
|
||
|
||
---
|
||
|
||
## Stage 1: `HrdpsClient` — per-variable fetch + wgrib2 -merge
|
||
|
||
**Decision recorded:** Fetch each variable's GRIB2 file concurrently, then run `wgrib2 -merge` to combine into a single multi-variable file before downstream decode. Keeps the existing `hrrr_profiles`-shaped derive pipeline intact downstream of the client. The disk-staging cost (~14 files × 3.5 MB into a tmp dir per cycle/forecast hour) is small compared to the fetch time.
|
||
|
||
**Files:**
|
||
- Create: `lib/microwaveprop/weather/hrdps_client.ex`
|
||
- Create: `test/microwaveprop/weather/hrdps_client_test.exs`
|
||
- Create: `test/support/fixtures/hrdps_sample_*.grib2` (small fixtures captured from a real cycle)
|
||
|
||
### Task 1.1: URL builder (TDD)
|
||
|
||
Write a failing test for `HrdpsClient.grib_url(:tmp_2m, run_time, forecast_hour)` returning the date-prefixed datamart URL with the MSC filename format. Verify against a known-good URL from the probe (`20260429T12Z_MSC_HRDPS_TMP_AGL-2m_RLatLon0.0225_PT000H.grib2`). Implement; commit.
|
||
|
||
### Task 1.2: Variable catalog
|
||
|
||
Define a `@variables` map mapping internal atoms (`:tmp_2m`, `:dpt_2m`, `:pres_sfc`, `:wind_u_10m`, `:wind_v_10m`, `:apcp_1h`, `:tcdc`, `:pwat`, `:tmp_850mb`, `:tmp_700mb`, `:tmp_500mb`, `:dpt_850mb`, `:dpt_700mb`, `:hgt_850mb`) to MSC variable + level slugs (`{VAR_AGL-2m}`, `{VAR_ISBL_0850}`, etc.).
|
||
|
||
This is the variable-mapping table the old plan called out as a Stage-1 risk; build it explicitly. TDD: a `variable_slug/1` test for each atom. Commit.
|
||
|
||
### Task 1.3: Concurrent fetch + tmp staging (TDD)
|
||
|
||
Write a failing test for `fetch_cycle_files(run_time, forecast_hour, opts)` that returns `{:ok, [paths]}` of the per-variable GRIB2 files staged in a tmp dir. Use `Task.async_stream` with `max_concurrency: 4` (datamart isn't aggressively rate-limited but we don't need >4 in flight). Mock with `Req.Test.stub` per project conventions.
|
||
|
||
Critical: every `{:exit, _}` and `{:error, _}` clause from the async_stream must `Logger.error` with variable name + URL — per CLAUDE.md "always log anything that would normally be swallowed" and the path-calculator outage memory. A silently-dropped fetch in production is a Canadian dead-zone we won't notice for hours.
|
||
|
||
Implement; commit.
|
||
|
||
### Task 1.4: wgrib2 -merge into multi-variable file (TDD)
|
||
|
||
Write a failing test for `merge_to_multivar(paths, output_path)`. Implement via `System.cmd("wgrib2", ["-merge", path1, path2, ..., "-grib", output_path])`. Verify the merged output's `wgrib2 file -V` lists all expected variables.
|
||
|
||
Commit.
|
||
|
||
### Task 1.5: `fetch_grid/3` — public API matching `HrrrClient.fetch_grid/3`
|
||
|
||
Same signature as HRRR so Stage 3's worker can call either polymorphically:
|
||
|
||
```elixir
|
||
@spec fetch_grid([{float(), float()}], DateTime.t(), keyword()) ::
|
||
{:ok, %{{float(), float()} => map()}} | {:error, term()}
|
||
def fetch_grid(points, run_time, opts \\ [])
|
||
```
|
||
|
||
Internally: `fetch_cycle_files` → `merge_to_multivar` → reuse `Microwaveprop.Weather.GribDecoder` (the existing HRRR decoder, projection-agnostic since wgrib2 handles rotation).
|
||
|
||
TDD; commit.
|
||
|
||
### Task 1.6: Projection sanity check (TDD)
|
||
|
||
Pick CYYR (Goose Bay, 53.32°N, -60.42°W) and a recent cycle for which we have a UWYO Canadian sounding (those exist via `CanadianSoundingFetchWorker`). Test that the HRDPS surface temp at CYYR is within ±3°C of the radiosonde surface temp for the same `valid_time`. If they don't match, the projection is wrong and `wgrib2 -lon` is being called incorrectly.
|
||
|
||
This is the most important test in the stage — burn whatever time is needed to make it pass. A 10 km projection offset is invisible on the map but destroys terrain analysis at Stage 5+.
|
||
|
||
Commit.
|
||
|
||
---
|
||
|
||
## Stage 2: `hrdps_profiles` schema
|
||
|
||
**Files:**
|
||
- Create: `priv/repo/migrations/{timestamp}_create_hrdps_profiles.exs`
|
||
- Create: `lib/microwaveprop/weather/hrdps_profile.ex`
|
||
- Create: `test/microwaveprop/weather/hrdps_profile_test.exs`
|
||
|
||
### Task 2.1: Migration mirrors `hrrr_profiles`
|
||
|
||
Same columns, same indexes, daily partitioning on `valid_time`. Separate table (not a unified `nwp_profiles`) so HRDPS can be dropped/reprocessed without disturbing the 4500+ HRRR profiles already in prod.
|
||
|
||
`@primary_key {:id, :binary_id, autogenerate: true}` per project convention.
|
||
|
||
Run migration in dev; verify table + indexes; commit.
|
||
|
||
### Task 2.2: `HrdpsProfile` schema + changeset (TDD)
|
||
|
||
Identical to `HrrrProfile`. Same derived fields (`surface_refractivity`, `min_refractivity_gradient`, `ducting_detected`, `duct_characteristics`, `pwat_mm`, `hpbl_m`).
|
||
|
||
Commit.
|
||
|
||
### Task 2.3: Partition pruning cron
|
||
|
||
Add an `Oban.Plugins.Cron` entry mirroring whatever HRRR uses (or add one if HRRR doesn't have one yet — the prod DB will fill up either way).
|
||
|
||
Commit.
|
||
|
||
---
|
||
|
||
## Stage 3: `HrdpsGridWorker` — score Canadian grid per forecast hour
|
||
|
||
**Files:**
|
||
- Create: `lib/microwaveprop/workers/hrdps_grid_worker.ex`
|
||
- Create: `test/microwaveprop/workers/hrdps_grid_worker_test.exs`
|
||
- Modify: `lib/microwaveprop/propagation/grid.ex` — add `hrdps_only_points/0`
|
||
|
||
### Task 3.1: Define the Canadian-only grid (TDD)
|
||
|
||
`hrdps_only_points/0` returns the subset of a 0.125°-spaced grid that is INSIDE the Canadian land mask AND OUTSIDE the existing CONUS HRRR footprint. Implement as a precomputed list (compile-time `@external_resource`) — the cells don't change.
|
||
|
||
Approximate bounds: 42°N to 60°N (north-of-60°N deferred to Stage 8), -141°W to -52°W. Filter to land using a coarse polygon — Natural Earth's `ne_50m_admin_0_countries` clipped to Canada is fine; even a rectangular bounding-box pass-through is acceptable for a first pass since over-water cells just get scored and ignored.
|
||
|
||
The mask must NEVER overlap HRRR cells. Test: `MapSet.disjoint?(MapSet.new(Grid.conus_points()), MapSet.new(Grid.hrdps_only_points()))`.
|
||
|
||
Commit.
|
||
|
||
### Task 3.2: Worker `perform/1` (TDD)
|
||
|
||
Mirror `PropagationGridWorker.process_forecast_hour/4`. Args: `{run_time, forecast_hour}`. Calls `HrdpsClient.fetch_grid(hrdps_only_points(), run_time, opts)`, derives profiles via the existing `Microwaveprop.Weather.SoundingParams`-style derivation, persists to `hrdps_profiles`, then scores via `Microwaveprop.Propagation.Scorer.score/3`.
|
||
|
||
`start_async`/`Task.async_stream` failure paths must `Logger.error` with point + reason (per CLAUDE.md and the swallowed-async-errors memory).
|
||
|
||
Commit.
|
||
|
||
### Task 3.3: Score upserts into existing `propagation_scores`
|
||
|
||
Single shared scores table — HRDPS rows look identical to HRRR rows, distinguished only by `(lat, lon)` falling in the HRDPS-only mask. Use the same `on_conflict: {:replace_all_except, [...]}` upsert pattern.
|
||
|
||
Confirm via SQL: a sample run produces no `(lat, lon, valid_time, band_mhz)` collisions with HRRR rows.
|
||
|
||
Commit.
|
||
|
||
### Task 3.4: Score-file `.prop` writer
|
||
|
||
The `.prop` file at `/data/scores/...` is what `MicrowavepropWeb.ScoresController.cells/2` reads (and what `ScoreCache` mirrors in ETS). Either:
|
||
|
||
(a) extend the `.prop` writer to handle the larger union grid (CONUS + Canada), or
|
||
(b) emit a separate `_canada.prop` file that the controller stitches at read time.
|
||
|
||
Decision: **(a)** — single file, simpler downstream. The Rust scorer (Stage 4) writes the same file; both must agree on the cell layout. Update `Microwaveprop.Propagation.ScoresFile` to support a wider grid extent. The cell-pack format already handles arbitrary lat/lon arrays per the `PSCR` magic spec.
|
||
|
||
TDD: write a fixture `.prop` file at the expanded extent, read it back, verify all cells round-trip. Commit.
|
||
|
||
---
|
||
|
||
## Stage 4: `prop-grid-rs` Rust worker — HRDPS branch
|
||
|
||
The Rust worker at `rust/prop_grid_rs/` claims tasks from `grid_tasks` via `FOR UPDATE SKIP LOCKED` and writes `.prop` files to `/data/scores`. It needs an HRDPS branch since it owns the hot path for f01..f18.
|
||
|
||
**Files:**
|
||
- Modify: `rust/prop_grid_rs/src/sources/` (likely add `hrdps.rs` next to `hrrr.rs`)
|
||
- Modify: `rust/prop_grid_rs/src/main.rs` — task source dispatch
|
||
- Migration: `priv/repo/migrations/{timestamp}_add_source_to_grid_tasks.exs`
|
||
|
||
### Task 4.1: Add `source` column to `grid_tasks`
|
||
|
||
`source TEXT NOT NULL DEFAULT 'hrrr'`. Backfill is a no-op since existing rows are all HRRR. Index on `(source, run_time)` for the worker's claim query.
|
||
|
||
Commit.
|
||
|
||
### Task 4.2: Rust HRDPS fetcher
|
||
|
||
Mirror the existing `hrrr.rs` shape. Differences:
|
||
|
||
- URL builder using the date-prefixed pattern + per-variable filenames
|
||
- Concurrent fetch of ~14 variable files (use `tokio::join_all` with a semaphore at 4)
|
||
- Shell out to `wgrib2 -merge` (same binary the Elixir client uses; no Rust GRIB2 reimplementation)
|
||
- Reuse the existing point-extraction code that wraps `wgrib2 -lon`
|
||
|
||
Keep the Rust HRDPS code as close to the Rust HRRR code as possible — copy-then-modify is fine; resist refactoring for shared abstraction until both branches are working.
|
||
|
||
Commit.
|
||
|
||
### Task 4.3: Task dispatch on `source` column
|
||
|
||
Worker's main loop branches on `task.source`: `"hrrr"` → existing path, `"hrdps"` → new path. The task seeder (`Microwaveprop.Workers.GridTaskEnqueuer`) populates `source` based on which worker enqueued the task.
|
||
|
||
Commit.
|
||
|
||
### Task 4.4: Build + push image
|
||
|
||
Existing CI pipeline (`.forgejo/workflows/build-grid-rs.yaml`) produces the `prop-grid-rs:main-*` image consumed by both `prop-grid-rs` and `hrrr-point-rs` deployments. New code rolls out automatically once merged. Verify in Argo / pod logs.
|
||
|
||
---
|
||
|
||
## Stage 5: `hrrr-point-rs` — HRDPS branch for per-QSO Canadian QSOs
|
||
|
||
**Files:**
|
||
- Modify: `rust/prop_grid_rs/src/bin/hrrr_point_worker.rs`
|
||
- Migration: `priv/repo/migrations/{timestamp}_add_source_to_hrrr_fetch_tasks.exs`
|
||
- Modify: `lib/microwaveprop/workers/hrrr_fetch_enqueue_worker.ex` (or wherever per-QSO fetches are enqueued)
|
||
|
||
### Task 5.1: Routing decision
|
||
|
||
When a QSO's lat/lon falls outside the HRRR CONUS footprint, enqueue an HRDPS-source task instead of an HRRR task. `Grid.point_source({lat, lon})` returns `:hrrr | :hrdps | :neither` (e.g. for a southern-hemisphere QSO). For `:neither`, fall back to the existing IEMRE/RAOB enrichment path.
|
||
|
||
TDD: cells at (45.0, -90.0) → `:hrrr`, (53.0, -113.0) → `:hrdps`, (-30.0, 150.0) → `:neither`.
|
||
|
||
Commit.
|
||
|
||
### Task 5.2: `hrrr_fetch_tasks.source` column + Rust branch
|
||
|
||
Same shape as Stage 4.1/4.3: add `source` column, branch the Rust worker, reuse Stage 1's URL/variable patterns ported to Rust.
|
||
|
||
Commit.
|
||
|
||
### Task 5.3: Per-QSO upserts into `hrdps_profiles`
|
||
|
||
QSO enrichment status column: add `hrdps_status` to `contacts` (similar to existing `hrrr_status`). LiveView/admin UI reads either, depending on which model covered the QSO.
|
||
|
||
Commit.
|
||
|
||
---
|
||
|
||
## Stage 6: Cycle scheduling + border stitching
|
||
|
||
**Files:**
|
||
- Modify: `config/runtime.exs` — Oban cron + queue
|
||
- Modify: `lib/microwaveprop/workers/grid_task_enqueuer.ex` (or equivalent)
|
||
|
||
### Task 6.1: Add `hrdps` queue
|
||
|
||
Production runtime config adds `hrdps: 1` (single concurrent fetch — wgrib2 working set during merge can hit ~500 MB and we don't want it competing with `propagation`).
|
||
|
||
### Task 6.2: Cycle cron at HH:35 of cycle hours
|
||
|
||
```elixir
|
||
{"35 5,11,17,23 * * *", Microwaveprop.Workers.HrdpsCycleSeedWorker}
|
||
```
|
||
|
||
Why these times: HRDPS runs at 00/06/12/18Z and publishes ~3-4h later, so 5h after cycle (at 05:35Z, 11:35Z, 17:35Z, 23:35Z) the f00 file is reliably available. Offset by 30 minutes from HRRR's `*/15` schedule to avoid wgrib2 contention.
|
||
|
||
`HrdpsCycleSeedWorker` enqueues f00..f18 grid tasks with `source: "hrdps"`. The existing chain worker (Rust + Elixir) drains them.
|
||
|
||
### Task 6.3: Border stitching rule
|
||
|
||
Already enforced at Stage 3.1: HRDPS-only mask cells are disjoint from HRRR's CONUS points. No runtime conflict. The seam at 49°N is implicit: HRRR cells exist down to ~25°N, HRDPS cells exist 49°N+. The boundary is a one-cell band of HRRR coverage abutting HRDPS — no overlap, no double-write.
|
||
|
||
Document the rule in `Grid.point_source/1`'s `@moduledoc` so future readers understand why the masks are exclusive.
|
||
|
||
Commit.
|
||
|
||
---
|
||
|
||
## Stage 7: Map overlay extension
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop_web/live/map_live.ex` — `@initial_bounds`
|
||
- Modify: `assets/js/propagation_map_hook.ts` — heatmap rendering bounds
|
||
|
||
### Task 7.1: Widen `@initial_bounds`
|
||
|
||
CONUS-only `(23.5, 49.5, -124.5, -66.9)` → `(23.5, 60.0, -141.0, -52.0)` (excluding Arctic until Stage 8 decides). Default zoom adjusted so all of North America is visible.
|
||
|
||
### Task 7.2: Smoke test (manual + LiveView test)
|
||
|
||
`mix phx.server`, open `/map`, confirm colored cells render north of 49°N after `HrdpsCycleSeedWorker` fires. The cell-pack endpoint at `/scores/cells` already serves whatever lat/lon points exist in the `.prop` file — no controller changes needed since Stage 3.4 expanded the file.
|
||
|
||
LiveView test: assert that an `/scores/cells?band=10000&south=50&north=55&west=-115&east=-105` request returns >0 cells after HRDPS data exists in fixtures.
|
||
|
||
### Task 7.3: ScoreCache chunk layout still correct
|
||
|
||
The ScoreCache layout (just shipped in commit 6a90d153 follow-up) uses 5°×5° chunks. Extending the grid into Canada adds ~30 new chunks but the same lookup logic applies. No code change; one regression test that fetches a Canadian-bounds viewport and asserts hit/miss telemetry works correctly.
|
||
|
||
Commit.
|
||
|
||
---
|
||
|
||
## Stage 8: SRTM gap above 60°N — scope decision
|
||
|
||
**Decision gate.** SRTM1 stops at 60°N. Yellowknife (62.45°N), Whitehorse (60.72°N), Inuvik (68.36°N), and Resolute (74.70°N) have NO SRTM coverage — terrain diffraction analysis would silently return wrong losses for paths involving those stations.
|
||
|
||
Three options:
|
||
|
||
1. **Cap Canadian coverage at 60°N for v1.** Document explicitly in `algo.md`. Most VE QSOs are below 60°N anyway. Defer Arctic to a follow-up plan.
|
||
2. **Use CDEM (Canadian Digital Elevation Model) above 60°N.** ~1 week of additional work — different format (TIFF tiles), different mounting strategy on the production NFS share. Touches `terrain_analysis.ex` and `elevation_client.ex`.
|
||
3. **Use ASTER GDEM v3.** Global, free, 30 m resolution, but registration-walled. Not great.
|
||
|
||
**Recommendation: Option 1 for v1.** Ship the 42-60°N coverage. If users report Arctic gaps, do CDEM as a separate plan.
|
||
|
||
Decision recorded in this stage; file a follow-up issue if Option 2 is pursued.
|
||
|
||
---
|
||
|
||
## Stage 9: Freshness monitoring + observability
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop/propagation/freshness_monitor.ex`
|
||
- Modify: wherever `/status` exposes data freshness (LiveView or controller)
|
||
|
||
### Task 9.1: Track HRDPS run-time staleness
|
||
|
||
Add `hrdps_last_run_at` to the monitor state. Surface on the status page next to HRRR's freshness.
|
||
|
||
### Task 9.2: Alert at >5h stale
|
||
|
||
HRDPS publishes ~4h after each cycle, so a healthy pipeline never exceeds ~5h between runs. >5h = something broken (datamart outage, our worker failed silently, k8s scheduling glitch).
|
||
|
||
`Logger.warning` (not `error` — datamart outages are common and recoverable). Telemetry event for the existing dashboard scrape.
|
||
|
||
Commit.
|
||
|
||
---
|
||
|
||
## Stage 10: Precommit, docs, ship
|
||
|
||
### Task 10.1: `mix precommit`
|
||
|
||
Must show 0 failures, 0 credo issues, all dialyzer warnings resolved. Full project gate.
|
||
|
||
### Task 10.2: Update `CLAUDE.md`
|
||
|
||
- Add HRDPS to **Key Data Sources**
|
||
- Add `HrdpsCycleSeedWorker` + `HrdpsGridWorker` + `HrdpsFetchWorker` to the **Background Job Queues** table
|
||
- Note the 60°N coverage cap and the cycle cadence (4×/day vs HRRR's hourly)
|
||
|
||
### Task 10.3: Update `memory/reference_hrdps_canada.md`
|
||
|
||
Mark plan as executed. Capture any projection gotchas, the final cycle latency observed in prod, and the actual border-stitching seam location.
|
||
|
||
### Task 10.4: Delete the probe wedge
|
||
|
||
`lib/mix/tasks/hrdps_probe.ex` is throwaway scaffolding — its risks were retired in Stage 0; the production code in `HrdpsClient` covers the same surface. Delete in this stage's commit, not earlier (keeping it around during stages 1-3 is useful for cross-checking the real client's output).
|
||
|
||
### Task 10.5: Push to `github` AND `origin`
|
||
|
||
Per the push-both-remotes feedback memory. Do NOT push to `dokku`. Argo/Flux handles the rollout from the kustomize manifest pulled from `git.mcintire.me`.
|
||
|
||
---
|
||
|
||
## Known risks + gotchas
|
||
|
||
- **Projection** — HRDPS rotated lat/lon is the single biggest bug source. Stage 1.6's CYYR validation against UWYO is the safety net; do not skip.
|
||
- **Per-variable file fan-out** — ~14 fetches per cycle/forecast hour × 4 cycles × 19 forecast hours = 1064 files/day. Each is small (~3.5 MB) but datamart's `200 OK` latency is variable. Concurrency-cap at 4 to avoid hammering ECCC; build retry-with-backoff into `HrdpsClient`.
|
||
- **Memory pressure** — wgrib2 `-merge` working set during the multi-variable combine peaks at ~500 MB per cycle. The HRRR worker already runs at `:system_memory_high_watermark` headroom; running HRDPS on the same pod risks OOM. Either pin HRDPS to a dedicated `prop-backfill`-style pod (probably overkill) OR keep it on the hot pods but cap `hrdps: 1` so it never overlaps with `propagation: 1`.
|
||
- **Cycle publish latency** — datamart can lag 90 min on good days, 4+h under load. The `HrdpsCycleSeedWorker` cron at HH:35 has a 1.5h margin which is comfortable but not infinite. If a cycle hasn't published by HH:35, the worker should fall back to seeding the previous cycle (still gives current+forecast scores, just one cycle stale).
|
||
- **Border seam visibility** — at the 49°N seam, HRRR's 0.125° cells abut HRDPS's 0.125° cells with no overlap. If the two models disagree by 5-10% on a given factor at the seam, users will see a faint horizontal stripe. Probably acceptable; document in `algo.md`.
|
||
- **Scope creep** — keep RDPS pressure-level coverage out of this plan even though they're related (RDPS plan is `2026-04-13-rdps-vertical-profiles.md`). Ship CONUS+Canada-via-HRDPS first; revisit RDPS for wider/coarser coverage afterward.
|
||
|
||
## Rough effort estimate
|
||
|
||
| Stage | Estimate |
|
||
|---|---|
|
||
| 0: Probe wedge | DONE (~3h, 2026-04-29) |
|
||
| 1: HrdpsClient | 4 days (Task 1.6 projection debug dominates) |
|
||
| 2: hrdps_profiles schema | 0.5 day |
|
||
| 3: HrdpsGridWorker + .prop writer | 3 days |
|
||
| 4: Rust prop-grid-rs HRDPS branch | 4 days |
|
||
| 5: Rust hrrr-point-rs HRDPS branch | 1.5 days |
|
||
| 6: Cycle scheduling + border stitching | 1 day |
|
||
| 7: Map overlay extension | 0.5 day |
|
||
| 8: SRTM scope decision | 0.5 day (decision + documentation, no code if Option 1) |
|
||
| 9: Freshness monitoring | 0.5 day |
|
||
| 10: Precommit + docs + ship | 0.5 day |
|
||
| **Total** | **~16 days (3 weeks calendar)** |
|
||
|
||
The Rust worker stages (4 + 5) are the majority of the schedule risk. If schedule is tight, an alternative is to ship Elixir-only first (Stages 1-3 + 6-10) and follow up with the Rust port — at the cost of slower Canadian forecast-chain processing during the gap.
|
||
|
||
---
|
||
|
||
## Open questions for confirmation before Stage 1
|
||
|
||
1. **Variable set:** Does the catalog at Task 1.2 cover everything the scorer needs for Canadian cells? Worth a 30-min look at `Microwaveprop.Propagation.Scorer.factors/2` to confirm the surface + pressure-level set is complete (especially native-level duct analysis — HRDPS's hybrid sigma-pressure levels may be available under a `*_HY_*` or `native/` directory that the probe didn't explore).
|
||
2. **60°N cap:** Confirm Option 1 from Stage 8 (defer Arctic CDEM) before kickoff so we don't end up with the Stage 1.1 grid mask covering territory we won't actually serve.
|
||
3. **Forecast horizon:** Default plan uses HRDPS f00..f18 to match HRRR's window. HRDPS goes out to f48 — would f00..f24 or even f00..f48 (full Day-2 forecast, only on the Canadian side) be useful? Storage/compute roughly doubles per extra 12h of horizon.
|