docs(hrdps): plan + probe wedge for Canadian propagation grid
Adds mix hrdps.probe — throwaway wedge that retires URL/projection/decode
risks against MSC Datamart. Verified against five Canadian cities with
plausible 2m temperatures via wgrib2's native rotated-lat/lon support.
Findings update what we knew:
- ECCC reorganized to date-prefixed paths; the old /model_hrdps/ root
404s now. Real URL is dd.weather.gc.ca/{YYYYMMDD}/WXO-DD/model_hrdps/...
- HRDPS publishes one variable per GRIB2 file (~3.5 MB each), not
HRRR's bundled wrfsfcf format. ~14 files per cycle/forecast hour.
- wgrib2 -lon LON LAT handles the rotated grid natively — no manual
reprojection math needed.
New plan at 2026-04-29-hrdps-canadian-prop-grid.md supersedes the
2026-04-13 one (which assumed HRRR-style idx + byte-range fetches that
don't apply). Probe deletion is a Stage 10 step in the new plan.
This commit is contained in:
parent
c20aceba95
commit
14e81f9251
3 changed files with 550 additions and 1 deletions
|
|
@ -1,4 +1,6 @@
|
|||
# HRDPS Canadian Propagation Grid Implementation Plan
|
||||
# HRDPS Canadian Propagation Grid Implementation Plan (SUPERSEDED)
|
||||
|
||||
> **SUPERSEDED 2026-04-29 by `docs/plans/2026-04-29-hrdps-canadian-prop-grid.md`.** A live probe of MSC Datamart on 2026-04-29 found that ECCC reorganized the URL structure (the `/model_hrdps/` flat root in the pre-flight steps below 404s now) and that HRDPS publishes one variable per GRIB2 file rather than HRRR's bundled wrfsfcf/wrfprsf format — so the Stage 1 idx + byte-range fetch shape this plan inherits from HrrrClient does not apply. Use the 2026-04-29 plan instead.
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
|
|
|
|||
382
docs/plans/2026-04-29-hrdps-canadian-prop-grid.md
Normal file
382
docs/plans/2026-04-29-hrdps-canadian-prop-grid.md
Normal file
|
|
@ -0,0 +1,382 @@
|
|||
# HRDPS Canadian Propagation Grid Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Supersedes:** `docs/plans/2026-04-13-hrdps-canadian-prop-grid.md` — that plan was written before the probe wedge surfaced two architectural realities that change Stage 1's design: ECCC moved the datamart URL structure (the old `/model_hrdps/` root 404s now), and HRDPS publishes one variable per GRIB2 file rather than HRRR's bundled wrfsfcf/wrfprsf format. Major filenames, fetch shape, and idx assumptions are different.
|
||||
|
||||
**Goal:** Extend the propagation scoring grid into Canada at 0.125° (matching the existing CONUS sample density) sourced from HRDPS, so `/map` shows scores across all of North America. Hourly Canadian scores via the cycle's f00..f18 forecast hours; HRRR wins where it has coverage and HRDPS fills the rest.
|
||||
|
||||
**Architecture:** Build a parallel NWP ingestion pipeline (`HrdpsClient` → `HrdpsGridWorker` → `hrdps_profiles` table) that mirrors the HRRR shape. The single-variable-per-file difference is hidden inside `HrdpsClient`: it fetches the ~14 needed variable files concurrently, then runs `wgrib2 -merge` to produce a single multi-variable GRIB2 that downstream code can consume identically to a `wrfsfcf`/`wrfprsf` blob. HRDPS runs at 00/06/12/18Z and seeds f00..f18 (hourly) into the score chain, so Canadian scores refresh once per cycle but provide hourly forecast steps until the next cycle lands. Border stitching: a static `Grid.hrdps_only_points/0` mask carves Canada-only cells so HRRR and HRDPS never both write the same `(lat, lon)`.
|
||||
|
||||
**Tech Stack:** Elixir 1.15, Req, Postgres + Ecto, Oban, wgrib2 (already in image), Rust (for `prop-grid-rs` and `hrrr-point-rs` HRDPS branches)
|
||||
|
||||
**Prerequisite:** Stage 0 already done — the probe at `lib/mix/tasks/hrdps_probe.ex` retired the URL/projection/decode risks against the live datamart on 2026-04-29.
|
||||
|
||||
---
|
||||
|
||||
## Stage 0: Probe wedge (DONE)
|
||||
|
||||
`mix hrdps.probe` proves the GRIB2 fetch + wgrib2 reprojection works end-to-end against the real ECCC datamart. Output verifies plausible Canadian temperatures at 5 cities. Throwaway code; will be deleted in Stage 10 once `HrdpsClient` covers the same surface area.
|
||||
|
||||
Findings that drive the rest of the plan:
|
||||
|
||||
- **URL:** `https://dd.weather.gc.ca/{YYYYMMDD}/WXO-DD/model_hrdps/continental/2.5km/{HH}/{FFF}/{YYYYMMDD}T{HH}Z_MSC_HRDPS_{VAR}_{LEVEL}_RLatLon0.0225_PT{FFF}H.grib2`
|
||||
- **Variable naming:** MSC's `{VAR}_{LEVEL_TYPE}-{LEVEL_VALUE}` (e.g. `TMP_AGL-2m`, `TMP_ISBL_0850`) — different from NCEP's HRRR short-name format
|
||||
- **One variable per file** (~3.5 MB each); no idx file / no byte-range partials
|
||||
- **wgrib2 reprojects rotated lat/lon natively** via `-lon LON LAT` — no manual rotation math
|
||||
- **Publish latency:** ~3-4h after each cycle (verified by cycle availability at probe time)
|
||||
|
||||
---
|
||||
|
||||
## Stage 1: `HrdpsClient` — per-variable fetch + wgrib2 -merge
|
||||
|
||||
**Decision recorded:** Fetch each variable's GRIB2 file concurrently, then run `wgrib2 -merge` to combine into a single multi-variable file before downstream decode. Keeps the existing `hrrr_profiles`-shaped derive pipeline intact downstream of the client. The disk-staging cost (~14 files × 3.5 MB into a tmp dir per cycle/forecast hour) is small compared to the fetch time.
|
||||
|
||||
**Files:**
|
||||
- Create: `lib/microwaveprop/weather/hrdps_client.ex`
|
||||
- Create: `test/microwaveprop/weather/hrdps_client_test.exs`
|
||||
- Create: `test/support/fixtures/hrdps_sample_*.grib2` (small fixtures captured from a real cycle)
|
||||
|
||||
### Task 1.1: URL builder (TDD)
|
||||
|
||||
Write a failing test for `HrdpsClient.grib_url(:tmp_2m, run_time, forecast_hour)` returning the date-prefixed datamart URL with the MSC filename format. Verify against a known-good URL from the probe (`20260429T12Z_MSC_HRDPS_TMP_AGL-2m_RLatLon0.0225_PT000H.grib2`). Implement; commit.
|
||||
|
||||
### Task 1.2: Variable catalog
|
||||
|
||||
Define a `@variables` map mapping internal atoms (`:tmp_2m`, `:dpt_2m`, `:pres_sfc`, `:wind_u_10m`, `:wind_v_10m`, `:apcp_1h`, `:tcdc`, `:pwat`, `:tmp_850mb`, `:tmp_700mb`, `:tmp_500mb`, `:dpt_850mb`, `:dpt_700mb`, `:hgt_850mb`) to MSC variable + level slugs (`{VAR_AGL-2m}`, `{VAR_ISBL_0850}`, etc.).
|
||||
|
||||
This is the variable-mapping table the old plan called out as a Stage-1 risk; build it explicitly. TDD: a `variable_slug/1` test for each atom. Commit.
|
||||
|
||||
### Task 1.3: Concurrent fetch + tmp staging (TDD)
|
||||
|
||||
Write a failing test for `fetch_cycle_files(run_time, forecast_hour, opts)` that returns `{:ok, [paths]}` of the per-variable GRIB2 files staged in a tmp dir. Use `Task.async_stream` with `max_concurrency: 4` (datamart isn't aggressively rate-limited but we don't need >4 in flight). Mock with `Req.Test.stub` per project conventions.
|
||||
|
||||
Critical: every `{:exit, _}` and `{:error, _}` clause from the async_stream must `Logger.error` with variable name + URL — per CLAUDE.md "always log anything that would normally be swallowed" and the path-calculator outage memory. A silently-dropped fetch in production is a Canadian dead-zone we won't notice for hours.
|
||||
|
||||
Implement; commit.
|
||||
|
||||
### Task 1.4: wgrib2 -merge into multi-variable file (TDD)
|
||||
|
||||
Write a failing test for `merge_to_multivar(paths, output_path)`. Implement via `System.cmd("wgrib2", ["-merge", path1, path2, ..., "-grib", output_path])`. Verify the merged output's `wgrib2 file -V` lists all expected variables.
|
||||
|
||||
Commit.
|
||||
|
||||
### Task 1.5: `fetch_grid/3` — public API matching `HrrrClient.fetch_grid/3`
|
||||
|
||||
Same signature as HRRR so Stage 3's worker can call either polymorphically:
|
||||
|
||||
```elixir
|
||||
@spec fetch_grid([{float(), float()}], DateTime.t(), keyword()) ::
|
||||
{:ok, %{{float(), float()} => map()}} | {:error, term()}
|
||||
def fetch_grid(points, run_time, opts \\ [])
|
||||
```
|
||||
|
||||
Internally: `fetch_cycle_files` → `merge_to_multivar` → reuse `Microwaveprop.Weather.GribDecoder` (the existing HRRR decoder, projection-agnostic since wgrib2 handles rotation).
|
||||
|
||||
TDD; commit.
|
||||
|
||||
### Task 1.6: Projection sanity check (TDD)
|
||||
|
||||
Pick CYYR (Goose Bay, 53.32°N, -60.42°W) and a recent cycle for which we have a UWYO Canadian sounding (those exist via `CanadianSoundingFetchWorker`). Test that the HRDPS surface temp at CYYR is within ±3°C of the radiosonde surface temp for the same `valid_time`. If they don't match, the projection is wrong and `wgrib2 -lon` is being called incorrectly.
|
||||
|
||||
This is the most important test in the stage — burn whatever time is needed to make it pass. A 10 km projection offset is invisible on the map but destroys terrain analysis at Stage 5+.
|
||||
|
||||
Commit.
|
||||
|
||||
---
|
||||
|
||||
## Stage 2: `hrdps_profiles` schema
|
||||
|
||||
**Files:**
|
||||
- Create: `priv/repo/migrations/{timestamp}_create_hrdps_profiles.exs`
|
||||
- Create: `lib/microwaveprop/weather/hrdps_profile.ex`
|
||||
- Create: `test/microwaveprop/weather/hrdps_profile_test.exs`
|
||||
|
||||
### Task 2.1: Migration mirrors `hrrr_profiles`
|
||||
|
||||
Same columns, same indexes, daily partitioning on `valid_time`. Separate table (not a unified `nwp_profiles`) so HRDPS can be dropped/reprocessed without disturbing the 4500+ HRRR profiles already in prod.
|
||||
|
||||
`@primary_key {:id, :binary_id, autogenerate: true}` per project convention.
|
||||
|
||||
Run migration in dev; verify table + indexes; commit.
|
||||
|
||||
### Task 2.2: `HrdpsProfile` schema + changeset (TDD)
|
||||
|
||||
Identical to `HrrrProfile`. Same derived fields (`surface_refractivity`, `min_refractivity_gradient`, `ducting_detected`, `duct_characteristics`, `pwat_mm`, `hpbl_m`).
|
||||
|
||||
Commit.
|
||||
|
||||
### Task 2.3: Partition pruning cron
|
||||
|
||||
Add an `Oban.Plugins.Cron` entry mirroring whatever HRRR uses (or add one if HRRR doesn't have one yet — the prod DB will fill up either way).
|
||||
|
||||
Commit.
|
||||
|
||||
---
|
||||
|
||||
## Stage 3: `HrdpsGridWorker` — score Canadian grid per forecast hour
|
||||
|
||||
**Files:**
|
||||
- Create: `lib/microwaveprop/workers/hrdps_grid_worker.ex`
|
||||
- Create: `test/microwaveprop/workers/hrdps_grid_worker_test.exs`
|
||||
- Modify: `lib/microwaveprop/propagation/grid.ex` — add `hrdps_only_points/0`
|
||||
|
||||
### Task 3.1: Define the Canadian-only grid (TDD)
|
||||
|
||||
`hrdps_only_points/0` returns the subset of a 0.125°-spaced grid that is INSIDE the Canadian land mask AND OUTSIDE the existing CONUS HRRR footprint. Implement as a precomputed list (compile-time `@external_resource`) — the cells don't change.
|
||||
|
||||
Approximate bounds: 42°N to 60°N (north-of-60°N deferred to Stage 8), -141°W to -52°W. Filter to land using a coarse polygon — Natural Earth's `ne_50m_admin_0_countries` clipped to Canada is fine; even a rectangular bounding-box pass-through is acceptable for a first pass since over-water cells just get scored and ignored.
|
||||
|
||||
The mask must NEVER overlap HRRR cells. Test: `MapSet.disjoint?(MapSet.new(Grid.conus_points()), MapSet.new(Grid.hrdps_only_points()))`.
|
||||
|
||||
Commit.
|
||||
|
||||
### Task 3.2: Worker `perform/1` (TDD)
|
||||
|
||||
Mirror `PropagationGridWorker.process_forecast_hour/4`. Args: `{run_time, forecast_hour}`. Calls `HrdpsClient.fetch_grid(hrdps_only_points(), run_time, opts)`, derives profiles via the existing `Microwaveprop.Weather.SoundingParams`-style derivation, persists to `hrdps_profiles`, then scores via `Microwaveprop.Propagation.Scorer.score/3`.
|
||||
|
||||
`start_async`/`Task.async_stream` failure paths must `Logger.error` with point + reason (per CLAUDE.md and the swallowed-async-errors memory).
|
||||
|
||||
Commit.
|
||||
|
||||
### Task 3.3: Score upserts into existing `propagation_scores`
|
||||
|
||||
Single shared scores table — HRDPS rows look identical to HRRR rows, distinguished only by `(lat, lon)` falling in the HRDPS-only mask. Use the same `on_conflict: {:replace_all_except, [...]}` upsert pattern.
|
||||
|
||||
Confirm via SQL: a sample run produces no `(lat, lon, valid_time, band_mhz)` collisions with HRRR rows.
|
||||
|
||||
Commit.
|
||||
|
||||
### Task 3.4: Score-file `.prop` writer
|
||||
|
||||
The `.prop` file at `/data/scores/...` is what `MicrowavepropWeb.ScoresController.cells/2` reads (and what `ScoreCache` mirrors in ETS). Either:
|
||||
|
||||
(a) extend the `.prop` writer to handle the larger union grid (CONUS + Canada), or
|
||||
(b) emit a separate `_canada.prop` file that the controller stitches at read time.
|
||||
|
||||
Decision: **(a)** — single file, simpler downstream. The Rust scorer (Stage 4) writes the same file; both must agree on the cell layout. Update `Microwaveprop.Propagation.ScoresFile` to support a wider grid extent. The cell-pack format already handles arbitrary lat/lon arrays per the `PSCR` magic spec.
|
||||
|
||||
TDD: write a fixture `.prop` file at the expanded extent, read it back, verify all cells round-trip. Commit.
|
||||
|
||||
---
|
||||
|
||||
## Stage 4: `prop-grid-rs` Rust worker — HRDPS branch
|
||||
|
||||
The Rust worker at `rust/prop_grid_rs/` claims tasks from `grid_tasks` via `FOR UPDATE SKIP LOCKED` and writes `.prop` files to `/data/scores`. It needs an HRDPS branch since it owns the hot path for f01..f18.
|
||||
|
||||
**Files:**
|
||||
- Modify: `rust/prop_grid_rs/src/sources/` (likely add `hrdps.rs` next to `hrrr.rs`)
|
||||
- Modify: `rust/prop_grid_rs/src/main.rs` — task source dispatch
|
||||
- Migration: `priv/repo/migrations/{timestamp}_add_source_to_grid_tasks.exs`
|
||||
|
||||
### Task 4.1: Add `source` column to `grid_tasks`
|
||||
|
||||
`source TEXT NOT NULL DEFAULT 'hrrr'`. Backfill is a no-op since existing rows are all HRRR. Index on `(source, run_time)` for the worker's claim query.
|
||||
|
||||
Commit.
|
||||
|
||||
### Task 4.2: Rust HRDPS fetcher
|
||||
|
||||
Mirror the existing `hrrr.rs` shape. Differences:
|
||||
|
||||
- URL builder using the date-prefixed pattern + per-variable filenames
|
||||
- Concurrent fetch of ~14 variable files (use `tokio::join_all` with a semaphore at 4)
|
||||
- Shell out to `wgrib2 -merge` (same binary the Elixir client uses; no Rust GRIB2 reimplementation)
|
||||
- Reuse the existing point-extraction code that wraps `wgrib2 -lon`
|
||||
|
||||
Keep the Rust HRDPS code as close to the Rust HRRR code as possible — copy-then-modify is fine; resist refactoring for shared abstraction until both branches are working.
|
||||
|
||||
Commit.
|
||||
|
||||
### Task 4.3: Task dispatch on `source` column
|
||||
|
||||
Worker's main loop branches on `task.source`: `"hrrr"` → existing path, `"hrdps"` → new path. The task seeder (`Microwaveprop.Workers.GridTaskEnqueuer`) populates `source` based on which worker enqueued the task.
|
||||
|
||||
Commit.
|
||||
|
||||
### Task 4.4: Build + push image
|
||||
|
||||
Existing CI pipeline (`.forgejo/workflows/build-grid-rs.yaml`) produces the `prop-grid-rs:main-*` image consumed by both `prop-grid-rs` and `hrrr-point-rs` deployments. New code rolls out automatically once merged. Verify in Argo / pod logs.
|
||||
|
||||
---
|
||||
|
||||
## Stage 5: `hrrr-point-rs` — HRDPS branch for per-QSO Canadian QSOs
|
||||
|
||||
**Files:**
|
||||
- Modify: `rust/prop_grid_rs/src/bin/hrrr_point_worker.rs`
|
||||
- Migration: `priv/repo/migrations/{timestamp}_add_source_to_hrrr_fetch_tasks.exs`
|
||||
- Modify: `lib/microwaveprop/workers/hrrr_fetch_enqueue_worker.ex` (or wherever per-QSO fetches are enqueued)
|
||||
|
||||
### Task 5.1: Routing decision
|
||||
|
||||
When a QSO's lat/lon falls outside the HRRR CONUS footprint, enqueue an HRDPS-source task instead of an HRRR task. `Grid.point_source({lat, lon})` returns `:hrrr | :hrdps | :neither` (e.g. for a southern-hemisphere QSO). For `:neither`, fall back to the existing IEMRE/RAOB enrichment path.
|
||||
|
||||
TDD: cells at (45.0, -90.0) → `:hrrr`, (53.0, -113.0) → `:hrdps`, (-30.0, 150.0) → `:neither`.
|
||||
|
||||
Commit.
|
||||
|
||||
### Task 5.2: `hrrr_fetch_tasks.source` column + Rust branch
|
||||
|
||||
Same shape as Stage 4.1/4.3: add `source` column, branch the Rust worker, reuse Stage 1's URL/variable patterns ported to Rust.
|
||||
|
||||
Commit.
|
||||
|
||||
### Task 5.3: Per-QSO upserts into `hrdps_profiles`
|
||||
|
||||
QSO enrichment status column: add `hrdps_status` to `contacts` (similar to existing `hrrr_status`). LiveView/admin UI reads either, depending on which model covered the QSO.
|
||||
|
||||
Commit.
|
||||
|
||||
---
|
||||
|
||||
## Stage 6: Cycle scheduling + border stitching
|
||||
|
||||
**Files:**
|
||||
- Modify: `config/runtime.exs` — Oban cron + queue
|
||||
- Modify: `lib/microwaveprop/workers/grid_task_enqueuer.ex` (or equivalent)
|
||||
|
||||
### Task 6.1: Add `hrdps` queue
|
||||
|
||||
Production runtime config adds `hrdps: 1` (single concurrent fetch — wgrib2 working set during merge can hit ~500 MB and we don't want it competing with `propagation`).
|
||||
|
||||
### Task 6.2: Cycle cron at HH:35 of cycle hours
|
||||
|
||||
```elixir
|
||||
{"35 5,11,17,23 * * *", Microwaveprop.Workers.HrdpsCycleSeedWorker}
|
||||
```
|
||||
|
||||
Why these times: HRDPS runs at 00/06/12/18Z and publishes ~3-4h later, so 5h after cycle (at 05:35Z, 11:35Z, 17:35Z, 23:35Z) the f00 file is reliably available. Offset by 30 minutes from HRRR's `*/15` schedule to avoid wgrib2 contention.
|
||||
|
||||
`HrdpsCycleSeedWorker` enqueues f00..f18 grid tasks with `source: "hrdps"`. The existing chain worker (Rust + Elixir) drains them.
|
||||
|
||||
### Task 6.3: Border stitching rule
|
||||
|
||||
Already enforced at Stage 3.1: HRDPS-only mask cells are disjoint from HRRR's CONUS points. No runtime conflict. The seam at 49°N is implicit: HRRR cells exist down to ~25°N, HRDPS cells exist 49°N+. The boundary is a one-cell band of HRRR coverage abutting HRDPS — no overlap, no double-write.
|
||||
|
||||
Document the rule in `Grid.point_source/1`'s `@moduledoc` so future readers understand why the masks are exclusive.
|
||||
|
||||
Commit.
|
||||
|
||||
---
|
||||
|
||||
## Stage 7: Map overlay extension
|
||||
|
||||
**Files:**
|
||||
- Modify: `lib/microwaveprop_web/live/map_live.ex` — `@initial_bounds`
|
||||
- Modify: `assets/js/propagation_map_hook.ts` — heatmap rendering bounds
|
||||
|
||||
### Task 7.1: Widen `@initial_bounds`
|
||||
|
||||
CONUS-only `(23.5, 49.5, -124.5, -66.9)` → `(23.5, 60.0, -141.0, -52.0)` (excluding Arctic until Stage 8 decides). Default zoom adjusted so all of North America is visible.
|
||||
|
||||
### Task 7.2: Smoke test (manual + LiveView test)
|
||||
|
||||
`mix phx.server`, open `/map`, confirm colored cells render north of 49°N after `HrdpsCycleSeedWorker` fires. The cell-pack endpoint at `/scores/cells` already serves whatever lat/lon points exist in the `.prop` file — no controller changes needed since Stage 3.4 expanded the file.
|
||||
|
||||
LiveView test: assert that an `/scores/cells?band=10000&south=50&north=55&west=-115&east=-105` request returns >0 cells after HRDPS data exists in fixtures.
|
||||
|
||||
### Task 7.3: ScoreCache chunk layout still correct
|
||||
|
||||
The ScoreCache layout (just shipped in commit 6a90d153 follow-up) uses 5°×5° chunks. Extending the grid into Canada adds ~30 new chunks but the same lookup logic applies. No code change; one regression test that fetches a Canadian-bounds viewport and asserts hit/miss telemetry works correctly.
|
||||
|
||||
Commit.
|
||||
|
||||
---
|
||||
|
||||
## Stage 8: SRTM gap above 60°N — scope decision
|
||||
|
||||
**Decision gate.** SRTM1 stops at 60°N. Yellowknife (62.45°N), Whitehorse (60.72°N), Inuvik (68.36°N), and Resolute (74.70°N) have NO SRTM coverage — terrain diffraction analysis would silently return wrong losses for paths involving those stations.
|
||||
|
||||
Three options:
|
||||
|
||||
1. **Cap Canadian coverage at 60°N for v1.** Document explicitly in `algo.md`. Most VE QSOs are below 60°N anyway. Defer Arctic to a follow-up plan.
|
||||
2. **Use CDEM (Canadian Digital Elevation Model) above 60°N.** ~1 week of additional work — different format (TIFF tiles), different mounting strategy on the production NFS share. Touches `terrain_analysis.ex` and `elevation_client.ex`.
|
||||
3. **Use ASTER GDEM v3.** Global, free, 30 m resolution, but registration-walled. Not great.
|
||||
|
||||
**Recommendation: Option 1 for v1.** Ship the 42-60°N coverage. If users report Arctic gaps, do CDEM as a separate plan.
|
||||
|
||||
Decision recorded in this stage; file a follow-up issue if Option 2 is pursued.
|
||||
|
||||
---
|
||||
|
||||
## Stage 9: Freshness monitoring + observability
|
||||
|
||||
**Files:**
|
||||
- Modify: `lib/microwaveprop/propagation/freshness_monitor.ex`
|
||||
- Modify: wherever `/status` exposes data freshness (LiveView or controller)
|
||||
|
||||
### Task 9.1: Track HRDPS run-time staleness
|
||||
|
||||
Add `hrdps_last_run_at` to the monitor state. Surface on the status page next to HRRR's freshness.
|
||||
|
||||
### Task 9.2: Alert at >5h stale
|
||||
|
||||
HRDPS publishes ~4h after each cycle, so a healthy pipeline never exceeds ~5h between runs. >5h = something broken (datamart outage, our worker failed silently, k8s scheduling glitch).
|
||||
|
||||
`Logger.warning` (not `error` — datamart outages are common and recoverable). Telemetry event for the existing dashboard scrape.
|
||||
|
||||
Commit.
|
||||
|
||||
---
|
||||
|
||||
## Stage 10: Precommit, docs, ship
|
||||
|
||||
### Task 10.1: `mix precommit`
|
||||
|
||||
Must show 0 failures, 0 credo issues, all dialyzer warnings resolved. Full project gate.
|
||||
|
||||
### Task 10.2: Update `CLAUDE.md`
|
||||
|
||||
- Add HRDPS to **Key Data Sources**
|
||||
- Add `HrdpsCycleSeedWorker` + `HrdpsGridWorker` + `HrdpsFetchWorker` to the **Background Job Queues** table
|
||||
- Note the 60°N coverage cap and the cycle cadence (4×/day vs HRRR's hourly)
|
||||
|
||||
### Task 10.3: Update `memory/reference_hrdps_canada.md`
|
||||
|
||||
Mark plan as executed. Capture any projection gotchas, the final cycle latency observed in prod, and the actual border-stitching seam location.
|
||||
|
||||
### Task 10.4: Delete the probe wedge
|
||||
|
||||
`lib/mix/tasks/hrdps_probe.ex` is throwaway scaffolding — its risks were retired in Stage 0; the production code in `HrdpsClient` covers the same surface. Delete in this stage's commit, not earlier (keeping it around during stages 1-3 is useful for cross-checking the real client's output).
|
||||
|
||||
### Task 10.5: Push to `github` AND `origin`
|
||||
|
||||
Per the push-both-remotes feedback memory. Do NOT push to `dokku`. Argo/Flux handles the rollout from the kustomize manifest pulled from `git.mcintire.me`.
|
||||
|
||||
---
|
||||
|
||||
## Known risks + gotchas
|
||||
|
||||
- **Projection** — HRDPS rotated lat/lon is the single biggest bug source. Stage 1.6's CYYR validation against UWYO is the safety net; do not skip.
|
||||
- **Per-variable file fan-out** — ~14 fetches per cycle/forecast hour × 4 cycles × 19 forecast hours = 1064 files/day. Each is small (~3.5 MB) but datamart's `200 OK` latency is variable. Concurrency-cap at 4 to avoid hammering ECCC; build retry-with-backoff into `HrdpsClient`.
|
||||
- **Memory pressure** — wgrib2 `-merge` working set during the multi-variable combine peaks at ~500 MB per cycle. The HRRR worker already runs at `:system_memory_high_watermark` headroom; running HRDPS on the same pod risks OOM. Either pin HRDPS to a dedicated `prop-backfill`-style pod (probably overkill) OR keep it on the hot pods but cap `hrdps: 1` so it never overlaps with `propagation: 1`.
|
||||
- **Cycle publish latency** — datamart can lag 90 min on good days, 4+h under load. The `HrdpsCycleSeedWorker` cron at HH:35 has a 1.5h margin which is comfortable but not infinite. If a cycle hasn't published by HH:35, the worker should fall back to seeding the previous cycle (still gives current+forecast scores, just one cycle stale).
|
||||
- **Border seam visibility** — at the 49°N seam, HRRR's 0.125° cells abut HRDPS's 0.125° cells with no overlap. If the two models disagree by 5-10% on a given factor at the seam, users will see a faint horizontal stripe. Probably acceptable; document in `algo.md`.
|
||||
- **Scope creep** — keep RDPS pressure-level coverage out of this plan even though they're related (RDPS plan is `2026-04-13-rdps-vertical-profiles.md`). Ship CONUS+Canada-via-HRDPS first; revisit RDPS for wider/coarser coverage afterward.
|
||||
|
||||
## Rough effort estimate
|
||||
|
||||
| Stage | Estimate |
|
||||
|---|---|
|
||||
| 0: Probe wedge | DONE (~3h, 2026-04-29) |
|
||||
| 1: HrdpsClient | 4 days (Task 1.6 projection debug dominates) |
|
||||
| 2: hrdps_profiles schema | 0.5 day |
|
||||
| 3: HrdpsGridWorker + .prop writer | 3 days |
|
||||
| 4: Rust prop-grid-rs HRDPS branch | 4 days |
|
||||
| 5: Rust hrrr-point-rs HRDPS branch | 1.5 days |
|
||||
| 6: Cycle scheduling + border stitching | 1 day |
|
||||
| 7: Map overlay extension | 0.5 day |
|
||||
| 8: SRTM scope decision | 0.5 day (decision + documentation, no code if Option 1) |
|
||||
| 9: Freshness monitoring | 0.5 day |
|
||||
| 10: Precommit + docs + ship | 0.5 day |
|
||||
| **Total** | **~16 days (3 weeks calendar)** |
|
||||
|
||||
The Rust worker stages (4 + 5) are the majority of the schedule risk. If schedule is tight, an alternative is to ship Elixir-only first (Stages 1-3 + 6-10) and follow up with the Rust port — at the cost of slower Canadian forecast-chain processing during the gap.
|
||||
|
||||
---
|
||||
|
||||
## Open questions for confirmation before Stage 1
|
||||
|
||||
1. **Variable set:** Does the catalog at Task 1.2 cover everything the scorer needs for Canadian cells? Worth a 30-min look at `Microwaveprop.Propagation.Scorer.factors/2` to confirm the surface + pressure-level set is complete (especially native-level duct analysis — HRDPS's hybrid sigma-pressure levels may be available under a `*_HY_*` or `native/` directory that the probe didn't explore).
|
||||
2. **60°N cap:** Confirm Option 1 from Stage 8 (defer Arctic CDEM) before kickoff so we don't end up with the Stage 1.1 grid mask covering territory we won't actually serve.
|
||||
3. **Forecast horizon:** Default plan uses HRDPS f00..f18 to match HRRR's window. HRDPS goes out to f48 — would f00..f24 or even f00..f48 (full Day-2 forecast, only on the Canadian side) be useful? Storage/compute roughly doubles per extra 12h of horizon.
|
||||
165
lib/mix/tasks/hrdps_probe.ex
Normal file
165
lib/mix/tasks/hrdps_probe.ex
Normal file
|
|
@ -0,0 +1,165 @@
|
|||
defmodule Mix.Tasks.Hrdps.Probe do
|
||||
@shortdoc "Throwaway probe: fetch one HRDPS variable, decode at sample cities"
|
||||
@moduledoc """
|
||||
One-shot probe for the Canadian HRDPS model. Fetches a single GRIB2
|
||||
file from MSC Datamart and uses `wgrib2 -lon` to extract values at
|
||||
five Canadian cities, printing a table for visual sanity-checking.
|
||||
|
||||
Designed as a wedge to retire risk before committing to a real
|
||||
HrdpsClient: validates URL pattern, GRIB2 decode, and rotated-lat/lon
|
||||
reprojection without touching the DB, Oban, or the score grid.
|
||||
|
||||
Throwaway code. Will be deleted once the real client lands.
|
||||
|
||||
mix hrdps.probe # latest cycle, f000, 2m temperature
|
||||
mix hrdps.probe --hour 6 # f006 from latest cycle
|
||||
mix hrdps.probe --cycle 20260429T12Z
|
||||
|
||||
URL pattern (current as of 2026-04-29; ECCC reorganized away from the
|
||||
old `/model_hrdps/` root to a date-prefixed structure):
|
||||
|
||||
https://dd.weather.gc.ca/{YYYYMMDD}/WXO-DD/model_hrdps/continental/2.5km/
|
||||
{HH}/{FFF}/{YYYYMMDD}T{HH}Z_MSC_HRDPS_{VAR}_{LEVEL}_RLatLon0.0225_PT{FFF}H.grib2
|
||||
"""
|
||||
use Mix.Task
|
||||
|
||||
@datamart_base "https://dd.weather.gc.ca"
|
||||
|
||||
# Variable / level slug for 2m temperature in HRDPS naming. HRRR-side
|
||||
# equivalent is `TMP:2 m above ground`; HRDPS uses MSC's per-variable
|
||||
# filename convention.
|
||||
@variable_slug "TMP_AGL-2m"
|
||||
|
||||
@cities [
|
||||
{"YYZ Toronto", 43.68, -79.63},
|
||||
{"YVR Vancouver", 49.19, -123.18},
|
||||
{"YYC Calgary", 51.11, -114.02},
|
||||
{"YOW Ottawa", 45.32, -75.67},
|
||||
{"YHZ Halifax", 44.88, -63.51}
|
||||
]
|
||||
|
||||
@impl Mix.Task
|
||||
def run(argv) do
|
||||
{opts, _, _} =
|
||||
OptionParser.parse(argv,
|
||||
strict: [cycle: :string, hour: :integer],
|
||||
aliases: [c: :cycle, h: :hour]
|
||||
)
|
||||
|
||||
{:ok, _} = Application.ensure_all_started(:req)
|
||||
|
||||
forecast_hour = Keyword.get(opts, :hour, 0)
|
||||
cycle = opts |> Keyword.get(:cycle) |> resolve_cycle()
|
||||
|
||||
url = build_url(cycle, forecast_hour)
|
||||
IO.puts("Fetching: #{url}")
|
||||
|
||||
{:ok, path} = download(url)
|
||||
IO.puts("Downloaded #{path} (#{File.stat!(path).size} bytes)")
|
||||
|
||||
print_inventory(path)
|
||||
print_city_table(path, cycle, forecast_hour)
|
||||
end
|
||||
|
||||
# ----- Cycle resolution -----
|
||||
|
||||
defp resolve_cycle(nil), do: latest_published_cycle()
|
||||
|
||||
defp resolve_cycle(<<year::binary-size(4), month::binary-size(2), day::binary-size(2), "T", hour::binary-size(2), "Z">>) do
|
||||
%{date: "#{year}#{month}#{day}", hour: hour}
|
||||
end
|
||||
|
||||
defp resolve_cycle(other) do
|
||||
Mix.raise("Invalid --cycle #{inspect(other)} — expected YYYYMMDDTHHZ")
|
||||
end
|
||||
|
||||
# HRDPS publishes ~3-4h after each cycle (00/06/12/18Z). Walk back from
|
||||
# "now - 4h" until we find a cycle whose root directory exists.
|
||||
defp latest_published_cycle do
|
||||
now = DateTime.utc_now()
|
||||
candidates = for h <- 0..3, do: DateTime.add(now, -((4 + h * 6) * 3600), :second)
|
||||
|
||||
candidates
|
||||
|> Enum.map(&snap_to_cycle/1)
|
||||
|> Enum.uniq()
|
||||
|> Enum.find(&cycle_published?/1)
|
||||
|> case do
|
||||
nil -> Mix.raise("No published HRDPS cycle found in the last 24h")
|
||||
cycle -> cycle
|
||||
end
|
||||
end
|
||||
|
||||
defp snap_to_cycle(%DateTime{} = dt) do
|
||||
cycle_hour = div(dt.hour, 6) * 6
|
||||
%{date: Calendar.strftime(dt, "%Y%m%d"), hour: String.pad_leading("#{cycle_hour}", 2, "0")}
|
||||
end
|
||||
|
||||
defp cycle_published?(%{date: date, hour: hour}) do
|
||||
url = "#{@datamart_base}/#{date}/WXO-DD/model_hrdps/continental/2.5km/#{hour}/000/"
|
||||
|
||||
case Req.head(url, retry: false, connect_options: [timeout: 5_000]) do
|
||||
{:ok, %{status: 200}} -> true
|
||||
_ -> false
|
||||
end
|
||||
end
|
||||
|
||||
# ----- URL + download -----
|
||||
|
||||
defp build_url(%{date: date, hour: hour}, forecast_hour) do
|
||||
fff = String.pad_leading("#{forecast_hour}", 3, "0")
|
||||
filename = "#{date}T#{hour}Z_MSC_HRDPS_#{@variable_slug}_RLatLon0.0225_PT#{fff}H.grib2"
|
||||
|
||||
"#{@datamart_base}/#{date}/WXO-DD/model_hrdps/continental/2.5km/#{hour}/#{fff}/#{filename}"
|
||||
end
|
||||
|
||||
defp download(url) do
|
||||
path = Path.join(System.tmp_dir!(), "hrdps_probe_#{System.unique_integer([:positive])}.grib2")
|
||||
|
||||
case Req.get(url, into: File.stream!(path), retry: false) do
|
||||
{:ok, %{status: 200}} -> {:ok, path}
|
||||
{:ok, %{status: status}} -> Mix.raise("Datamart returned HTTP #{status} for #{url}")
|
||||
{:error, reason} -> Mix.raise("Fetch failed: #{inspect(reason)}")
|
||||
end
|
||||
end
|
||||
|
||||
# ----- wgrib2 inspection -----
|
||||
|
||||
defp print_inventory(path) do
|
||||
{output, 0} = System.cmd("wgrib2", [path], stderr_to_stdout: true)
|
||||
IO.puts("\nGRIB2 inventory:")
|
||||
IO.puts(" " <> String.trim(output))
|
||||
end
|
||||
|
||||
defp print_city_table(path, cycle, forecast_hour) do
|
||||
args = [path] ++ Enum.flat_map(@cities, fn {_, lat, lon} -> ["-lon", "#{lon}", "#{lat}"] end)
|
||||
{output, 0} = System.cmd("wgrib2", args, stderr_to_stdout: true)
|
||||
|
||||
values = parse_wgrib2_lon_output(output)
|
||||
|
||||
IO.puts("\nHRDPS 2m temperature, cycle #{cycle.date}T#{cycle.hour}Z f#{pad3(forecast_hour)}:\n")
|
||||
IO.puts(" city lat,lon temp")
|
||||
IO.puts(" -------------------- -------------------- ------")
|
||||
|
||||
@cities
|
||||
|> Enum.zip(values)
|
||||
|> Enum.each(fn {{name, lat, lon}, kelvin} ->
|
||||
celsius = kelvin - 273.15
|
||||
|
||||
IO.puts(" #{String.pad_trailing(name, 20)} #{format_coord(lat, lon)} #{format_temp(celsius)}")
|
||||
end)
|
||||
end
|
||||
|
||||
# wgrib2 -lon emits one line for the GRIB record with all `lon=…lat=…val=…`
|
||||
# entries colon-separated. For five cities we get five `val=` matches.
|
||||
defp parse_wgrib2_lon_output(output) do
|
||||
~r/val=([\-\d\.]+)/
|
||||
|> Regex.scan(output)
|
||||
|> Enum.map(fn [_, v] -> String.to_float(v) end)
|
||||
end
|
||||
|
||||
defp format_coord(lat, lon), do: "#{:io_lib.format("~6.2f", [lat])},#{:io_lib.format("~7.2f", [lon])}"
|
||||
|
||||
defp format_temp(celsius), do: "#{:io_lib.format("~5.1f", [celsius])}°C"
|
||||
|
||||
defp pad3(n), do: String.pad_leading("#{n}", 3, "0")
|
||||
end
|
||||
Loading…
Add table
Reference in a new issue