From 14e81f9251db93a78abea11202c6ac4156d0d43d Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Wed, 29 Apr 2026 16:47:15 -0500 Subject: [PATCH] docs(hrdps): plan + probe wedge for Canadian propagation grid MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds mix hrdps.probe — throwaway wedge that retires URL/projection/decode risks against MSC Datamart. Verified against five Canadian cities with plausible 2m temperatures via wgrib2's native rotated-lat/lon support. Findings update what we knew: - ECCC reorganized to date-prefixed paths; the old /model_hrdps/ root 404s now. Real URL is dd.weather.gc.ca/{YYYYMMDD}/WXO-DD/model_hrdps/... - HRDPS publishes one variable per GRIB2 file (~3.5 MB each), not HRRR's bundled wrfsfcf format. ~14 files per cycle/forecast hour. - wgrib2 -lon LON LAT handles the rotated grid natively — no manual reprojection math needed. New plan at 2026-04-29-hrdps-canadian-prop-grid.md supersedes the 2026-04-13 one (which assumed HRRR-style idx + byte-range fetches that don't apply). Probe deletion is a Stage 10 step in the new plan. --- .../2026-04-13-hrdps-canadian-prop-grid.md | 4 +- .../2026-04-29-hrdps-canadian-prop-grid.md | 382 ++++++++++++++++++ lib/mix/tasks/hrdps_probe.ex | 165 ++++++++ 3 files changed, 550 insertions(+), 1 deletion(-) create mode 100644 docs/plans/2026-04-29-hrdps-canadian-prop-grid.md create mode 100644 lib/mix/tasks/hrdps_probe.ex diff --git a/docs/plans/2026-04-13-hrdps-canadian-prop-grid.md b/docs/plans/2026-04-13-hrdps-canadian-prop-grid.md index 1a6d31ce..487629cf 100644 --- a/docs/plans/2026-04-13-hrdps-canadian-prop-grid.md +++ b/docs/plans/2026-04-13-hrdps-canadian-prop-grid.md @@ -1,4 +1,6 @@ -# HRDPS Canadian Propagation Grid Implementation Plan +# HRDPS Canadian Propagation Grid Implementation Plan (SUPERSEDED) + +> **SUPERSEDED 2026-04-29 by `docs/plans/2026-04-29-hrdps-canadian-prop-grid.md`.** A live probe of MSC Datamart on 2026-04-29 found that ECCC reorganized the URL structure (the `/model_hrdps/` flat root in the pre-flight steps below 404s now) and that HRDPS publishes one variable per GRIB2 file rather than HRRR's bundled wrfsfcf/wrfprsf format — so the Stage 1 idx + byte-range fetch shape this plan inherits from HrrrClient does not apply. Use the 2026-04-29 plan instead. > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. diff --git a/docs/plans/2026-04-29-hrdps-canadian-prop-grid.md b/docs/plans/2026-04-29-hrdps-canadian-prop-grid.md new file mode 100644 index 00000000..1e46fde0 --- /dev/null +++ b/docs/plans/2026-04-29-hrdps-canadian-prop-grid.md @@ -0,0 +1,382 @@ +# HRDPS Canadian Propagation Grid Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Supersedes:** `docs/plans/2026-04-13-hrdps-canadian-prop-grid.md` — that plan was written before the probe wedge surfaced two architectural realities that change Stage 1's design: ECCC moved the datamart URL structure (the old `/model_hrdps/` root 404s now), and HRDPS publishes one variable per GRIB2 file rather than HRRR's bundled wrfsfcf/wrfprsf format. Major filenames, fetch shape, and idx assumptions are different. + +**Goal:** Extend the propagation scoring grid into Canada at 0.125° (matching the existing CONUS sample density) sourced from HRDPS, so `/map` shows scores across all of North America. Hourly Canadian scores via the cycle's f00..f18 forecast hours; HRRR wins where it has coverage and HRDPS fills the rest. + +**Architecture:** Build a parallel NWP ingestion pipeline (`HrdpsClient` → `HrdpsGridWorker` → `hrdps_profiles` table) that mirrors the HRRR shape. The single-variable-per-file difference is hidden inside `HrdpsClient`: it fetches the ~14 needed variable files concurrently, then runs `wgrib2 -merge` to produce a single multi-variable GRIB2 that downstream code can consume identically to a `wrfsfcf`/`wrfprsf` blob. HRDPS runs at 00/06/12/18Z and seeds f00..f18 (hourly) into the score chain, so Canadian scores refresh once per cycle but provide hourly forecast steps until the next cycle lands. Border stitching: a static `Grid.hrdps_only_points/0` mask carves Canada-only cells so HRRR and HRDPS never both write the same `(lat, lon)`. + +**Tech Stack:** Elixir 1.15, Req, Postgres + Ecto, Oban, wgrib2 (already in image), Rust (for `prop-grid-rs` and `hrrr-point-rs` HRDPS branches) + +**Prerequisite:** Stage 0 already done — the probe at `lib/mix/tasks/hrdps_probe.ex` retired the URL/projection/decode risks against the live datamart on 2026-04-29. + +--- + +## Stage 0: Probe wedge (DONE) + +`mix hrdps.probe` proves the GRIB2 fetch + wgrib2 reprojection works end-to-end against the real ECCC datamart. Output verifies plausible Canadian temperatures at 5 cities. Throwaway code; will be deleted in Stage 10 once `HrdpsClient` covers the same surface area. + +Findings that drive the rest of the plan: + +- **URL:** `https://dd.weather.gc.ca/{YYYYMMDD}/WXO-DD/model_hrdps/continental/2.5km/{HH}/{FFF}/{YYYYMMDD}T{HH}Z_MSC_HRDPS_{VAR}_{LEVEL}_RLatLon0.0225_PT{FFF}H.grib2` +- **Variable naming:** MSC's `{VAR}_{LEVEL_TYPE}-{LEVEL_VALUE}` (e.g. `TMP_AGL-2m`, `TMP_ISBL_0850`) — different from NCEP's HRRR short-name format +- **One variable per file** (~3.5 MB each); no idx file / no byte-range partials +- **wgrib2 reprojects rotated lat/lon natively** via `-lon LON LAT` — no manual rotation math +- **Publish latency:** ~3-4h after each cycle (verified by cycle availability at probe time) + +--- + +## Stage 1: `HrdpsClient` — per-variable fetch + wgrib2 -merge + +**Decision recorded:** Fetch each variable's GRIB2 file concurrently, then run `wgrib2 -merge` to combine into a single multi-variable file before downstream decode. Keeps the existing `hrrr_profiles`-shaped derive pipeline intact downstream of the client. The disk-staging cost (~14 files × 3.5 MB into a tmp dir per cycle/forecast hour) is small compared to the fetch time. + +**Files:** +- Create: `lib/microwaveprop/weather/hrdps_client.ex` +- Create: `test/microwaveprop/weather/hrdps_client_test.exs` +- Create: `test/support/fixtures/hrdps_sample_*.grib2` (small fixtures captured from a real cycle) + +### Task 1.1: URL builder (TDD) + +Write a failing test for `HrdpsClient.grib_url(:tmp_2m, run_time, forecast_hour)` returning the date-prefixed datamart URL with the MSC filename format. Verify against a known-good URL from the probe (`20260429T12Z_MSC_HRDPS_TMP_AGL-2m_RLatLon0.0225_PT000H.grib2`). Implement; commit. + +### Task 1.2: Variable catalog + +Define a `@variables` map mapping internal atoms (`:tmp_2m`, `:dpt_2m`, `:pres_sfc`, `:wind_u_10m`, `:wind_v_10m`, `:apcp_1h`, `:tcdc`, `:pwat`, `:tmp_850mb`, `:tmp_700mb`, `:tmp_500mb`, `:dpt_850mb`, `:dpt_700mb`, `:hgt_850mb`) to MSC variable + level slugs (`{VAR_AGL-2m}`, `{VAR_ISBL_0850}`, etc.). + +This is the variable-mapping table the old plan called out as a Stage-1 risk; build it explicitly. TDD: a `variable_slug/1` test for each atom. Commit. + +### Task 1.3: Concurrent fetch + tmp staging (TDD) + +Write a failing test for `fetch_cycle_files(run_time, forecast_hour, opts)` that returns `{:ok, [paths]}` of the per-variable GRIB2 files staged in a tmp dir. Use `Task.async_stream` with `max_concurrency: 4` (datamart isn't aggressively rate-limited but we don't need >4 in flight). Mock with `Req.Test.stub` per project conventions. + +Critical: every `{:exit, _}` and `{:error, _}` clause from the async_stream must `Logger.error` with variable name + URL — per CLAUDE.md "always log anything that would normally be swallowed" and the path-calculator outage memory. A silently-dropped fetch in production is a Canadian dead-zone we won't notice for hours. + +Implement; commit. + +### Task 1.4: wgrib2 -merge into multi-variable file (TDD) + +Write a failing test for `merge_to_multivar(paths, output_path)`. Implement via `System.cmd("wgrib2", ["-merge", path1, path2, ..., "-grib", output_path])`. Verify the merged output's `wgrib2 file -V` lists all expected variables. + +Commit. + +### Task 1.5: `fetch_grid/3` — public API matching `HrrrClient.fetch_grid/3` + +Same signature as HRRR so Stage 3's worker can call either polymorphically: + +```elixir +@spec fetch_grid([{float(), float()}], DateTime.t(), keyword()) :: + {:ok, %{{float(), float()} => map()}} | {:error, term()} +def fetch_grid(points, run_time, opts \\ []) +``` + +Internally: `fetch_cycle_files` → `merge_to_multivar` → reuse `Microwaveprop.Weather.GribDecoder` (the existing HRRR decoder, projection-agnostic since wgrib2 handles rotation). + +TDD; commit. + +### Task 1.6: Projection sanity check (TDD) + +Pick CYYR (Goose Bay, 53.32°N, -60.42°W) and a recent cycle for which we have a UWYO Canadian sounding (those exist via `CanadianSoundingFetchWorker`). Test that the HRDPS surface temp at CYYR is within ±3°C of the radiosonde surface temp for the same `valid_time`. If they don't match, the projection is wrong and `wgrib2 -lon` is being called incorrectly. + +This is the most important test in the stage — burn whatever time is needed to make it pass. A 10 km projection offset is invisible on the map but destroys terrain analysis at Stage 5+. + +Commit. + +--- + +## Stage 2: `hrdps_profiles` schema + +**Files:** +- Create: `priv/repo/migrations/{timestamp}_create_hrdps_profiles.exs` +- Create: `lib/microwaveprop/weather/hrdps_profile.ex` +- Create: `test/microwaveprop/weather/hrdps_profile_test.exs` + +### Task 2.1: Migration mirrors `hrrr_profiles` + +Same columns, same indexes, daily partitioning on `valid_time`. Separate table (not a unified `nwp_profiles`) so HRDPS can be dropped/reprocessed without disturbing the 4500+ HRRR profiles already in prod. + +`@primary_key {:id, :binary_id, autogenerate: true}` per project convention. + +Run migration in dev; verify table + indexes; commit. + +### Task 2.2: `HrdpsProfile` schema + changeset (TDD) + +Identical to `HrrrProfile`. Same derived fields (`surface_refractivity`, `min_refractivity_gradient`, `ducting_detected`, `duct_characteristics`, `pwat_mm`, `hpbl_m`). + +Commit. + +### Task 2.3: Partition pruning cron + +Add an `Oban.Plugins.Cron` entry mirroring whatever HRRR uses (or add one if HRRR doesn't have one yet — the prod DB will fill up either way). + +Commit. + +--- + +## Stage 3: `HrdpsGridWorker` — score Canadian grid per forecast hour + +**Files:** +- Create: `lib/microwaveprop/workers/hrdps_grid_worker.ex` +- Create: `test/microwaveprop/workers/hrdps_grid_worker_test.exs` +- Modify: `lib/microwaveprop/propagation/grid.ex` — add `hrdps_only_points/0` + +### Task 3.1: Define the Canadian-only grid (TDD) + +`hrdps_only_points/0` returns the subset of a 0.125°-spaced grid that is INSIDE the Canadian land mask AND OUTSIDE the existing CONUS HRRR footprint. Implement as a precomputed list (compile-time `@external_resource`) — the cells don't change. + +Approximate bounds: 42°N to 60°N (north-of-60°N deferred to Stage 8), -141°W to -52°W. Filter to land using a coarse polygon — Natural Earth's `ne_50m_admin_0_countries` clipped to Canada is fine; even a rectangular bounding-box pass-through is acceptable for a first pass since over-water cells just get scored and ignored. + +The mask must NEVER overlap HRRR cells. Test: `MapSet.disjoint?(MapSet.new(Grid.conus_points()), MapSet.new(Grid.hrdps_only_points()))`. + +Commit. + +### Task 3.2: Worker `perform/1` (TDD) + +Mirror `PropagationGridWorker.process_forecast_hour/4`. Args: `{run_time, forecast_hour}`. Calls `HrdpsClient.fetch_grid(hrdps_only_points(), run_time, opts)`, derives profiles via the existing `Microwaveprop.Weather.SoundingParams`-style derivation, persists to `hrdps_profiles`, then scores via `Microwaveprop.Propagation.Scorer.score/3`. + +`start_async`/`Task.async_stream` failure paths must `Logger.error` with point + reason (per CLAUDE.md and the swallowed-async-errors memory). + +Commit. + +### Task 3.3: Score upserts into existing `propagation_scores` + +Single shared scores table — HRDPS rows look identical to HRRR rows, distinguished only by `(lat, lon)` falling in the HRDPS-only mask. Use the same `on_conflict: {:replace_all_except, [...]}` upsert pattern. + +Confirm via SQL: a sample run produces no `(lat, lon, valid_time, band_mhz)` collisions with HRRR rows. + +Commit. + +### Task 3.4: Score-file `.prop` writer + +The `.prop` file at `/data/scores/...` is what `MicrowavepropWeb.ScoresController.cells/2` reads (and what `ScoreCache` mirrors in ETS). Either: + +(a) extend the `.prop` writer to handle the larger union grid (CONUS + Canada), or +(b) emit a separate `_canada.prop` file that the controller stitches at read time. + +Decision: **(a)** — single file, simpler downstream. The Rust scorer (Stage 4) writes the same file; both must agree on the cell layout. Update `Microwaveprop.Propagation.ScoresFile` to support a wider grid extent. The cell-pack format already handles arbitrary lat/lon arrays per the `PSCR` magic spec. + +TDD: write a fixture `.prop` file at the expanded extent, read it back, verify all cells round-trip. Commit. + +--- + +## Stage 4: `prop-grid-rs` Rust worker — HRDPS branch + +The Rust worker at `rust/prop_grid_rs/` claims tasks from `grid_tasks` via `FOR UPDATE SKIP LOCKED` and writes `.prop` files to `/data/scores`. It needs an HRDPS branch since it owns the hot path for f01..f18. + +**Files:** +- Modify: `rust/prop_grid_rs/src/sources/` (likely add `hrdps.rs` next to `hrrr.rs`) +- Modify: `rust/prop_grid_rs/src/main.rs` — task source dispatch +- Migration: `priv/repo/migrations/{timestamp}_add_source_to_grid_tasks.exs` + +### Task 4.1: Add `source` column to `grid_tasks` + +`source TEXT NOT NULL DEFAULT 'hrrr'`. Backfill is a no-op since existing rows are all HRRR. Index on `(source, run_time)` for the worker's claim query. + +Commit. + +### Task 4.2: Rust HRDPS fetcher + +Mirror the existing `hrrr.rs` shape. Differences: + +- URL builder using the date-prefixed pattern + per-variable filenames +- Concurrent fetch of ~14 variable files (use `tokio::join_all` with a semaphore at 4) +- Shell out to `wgrib2 -merge` (same binary the Elixir client uses; no Rust GRIB2 reimplementation) +- Reuse the existing point-extraction code that wraps `wgrib2 -lon` + +Keep the Rust HRDPS code as close to the Rust HRRR code as possible — copy-then-modify is fine; resist refactoring for shared abstraction until both branches are working. + +Commit. + +### Task 4.3: Task dispatch on `source` column + +Worker's main loop branches on `task.source`: `"hrrr"` → existing path, `"hrdps"` → new path. The task seeder (`Microwaveprop.Workers.GridTaskEnqueuer`) populates `source` based on which worker enqueued the task. + +Commit. + +### Task 4.4: Build + push image + +Existing CI pipeline (`.forgejo/workflows/build-grid-rs.yaml`) produces the `prop-grid-rs:main-*` image consumed by both `prop-grid-rs` and `hrrr-point-rs` deployments. New code rolls out automatically once merged. Verify in Argo / pod logs. + +--- + +## Stage 5: `hrrr-point-rs` — HRDPS branch for per-QSO Canadian QSOs + +**Files:** +- Modify: `rust/prop_grid_rs/src/bin/hrrr_point_worker.rs` +- Migration: `priv/repo/migrations/{timestamp}_add_source_to_hrrr_fetch_tasks.exs` +- Modify: `lib/microwaveprop/workers/hrrr_fetch_enqueue_worker.ex` (or wherever per-QSO fetches are enqueued) + +### Task 5.1: Routing decision + +When a QSO's lat/lon falls outside the HRRR CONUS footprint, enqueue an HRDPS-source task instead of an HRRR task. `Grid.point_source({lat, lon})` returns `:hrrr | :hrdps | :neither` (e.g. for a southern-hemisphere QSO). For `:neither`, fall back to the existing IEMRE/RAOB enrichment path. + +TDD: cells at (45.0, -90.0) → `:hrrr`, (53.0, -113.0) → `:hrdps`, (-30.0, 150.0) → `:neither`. + +Commit. + +### Task 5.2: `hrrr_fetch_tasks.source` column + Rust branch + +Same shape as Stage 4.1/4.3: add `source` column, branch the Rust worker, reuse Stage 1's URL/variable patterns ported to Rust. + +Commit. + +### Task 5.3: Per-QSO upserts into `hrdps_profiles` + +QSO enrichment status column: add `hrdps_status` to `contacts` (similar to existing `hrrr_status`). LiveView/admin UI reads either, depending on which model covered the QSO. + +Commit. + +--- + +## Stage 6: Cycle scheduling + border stitching + +**Files:** +- Modify: `config/runtime.exs` — Oban cron + queue +- Modify: `lib/microwaveprop/workers/grid_task_enqueuer.ex` (or equivalent) + +### Task 6.1: Add `hrdps` queue + +Production runtime config adds `hrdps: 1` (single concurrent fetch — wgrib2 working set during merge can hit ~500 MB and we don't want it competing with `propagation`). + +### Task 6.2: Cycle cron at HH:35 of cycle hours + +```elixir +{"35 5,11,17,23 * * *", Microwaveprop.Workers.HrdpsCycleSeedWorker} +``` + +Why these times: HRDPS runs at 00/06/12/18Z and publishes ~3-4h later, so 5h after cycle (at 05:35Z, 11:35Z, 17:35Z, 23:35Z) the f00 file is reliably available. Offset by 30 minutes from HRRR's `*/15` schedule to avoid wgrib2 contention. + +`HrdpsCycleSeedWorker` enqueues f00..f18 grid tasks with `source: "hrdps"`. The existing chain worker (Rust + Elixir) drains them. + +### Task 6.3: Border stitching rule + +Already enforced at Stage 3.1: HRDPS-only mask cells are disjoint from HRRR's CONUS points. No runtime conflict. The seam at 49°N is implicit: HRRR cells exist down to ~25°N, HRDPS cells exist 49°N+. The boundary is a one-cell band of HRRR coverage abutting HRDPS — no overlap, no double-write. + +Document the rule in `Grid.point_source/1`'s `@moduledoc` so future readers understand why the masks are exclusive. + +Commit. + +--- + +## Stage 7: Map overlay extension + +**Files:** +- Modify: `lib/microwaveprop_web/live/map_live.ex` — `@initial_bounds` +- Modify: `assets/js/propagation_map_hook.ts` — heatmap rendering bounds + +### Task 7.1: Widen `@initial_bounds` + +CONUS-only `(23.5, 49.5, -124.5, -66.9)` → `(23.5, 60.0, -141.0, -52.0)` (excluding Arctic until Stage 8 decides). Default zoom adjusted so all of North America is visible. + +### Task 7.2: Smoke test (manual + LiveView test) + +`mix phx.server`, open `/map`, confirm colored cells render north of 49°N after `HrdpsCycleSeedWorker` fires. The cell-pack endpoint at `/scores/cells` already serves whatever lat/lon points exist in the `.prop` file — no controller changes needed since Stage 3.4 expanded the file. + +LiveView test: assert that an `/scores/cells?band=10000&south=50&north=55&west=-115&east=-105` request returns >0 cells after HRDPS data exists in fixtures. + +### Task 7.3: ScoreCache chunk layout still correct + +The ScoreCache layout (just shipped in commit 6a90d153 follow-up) uses 5°×5° chunks. Extending the grid into Canada adds ~30 new chunks but the same lookup logic applies. No code change; one regression test that fetches a Canadian-bounds viewport and asserts hit/miss telemetry works correctly. + +Commit. + +--- + +## Stage 8: SRTM gap above 60°N — scope decision + +**Decision gate.** SRTM1 stops at 60°N. Yellowknife (62.45°N), Whitehorse (60.72°N), Inuvik (68.36°N), and Resolute (74.70°N) have NO SRTM coverage — terrain diffraction analysis would silently return wrong losses for paths involving those stations. + +Three options: + +1. **Cap Canadian coverage at 60°N for v1.** Document explicitly in `algo.md`. Most VE QSOs are below 60°N anyway. Defer Arctic to a follow-up plan. +2. **Use CDEM (Canadian Digital Elevation Model) above 60°N.** ~1 week of additional work — different format (TIFF tiles), different mounting strategy on the production NFS share. Touches `terrain_analysis.ex` and `elevation_client.ex`. +3. **Use ASTER GDEM v3.** Global, free, 30 m resolution, but registration-walled. Not great. + +**Recommendation: Option 1 for v1.** Ship the 42-60°N coverage. If users report Arctic gaps, do CDEM as a separate plan. + +Decision recorded in this stage; file a follow-up issue if Option 2 is pursued. + +--- + +## Stage 9: Freshness monitoring + observability + +**Files:** +- Modify: `lib/microwaveprop/propagation/freshness_monitor.ex` +- Modify: wherever `/status` exposes data freshness (LiveView or controller) + +### Task 9.1: Track HRDPS run-time staleness + +Add `hrdps_last_run_at` to the monitor state. Surface on the status page next to HRRR's freshness. + +### Task 9.2: Alert at >5h stale + +HRDPS publishes ~4h after each cycle, so a healthy pipeline never exceeds ~5h between runs. >5h = something broken (datamart outage, our worker failed silently, k8s scheduling glitch). + +`Logger.warning` (not `error` — datamart outages are common and recoverable). Telemetry event for the existing dashboard scrape. + +Commit. + +--- + +## Stage 10: Precommit, docs, ship + +### Task 10.1: `mix precommit` + +Must show 0 failures, 0 credo issues, all dialyzer warnings resolved. Full project gate. + +### Task 10.2: Update `CLAUDE.md` + +- Add HRDPS to **Key Data Sources** +- Add `HrdpsCycleSeedWorker` + `HrdpsGridWorker` + `HrdpsFetchWorker` to the **Background Job Queues** table +- Note the 60°N coverage cap and the cycle cadence (4×/day vs HRRR's hourly) + +### Task 10.3: Update `memory/reference_hrdps_canada.md` + +Mark plan as executed. Capture any projection gotchas, the final cycle latency observed in prod, and the actual border-stitching seam location. + +### Task 10.4: Delete the probe wedge + +`lib/mix/tasks/hrdps_probe.ex` is throwaway scaffolding — its risks were retired in Stage 0; the production code in `HrdpsClient` covers the same surface. Delete in this stage's commit, not earlier (keeping it around during stages 1-3 is useful for cross-checking the real client's output). + +### Task 10.5: Push to `github` AND `origin` + +Per the push-both-remotes feedback memory. Do NOT push to `dokku`. Argo/Flux handles the rollout from the kustomize manifest pulled from `git.mcintire.me`. + +--- + +## Known risks + gotchas + +- **Projection** — HRDPS rotated lat/lon is the single biggest bug source. Stage 1.6's CYYR validation against UWYO is the safety net; do not skip. +- **Per-variable file fan-out** — ~14 fetches per cycle/forecast hour × 4 cycles × 19 forecast hours = 1064 files/day. Each is small (~3.5 MB) but datamart's `200 OK` latency is variable. Concurrency-cap at 4 to avoid hammering ECCC; build retry-with-backoff into `HrdpsClient`. +- **Memory pressure** — wgrib2 `-merge` working set during the multi-variable combine peaks at ~500 MB per cycle. The HRRR worker already runs at `:system_memory_high_watermark` headroom; running HRDPS on the same pod risks OOM. Either pin HRDPS to a dedicated `prop-backfill`-style pod (probably overkill) OR keep it on the hot pods but cap `hrdps: 1` so it never overlaps with `propagation: 1`. +- **Cycle publish latency** — datamart can lag 90 min on good days, 4+h under load. The `HrdpsCycleSeedWorker` cron at HH:35 has a 1.5h margin which is comfortable but not infinite. If a cycle hasn't published by HH:35, the worker should fall back to seeding the previous cycle (still gives current+forecast scores, just one cycle stale). +- **Border seam visibility** — at the 49°N seam, HRRR's 0.125° cells abut HRDPS's 0.125° cells with no overlap. If the two models disagree by 5-10% on a given factor at the seam, users will see a faint horizontal stripe. Probably acceptable; document in `algo.md`. +- **Scope creep** — keep RDPS pressure-level coverage out of this plan even though they're related (RDPS plan is `2026-04-13-rdps-vertical-profiles.md`). Ship CONUS+Canada-via-HRDPS first; revisit RDPS for wider/coarser coverage afterward. + +## Rough effort estimate + +| Stage | Estimate | +|---|---| +| 0: Probe wedge | DONE (~3h, 2026-04-29) | +| 1: HrdpsClient | 4 days (Task 1.6 projection debug dominates) | +| 2: hrdps_profiles schema | 0.5 day | +| 3: HrdpsGridWorker + .prop writer | 3 days | +| 4: Rust prop-grid-rs HRDPS branch | 4 days | +| 5: Rust hrrr-point-rs HRDPS branch | 1.5 days | +| 6: Cycle scheduling + border stitching | 1 day | +| 7: Map overlay extension | 0.5 day | +| 8: SRTM scope decision | 0.5 day (decision + documentation, no code if Option 1) | +| 9: Freshness monitoring | 0.5 day | +| 10: Precommit + docs + ship | 0.5 day | +| **Total** | **~16 days (3 weeks calendar)** | + +The Rust worker stages (4 + 5) are the majority of the schedule risk. If schedule is tight, an alternative is to ship Elixir-only first (Stages 1-3 + 6-10) and follow up with the Rust port — at the cost of slower Canadian forecast-chain processing during the gap. + +--- + +## Open questions for confirmation before Stage 1 + +1. **Variable set:** Does the catalog at Task 1.2 cover everything the scorer needs for Canadian cells? Worth a 30-min look at `Microwaveprop.Propagation.Scorer.factors/2` to confirm the surface + pressure-level set is complete (especially native-level duct analysis — HRDPS's hybrid sigma-pressure levels may be available under a `*_HY_*` or `native/` directory that the probe didn't explore). +2. **60°N cap:** Confirm Option 1 from Stage 8 (defer Arctic CDEM) before kickoff so we don't end up with the Stage 1.1 grid mask covering territory we won't actually serve. +3. **Forecast horizon:** Default plan uses HRDPS f00..f18 to match HRRR's window. HRDPS goes out to f48 — would f00..f24 or even f00..f48 (full Day-2 forecast, only on the Canadian side) be useful? Storage/compute roughly doubles per extra 12h of horizon. diff --git a/lib/mix/tasks/hrdps_probe.ex b/lib/mix/tasks/hrdps_probe.ex new file mode 100644 index 00000000..4bf96ac1 --- /dev/null +++ b/lib/mix/tasks/hrdps_probe.ex @@ -0,0 +1,165 @@ +defmodule Mix.Tasks.Hrdps.Probe do + @shortdoc "Throwaway probe: fetch one HRDPS variable, decode at sample cities" + @moduledoc """ + One-shot probe for the Canadian HRDPS model. Fetches a single GRIB2 + file from MSC Datamart and uses `wgrib2 -lon` to extract values at + five Canadian cities, printing a table for visual sanity-checking. + + Designed as a wedge to retire risk before committing to a real + HrdpsClient: validates URL pattern, GRIB2 decode, and rotated-lat/lon + reprojection without touching the DB, Oban, or the score grid. + + Throwaway code. Will be deleted once the real client lands. + + mix hrdps.probe # latest cycle, f000, 2m temperature + mix hrdps.probe --hour 6 # f006 from latest cycle + mix hrdps.probe --cycle 20260429T12Z + + URL pattern (current as of 2026-04-29; ECCC reorganized away from the + old `/model_hrdps/` root to a date-prefixed structure): + + https://dd.weather.gc.ca/{YYYYMMDD}/WXO-DD/model_hrdps/continental/2.5km/ + {HH}/{FFF}/{YYYYMMDD}T{HH}Z_MSC_HRDPS_{VAR}_{LEVEL}_RLatLon0.0225_PT{FFF}H.grib2 + """ + use Mix.Task + + @datamart_base "https://dd.weather.gc.ca" + + # Variable / level slug for 2m temperature in HRDPS naming. HRRR-side + # equivalent is `TMP:2 m above ground`; HRDPS uses MSC's per-variable + # filename convention. + @variable_slug "TMP_AGL-2m" + + @cities [ + {"YYZ Toronto", 43.68, -79.63}, + {"YVR Vancouver", 49.19, -123.18}, + {"YYC Calgary", 51.11, -114.02}, + {"YOW Ottawa", 45.32, -75.67}, + {"YHZ Halifax", 44.88, -63.51} + ] + + @impl Mix.Task + def run(argv) do + {opts, _, _} = + OptionParser.parse(argv, + strict: [cycle: :string, hour: :integer], + aliases: [c: :cycle, h: :hour] + ) + + {:ok, _} = Application.ensure_all_started(:req) + + forecast_hour = Keyword.get(opts, :hour, 0) + cycle = opts |> Keyword.get(:cycle) |> resolve_cycle() + + url = build_url(cycle, forecast_hour) + IO.puts("Fetching: #{url}") + + {:ok, path} = download(url) + IO.puts("Downloaded #{path} (#{File.stat!(path).size} bytes)") + + print_inventory(path) + print_city_table(path, cycle, forecast_hour) + end + + # ----- Cycle resolution ----- + + defp resolve_cycle(nil), do: latest_published_cycle() + + defp resolve_cycle(<>) do + %{date: "#{year}#{month}#{day}", hour: hour} + end + + defp resolve_cycle(other) do + Mix.raise("Invalid --cycle #{inspect(other)} — expected YYYYMMDDTHHZ") + end + + # HRDPS publishes ~3-4h after each cycle (00/06/12/18Z). Walk back from + # "now - 4h" until we find a cycle whose root directory exists. + defp latest_published_cycle do + now = DateTime.utc_now() + candidates = for h <- 0..3, do: DateTime.add(now, -((4 + h * 6) * 3600), :second) + + candidates + |> Enum.map(&snap_to_cycle/1) + |> Enum.uniq() + |> Enum.find(&cycle_published?/1) + |> case do + nil -> Mix.raise("No published HRDPS cycle found in the last 24h") + cycle -> cycle + end + end + + defp snap_to_cycle(%DateTime{} = dt) do + cycle_hour = div(dt.hour, 6) * 6 + %{date: Calendar.strftime(dt, "%Y%m%d"), hour: String.pad_leading("#{cycle_hour}", 2, "0")} + end + + defp cycle_published?(%{date: date, hour: hour}) do + url = "#{@datamart_base}/#{date}/WXO-DD/model_hrdps/continental/2.5km/#{hour}/000/" + + case Req.head(url, retry: false, connect_options: [timeout: 5_000]) do + {:ok, %{status: 200}} -> true + _ -> false + end + end + + # ----- URL + download ----- + + defp build_url(%{date: date, hour: hour}, forecast_hour) do + fff = String.pad_leading("#{forecast_hour}", 3, "0") + filename = "#{date}T#{hour}Z_MSC_HRDPS_#{@variable_slug}_RLatLon0.0225_PT#{fff}H.grib2" + + "#{@datamart_base}/#{date}/WXO-DD/model_hrdps/continental/2.5km/#{hour}/#{fff}/#{filename}" + end + + defp download(url) do + path = Path.join(System.tmp_dir!(), "hrdps_probe_#{System.unique_integer([:positive])}.grib2") + + case Req.get(url, into: File.stream!(path), retry: false) do + {:ok, %{status: 200}} -> {:ok, path} + {:ok, %{status: status}} -> Mix.raise("Datamart returned HTTP #{status} for #{url}") + {:error, reason} -> Mix.raise("Fetch failed: #{inspect(reason)}") + end + end + + # ----- wgrib2 inspection ----- + + defp print_inventory(path) do + {output, 0} = System.cmd("wgrib2", [path], stderr_to_stdout: true) + IO.puts("\nGRIB2 inventory:") + IO.puts(" " <> String.trim(output)) + end + + defp print_city_table(path, cycle, forecast_hour) do + args = [path] ++ Enum.flat_map(@cities, fn {_, lat, lon} -> ["-lon", "#{lon}", "#{lat}"] end) + {output, 0} = System.cmd("wgrib2", args, stderr_to_stdout: true) + + values = parse_wgrib2_lon_output(output) + + IO.puts("\nHRDPS 2m temperature, cycle #{cycle.date}T#{cycle.hour}Z f#{pad3(forecast_hour)}:\n") + IO.puts(" city lat,lon temp") + IO.puts(" -------------------- -------------------- ------") + + @cities + |> Enum.zip(values) + |> Enum.each(fn {{name, lat, lon}, kelvin} -> + celsius = kelvin - 273.15 + + IO.puts(" #{String.pad_trailing(name, 20)} #{format_coord(lat, lon)} #{format_temp(celsius)}") + end) + end + + # wgrib2 -lon emits one line for the GRIB record with all `lon=…lat=…val=…` + # entries colon-separated. For five cities we get five `val=` matches. + defp parse_wgrib2_lon_output(output) do + ~r/val=([\-\d\.]+)/ + |> Regex.scan(output) + |> Enum.map(fn [_, v] -> String.to_float(v) end) + end + + defp format_coord(lat, lon), do: "#{:io_lib.format("~6.2f", [lat])},#{:io_lib.format("~7.2f", [lon])}" + + defp format_temp(celsius), do: "#{:io_lib.format("~5.1f", [celsius])}°C" + + defp pad3(n), do: String.pad_leading("#{n}", 3, "0") +end