Graham McIntire
14e81f9251
Adds mix hrdps.probe — throwaway wedge that retires URL/projection/decode
risks against MSC Datamart. Verified against five Canadian cities with
plausible 2m temperatures via wgrib2's native rotated-lat/lon support.
Findings update what we knew:
- ECCC reorganized to date-prefixed paths; the old /model_hrdps/ root
404s now. Real URL is dd.weather.gc.ca/{YYYYMMDD}/WXO-DD/model_hrdps/...
- HRDPS publishes one variable per GRIB2 file (~3.5 MB each), not
HRRR's bundled wrfsfcf format. ~14 files per cycle/forecast hour.
- wgrib2 -lon LON LAT handles the rotated grid natively — no manual
reprojection math needed.
New plan at 2026-04-29-hrdps-canadian-prop-grid.md supersedes the
2026-04-13 one (which assumed HRRR-style idx + byte-range fetches that
don't apply). Probe deletion is a Stage 10 step in the new plan.
313 lines
14 KiB
Markdown
313 lines
14 KiB
Markdown
# HRDPS Canadian Propagation Grid Implementation Plan (SUPERSEDED)
|
||
|
||
> **SUPERSEDED 2026-04-29 by `docs/plans/2026-04-29-hrdps-canadian-prop-grid.md`.** A live probe of MSC Datamart on 2026-04-29 found that ECCC reorganized the URL structure (the `/model_hrdps/` flat root in the pre-flight steps below 404s now) and that HRDPS publishes one variable per GRIB2 file rather than HRRR's bundled wrfsfcf/wrfprsf format — so the Stage 1 idx + byte-range fetch shape this plan inherits from HrrrClient does not apply. Use the 2026-04-29 plan instead.
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** Extend the propagation scoring grid to cover all of Canada at HRDPS's 2.5 km resolution — the Canadian analog to HRRR. This lights up the `/map` for VE stations, gives terrain-diffraction analysis for VE QSOs, and unlocks cross-border path analysis for KP/W7-to-VE7 tropo paths.
|
||
|
||
**Architecture:** Build a parallel NWP ingestion pipeline (`HrdpsClient` + `HrdpsGridWorker` + `hrdps_profiles` table) that matches the HRRR structure one-for-one, with three major differences: rotated lat/lon projection, CMC file naming (`CMC_hrdps_continental_*_ps2.5km_*_{FFF}.grib2`), and a different native-level grid spec for duct analysis. A new `propagation_scores` region tag distinguishes HRRR-sourced vs HRDPS-sourced cells so the front-end can render them uniformly.
|
||
|
||
**Tech Stack:** Elixir 1.15, Req, Nx (optional for duct analysis), Postgres, Oban, wgrib2, proj (for projection conversions if needed)
|
||
|
||
**Prerequisite:** Task 6 of the RDPS plan (`2026-04-13-rdps-vertical-profiles.md`) — or at minimum a working pattern for "score grid_data from a non-HRRR source". If you're starting HRDPS fresh, Task 6 below stands alone but will duplicate some work from the RDPS plan.
|
||
|
||
---
|
||
|
||
## Pre-flight research (1–2 hours)
|
||
|
||
**Step R.1: Confirm HRDPS datamart URL pattern**
|
||
|
||
```bash
|
||
curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/ 2>&1 | head -30
|
||
curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/00/000/ 2>&1 | head -30
|
||
```
|
||
|
||
Expected: file list with entries like `CMC_hrdps_continental_TMP_TGL_2_ps2.5km_{YYYYMMDDHH}_P{FFF}-00.grib2`.
|
||
|
||
**Step R.2: Download one sample + inspect with wgrib2**
|
||
|
||
```bash
|
||
wgrib2 CMC_hrdps_continental_TMP_TGL_2_ps2.5km_2026041300_P000-00.grib2 -grid -v
|
||
```
|
||
|
||
Expected: confirms grid is rotated lat/lon, gives dimensions (typically ~2500×1800), and lists all variables.
|
||
|
||
**Step R.3: Confirm pressure-level files exist**
|
||
|
||
HRDPS publishes surface + pressure-level files separately (like HRRR wrfsfc vs wrfprs). Find the pressure-level variant:
|
||
|
||
```bash
|
||
curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/00/000/ | grep -i 'ISBL'
|
||
```
|
||
|
||
If separate files per pressure level, the ingestion must fan out like `HrrrClient.pressure_messages/0`. If a single combined file, one download gets everything.
|
||
|
||
**Step R.4: Decide projection strategy**
|
||
|
||
wgrib2 can extract points from rotated lat/lon directly with `-lon LON LAT`. For extracting the full grid as a flat array, you need `proj` or a wgrib2 `-new_grid` reprojection. **Recommended**: never reproject — always extract point-by-point for the grid points we care about. Slower than bulk extraction but avoids projection bugs.
|
||
|
||
**Step R.5: Confirm run frequency and forecast range**
|
||
|
||
HRDPS: 4× daily (00/06/12/18Z), forecasts to 48h. Verify via datamart. This is half HRRR's run frequency (hourly) and 2.7× longer (48h vs 18h).
|
||
|
||
**Decision gate:** After R.5, confirm with your human partner which forecast range to ingest. Recommendation: start with f00–f24 to match the "current + 1 day" use case. Extending to f48 doubles storage and compute and probably isn't worth it for amateur radio planning.
|
||
|
||
---
|
||
|
||
## Task 1: `HrdpsClient` — surface + pressure fetch
|
||
|
||
**Files:**
|
||
- Create: `lib/microwaveprop/weather/hrdps_client.ex`
|
||
- Test: `test/microwaveprop/weather/hrdps_client_test.exs`
|
||
- Fixture: `test/support/fixtures/hrdps_sample_sfc.grib2` + `test/support/fixtures/hrdps_sample_prs.grib2`
|
||
|
||
**Step 1.1: Write failing URL builder test**
|
||
|
||
```elixir
|
||
test "builds correct CMC datamart URL for surface temp" do
|
||
url = HrdpsClient.grib_url(~D[2026-04-13], 12, 6, :surface, :tmp)
|
||
assert url =~ "/model_hrdps/continental/2.5km/12/006/CMC_hrdps_continental_TMP_TGL_2_ps2.5km_2026041312_P006-00.grib2"
|
||
end
|
||
```
|
||
|
||
Adjust the format string after pre-flight research confirms the exact naming.
|
||
|
||
**Step 1.2–1.4: Write → fail → implement → pass**
|
||
|
||
Reuse `HrrrClient.parse_idx/1`, `HrrrClient.byte_ranges_for_messages/2`, and `HrrrClient.download_grib_ranges/2` — they're format-agnostic. Only the URL pattern and the message set differ.
|
||
|
||
**Step 1.5: Implement `fetch_grid/3`**
|
||
|
||
```elixir
|
||
@spec fetch_grid([{float(), float()}], DateTime.t(), keyword()) ::
|
||
{:ok, %{{float(), float()} => map()}} | {:error, term()}
|
||
def fetch_grid(points, run_time, opts \\ [])
|
||
```
|
||
|
||
Same signature as `HrrrClient.fetch_grid/3` so `PropagationGridWorker.process_forecast_hour/4` can call either one polymorphically in Task 6.
|
||
|
||
**Step 1.6: Nail the projection**
|
||
|
||
Write a focused test: pick a known Canadian airport (CYYR = Goose Bay, 53.32°N, 60.42°W) and verify the HRDPS profile at that point matches the UWYO radiosonde for the same `valid_time` within a reasonable tolerance (surface temp ±3°C, dewpoint ±4°C). If they don't match, the projection is wrong and you need to debug wgrib2's `-lon` extraction vs the CMC rotated grid.
|
||
|
||
**Step 1.7: Commit**
|
||
|
||
---
|
||
|
||
## Task 2: Native-level duct metrics via `HrdpsNativeClient`
|
||
|
||
**Files:**
|
||
- Create: `lib/microwaveprop/weather/hrdps_native_client.ex`
|
||
- Test: `test/microwaveprop/weather/hrdps_native_client_test.exs`
|
||
|
||
**Step 2.1: Confirm HRDPS publishes native hybrid levels**
|
||
|
||
HRDPS uses CMC's GEM dynamical core with hybrid sigma-pressure levels (62 of them). The datamart file naming for native levels is `*_HY_*` or within `model_hrdps/continental/2.5km/native/` — confirm in pre-flight.
|
||
|
||
**If native levels are NOT published**: this task is impossible; skip it and fall back to pressure-level duct analysis (coarser — 31 levels vs 62 — but still works). The map's ducting detection quality will be slightly worse over VE airspace than over CONUS, which is acceptable for a first pass.
|
||
|
||
**If native levels ARE published**: mirror `HrrrNativeClient.fetch_native_duct_grid/4`. Use `Microwaveprop.Propagation.Duct.analyze/1` which is projection-agnostic — it just needs a list of levels with temperature, pressure, and specific humidity.
|
||
|
||
**Step 2.2–2.5: TDD cycle as above**
|
||
|
||
**Step 2.6: Commit**
|
||
|
||
---
|
||
|
||
## Task 3: `hrdps_profiles` table + schema
|
||
|
||
**Files:**
|
||
- Create: `priv/repo/migrations/{timestamp}_create_hrdps_profiles.exs`
|
||
- Create: `lib/microwaveprop/weather/hrdps_profile.ex`
|
||
- Test: `test/microwaveprop/weather/hrdps_profile_test.exs`
|
||
|
||
**Step 3.1–3.4**: Mirror `hrrr_profiles` exactly. Same columns, same indexes. The only reason to have a separate table is operational isolation — don't contaminate the 10M-row HRRR partition with 8M-row HRDPS rows, and let you drop/reprocess HRDPS without touching HRRR.
|
||
|
||
Use daily partitioning on `valid_time` just like `hrrr_profiles`.
|
||
|
||
**Step 3.5: Partition-pruning migration**
|
||
|
||
Add an `Oban.Plugins.Cron` entry for `HrdpsPartitionCleanup` similar to `HrrrPartitionCleanup` if one exists.
|
||
|
||
**Step 3.6: Commit**
|
||
|
||
---
|
||
|
||
## Task 4: `HrdpsGridWorker` — hourly compute over Canadian grid
|
||
|
||
**Files:**
|
||
- Create: `lib/microwaveprop/workers/hrdps_grid_worker.ex`
|
||
- Test: `test/microwaveprop/workers/hrdps_grid_worker_test.exs`
|
||
- Modify: `lib/microwaveprop/propagation/grid.ex` (add `canadian_points/0`)
|
||
|
||
**Step 4.1: Define the Canadian grid**
|
||
|
||
At HRDPS 2.5 km resolution, Canada (42N-83N, -141 to -52W) has ~1.5M cells. Too many to score every hour. Instead:
|
||
- Sample at 0.125° (14 km) to match HRRR's output grid → ~100K points
|
||
- For the subset of points that are INSIDE HRRR's CONUS footprint, prefer HRRR; OUTSIDE, use HRDPS
|
||
- Store the "outside HRRR" mask as `Grid.hrdps_only_points/0`
|
||
|
||
**Step 4.2: Worker perform/1**
|
||
|
||
Mirror `PropagationGridWorker.process_forecast_hour/4` but with `HrdpsClient.fetch_grid/3` and `HrdpsNativeClient.fetch_native_duct_grid/4`.
|
||
|
||
**Step 4.3: Scoring**
|
||
|
||
Call `Propagation.score_grid_point/4` — it's already projection-agnostic. It takes a profile map and valid_time and produces per-band scores. No changes needed to the scorer.
|
||
|
||
**Step 4.4: Upsert into `propagation_scores`**
|
||
|
||
`grid_scores` is already indexed by `(lat, lon, valid_time, band_mhz)`. HRDPS and HRRR both write into the same table — the unique key prevents collisions because HRDPS cells have different (lat, lon) than HRRR cells at the shared 0.125° resolution IF you don't re-score HRRR's cells from HRDPS. Use the `hrdps_only_points/0` mask to guarantee no overlap.
|
||
|
||
**Step 4.5: TDD cycle, commit**
|
||
|
||
---
|
||
|
||
## Task 5: Oban queue + cron
|
||
|
||
**Files:**
|
||
- Modify: `config/config.exs`, `config/runtime.exs`
|
||
|
||
**Step 5.1: Add `hrdps` queue**
|
||
|
||
```elixir
|
||
queues: [
|
||
# ... existing
|
||
hrdps: 1 # HRDPS fetch is sequential like HRRR (memory pressure)
|
||
]
|
||
```
|
||
|
||
**Step 5.2: Add cron**
|
||
|
||
```elixir
|
||
# HRDPS runs 00/06/12/18Z. Aligned with RDPS cadence.
|
||
{"35 */6 * * *", Microwaveprop.Workers.HrdpsGridWorker}
|
||
```
|
||
|
||
Note: 35 past the hour, not 5, because PropagationGridWorker (HRRR) fires at 5 past. Stagger by 30 minutes so they don't compete for the DB and the wgrib2 subprocess slot.
|
||
|
||
**Step 5.3: `mix compile` sanity-check**
|
||
|
||
**Step 5.4: Commit**
|
||
|
||
---
|
||
|
||
## Task 6: Map overlay — extend prop map to cover Canada
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop_web/live/map_live.ex` — `@initial_bounds`, timeline
|
||
- Modify: `assets/js/propagation_map_hook.ts` — heatmap rendering bounds
|
||
- Modify: `lib/microwaveprop/propagation/grid.ex` — export `north_america_points/0` that returns HRRR ∪ HRDPS
|
||
|
||
**Step 6.1: Write failing LiveView test**
|
||
|
||
```elixir
|
||
test "/map renders scores for a Canadian grid cell (53N, -113W)" do
|
||
{:ok, view, html} = live(conn, ~p"/map")
|
||
# after PropagationGridWorker (HRRR) and HrdpsGridWorker (HRDPS) have both written scores
|
||
assert html =~ "53.0" # or whatever assertion proves the HRDPS cell made it to the client
|
||
end
|
||
```
|
||
|
||
**Step 6.2: Widen `@initial_bounds` in map_live.ex**
|
||
|
||
From the CONUS-only `(23.5, 49.5, -124.5, -66.9)` to `(23.5, 83.0, -141.0, -52.0)`. Zoom out default to show all of North America.
|
||
|
||
**Step 6.3: Verify front-end renders Canadian points**
|
||
|
||
```bash
|
||
mix phx.server
|
||
# Open http://localhost:4000/map
|
||
# Confirm colored cells north of 49N after the first HRDPS run completes
|
||
```
|
||
|
||
**Step 6.4: Commit**
|
||
|
||
---
|
||
|
||
## Task 7: Terrain + QSO path analysis north of the border
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop/terrain/terrain_analysis.ex` (if needed)
|
||
- Modify: `lib/microwaveprop/workers/terrain_profile_worker.ex`
|
||
- Possibly: mount SRTM data covering Canada onto the production pods
|
||
|
||
**Step 7.1: Check SRTM coverage**
|
||
|
||
SRTM1 covers 60°S–60°N. Canadian radiosondes and stations above 60°N (Yellowknife, Whitehorse, Inuvik, Resolute) have NO SRTM coverage. The terrain analysis pipeline needs a fallback — likely CDEM (Canadian Digital Elevation Model) or ASTER GDEM.
|
||
|
||
**Decision gate with your human partner**: Does the feature need to work above 60°N? If yes, this is a multi-day sub-project (Task 7 becomes a whole new plan). If no — scope to 42–60°N and explicitly document that Inuvik paths won't get terrain analysis.
|
||
|
||
**Step 7.2 onward**: Out of scope for this plan. Write a follow-up plan for Arctic terrain if your human partner wants coverage.
|
||
|
||
**Step 7.3: Commit the scope decision to `docs/plans/` or a memory file**
|
||
|
||
---
|
||
|
||
## Task 8: Freshness monitoring + observability
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop/propagation/freshness_monitor.ex`
|
||
- Modify: `lib/microwaveprop_web/live/status_live.ex`
|
||
|
||
**Step 8.1: Extend FreshnessMonitor to track HRDPS run time**
|
||
|
||
Add a `hrdps_last_run_at` field to the monitor state, surface it on `/status`.
|
||
|
||
**Step 8.2: Alert when HRDPS run_time is > 4 hours stale**
|
||
|
||
HRDPS runs every 6 hours, publishes ~90 min after, so a healthy pipeline never exceeds ~2.5 hours stale. 4 hours = something broken.
|
||
|
||
**Step 8.3: Commit**
|
||
|
||
---
|
||
|
||
## Task 9: Precommit, docs, PR
|
||
|
||
**Step 9.1: `mix precommit`**
|
||
|
||
Must show 0 failures.
|
||
|
||
**Step 9.2: Update `CLAUDE.md`**
|
||
|
||
- Add HRDPS to `### Key Data Sources`
|
||
- Add `HrdpsGridWorker` + `HrdpsFetchWorker` to the background queue table
|
||
- Add a note on the CONUS → North America scope change
|
||
|
||
**Step 9.3: Update `memory/reference_hrdps_canada.md`**
|
||
|
||
Mark the plan as executed — include the final ingestion rate, queue config, and any projection gotchas that surfaced.
|
||
|
||
**Step 9.4: Commit, push to `github` and `origin`, let Flux deploy**
|
||
|
||
Do NOT push to `dokku`.
|
||
|
||
---
|
||
|
||
## Known risks + gotchas
|
||
|
||
- **Projection**: HRDPS rotated lat/lon is the #1 source of bugs. Burn a day in Task 1.6 validating a dozen points against UWYO or ASOS surface obs before moving on. A 10 km projection offset is invisible on the propagation map but destroys terrain-path analysis.
|
||
- **Memory pressure**: The existing HRRR worker hits `:system_memory_high_watermark` alarms at ~4 GB. HRDPS grid is 3× larger in cell count. Plan for 8 GB per pod for the prop namespace, or offload HRDPS fetch to a dedicated node via Kubernetes node affinity.
|
||
- **Publish time**: CMC's publish delay can be erratic — 90 min on good days, 3+ hours under load. Build retry-with-backoff into `HrdpsGridWorker.perform/1` so the cron doesn't fire, find no data, and dead-letter immediately.
|
||
- **Arctic terrain**: SRTM stops at 60°N. Accept or fix — don't silently serve wrong diffraction losses to stations in Yellowknife.
|
||
- **HRRR/HRDPS overlap**: The two models disagree on the US/Canadian border. Pick a rule (prefer HRRR south of 49°N, prefer HRDPS north) and encode it in `Grid.point_source/2`.
|
||
- **Scope creep**: This plan is already ~5 days of work. Resist adding "while we're at it, let's also fix the native-level ducting for rotated grids" side-quests. Ship the minimum viable Canadian prop grid first.
|
||
|
||
## Rough effort estimate
|
||
|
||
| Task | Estimate |
|
||
|---|---|
|
||
| Pre-flight research | 2 hours |
|
||
| Task 1: HrdpsClient | 1 day (projection debug dominates) |
|
||
| Task 2: HrdpsNativeClient | 0.5 day (or skipped if no native) |
|
||
| Task 3: hrdps_profiles schema | 2 hours |
|
||
| Task 4: HrdpsGridWorker | 1 day |
|
||
| Task 5: Cron wiring | 1 hour |
|
||
| Task 6: Map overlay | 0.5 day |
|
||
| Task 7: Terrain coverage decision | 2 hours (punt if Arctic gets out-of-scope) |
|
||
| Task 8: Freshness monitoring | 2 hours |
|
||
| Task 9: Precommit + docs | 2 hours |
|
||
| **Total** | **~5 days** |
|
||
|
||
Compare to RDPS plan (~2 days) and the UWYO Canadian soundings (~4 hours, already done). Do HRDPS only when the RDPS 10 km coverage feels limiting, not before.
|