# HRDPS Canadian Propagation Grid Implementation Plan (SUPERSEDED) > **SUPERSEDED 2026-04-29 by `docs/plans/2026-04-29-hrdps-canadian-prop-grid.md`.** A live probe of MSC Datamart on 2026-04-29 found that ECCC reorganized the URL structure (the `/model_hrdps/` flat root in the pre-flight steps below 404s now) and that HRDPS publishes one variable per GRIB2 file rather than HRRR's bundled wrfsfcf/wrfprsf format — so the Stage 1 idx + byte-range fetch shape this plan inherits from HrrrClient does not apply. Use the 2026-04-29 plan instead. > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. **Goal:** Extend the propagation scoring grid to cover all of Canada at HRDPS's 2.5 km resolution — the Canadian analog to HRRR. This lights up the `/map` for VE stations, gives terrain-diffraction analysis for VE QSOs, and unlocks cross-border path analysis for KP/W7-to-VE7 tropo paths. **Architecture:** Build a parallel NWP ingestion pipeline (`HrdpsClient` + `HrdpsGridWorker` + `hrdps_profiles` table) that matches the HRRR structure one-for-one, with three major differences: rotated lat/lon projection, CMC file naming (`CMC_hrdps_continental_*_ps2.5km_*_{FFF}.grib2`), and a different native-level grid spec for duct analysis. A new `propagation_scores` region tag distinguishes HRRR-sourced vs HRDPS-sourced cells so the front-end can render them uniformly. **Tech Stack:** Elixir 1.15, Req, Nx (optional for duct analysis), Postgres, Oban, wgrib2, proj (for projection conversions if needed) **Prerequisite:** Task 6 of the RDPS plan (`2026-04-13-rdps-vertical-profiles.md`) — or at minimum a working pattern for "score grid_data from a non-HRRR source". If you're starting HRDPS fresh, Task 6 below stands alone but will duplicate some work from the RDPS plan. --- ## Pre-flight research (1–2 hours) **Step R.1: Confirm HRDPS datamart URL pattern** ```bash curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/ 2>&1 | head -30 curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/00/000/ 2>&1 | head -30 ``` Expected: file list with entries like `CMC_hrdps_continental_TMP_TGL_2_ps2.5km_{YYYYMMDDHH}_P{FFF}-00.grib2`. **Step R.2: Download one sample + inspect with wgrib2** ```bash wgrib2 CMC_hrdps_continental_TMP_TGL_2_ps2.5km_2026041300_P000-00.grib2 -grid -v ``` Expected: confirms grid is rotated lat/lon, gives dimensions (typically ~2500×1800), and lists all variables. **Step R.3: Confirm pressure-level files exist** HRDPS publishes surface + pressure-level files separately (like HRRR wrfsfc vs wrfprs). Find the pressure-level variant: ```bash curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/00/000/ | grep -i 'ISBL' ``` If separate files per pressure level, the ingestion must fan out like `HrrrClient.pressure_messages/0`. If a single combined file, one download gets everything. **Step R.4: Decide projection strategy** wgrib2 can extract points from rotated lat/lon directly with `-lon LON LAT`. For extracting the full grid as a flat array, you need `proj` or a wgrib2 `-new_grid` reprojection. **Recommended**: never reproject — always extract point-by-point for the grid points we care about. Slower than bulk extraction but avoids projection bugs. **Step R.5: Confirm run frequency and forecast range** HRDPS: 4× daily (00/06/12/18Z), forecasts to 48h. Verify via datamart. This is half HRRR's run frequency (hourly) and 2.7× longer (48h vs 18h). **Decision gate:** After R.5, confirm with your human partner which forecast range to ingest. Recommendation: start with f00–f24 to match the "current + 1 day" use case. Extending to f48 doubles storage and compute and probably isn't worth it for amateur radio planning. --- ## Task 1: `HrdpsClient` — surface + pressure fetch **Files:** - Create: `lib/microwaveprop/weather/hrdps_client.ex` - Test: `test/microwaveprop/weather/hrdps_client_test.exs` - Fixture: `test/support/fixtures/hrdps_sample_sfc.grib2` + `test/support/fixtures/hrdps_sample_prs.grib2` **Step 1.1: Write failing URL builder test** ```elixir test "builds correct CMC datamart URL for surface temp" do url = HrdpsClient.grib_url(~D[2026-04-13], 12, 6, :surface, :tmp) assert url =~ "/model_hrdps/continental/2.5km/12/006/CMC_hrdps_continental_TMP_TGL_2_ps2.5km_2026041312_P006-00.grib2" end ``` Adjust the format string after pre-flight research confirms the exact naming. **Step 1.2–1.4: Write → fail → implement → pass** Reuse `HrrrClient.parse_idx/1`, `HrrrClient.byte_ranges_for_messages/2`, and `HrrrClient.download_grib_ranges/2` — they're format-agnostic. Only the URL pattern and the message set differ. **Step 1.5: Implement `fetch_grid/3`** ```elixir @spec fetch_grid([{float(), float()}], DateTime.t(), keyword()) :: {:ok, %{{float(), float()} => map()}} | {:error, term()} def fetch_grid(points, run_time, opts \\ []) ``` Same signature as `HrrrClient.fetch_grid/3` so `PropagationGridWorker.process_forecast_hour/4` can call either one polymorphically in Task 6. **Step 1.6: Nail the projection** Write a focused test: pick a known Canadian airport (CYYR = Goose Bay, 53.32°N, 60.42°W) and verify the HRDPS profile at that point matches the UWYO radiosonde for the same `valid_time` within a reasonable tolerance (surface temp ±3°C, dewpoint ±4°C). If they don't match, the projection is wrong and you need to debug wgrib2's `-lon` extraction vs the CMC rotated grid. **Step 1.7: Commit** --- ## Task 2: Native-level duct metrics via `HrdpsNativeClient` **Files:** - Create: `lib/microwaveprop/weather/hrdps_native_client.ex` - Test: `test/microwaveprop/weather/hrdps_native_client_test.exs` **Step 2.1: Confirm HRDPS publishes native hybrid levels** HRDPS uses CMC's GEM dynamical core with hybrid sigma-pressure levels (62 of them). The datamart file naming for native levels is `*_HY_*` or within `model_hrdps/continental/2.5km/native/` — confirm in pre-flight. **If native levels are NOT published**: this task is impossible; skip it and fall back to pressure-level duct analysis (coarser — 31 levels vs 62 — but still works). The map's ducting detection quality will be slightly worse over VE airspace than over CONUS, which is acceptable for a first pass. **If native levels ARE published**: mirror `HrrrNativeClient.fetch_native_duct_grid/4`. Use `Microwaveprop.Propagation.Duct.analyze/1` which is projection-agnostic — it just needs a list of levels with temperature, pressure, and specific humidity. **Step 2.2–2.5: TDD cycle as above** **Step 2.6: Commit** --- ## Task 3: `hrdps_profiles` table + schema **Files:** - Create: `priv/repo/migrations/{timestamp}_create_hrdps_profiles.exs` - Create: `lib/microwaveprop/weather/hrdps_profile.ex` - Test: `test/microwaveprop/weather/hrdps_profile_test.exs` **Step 3.1–3.4**: Mirror `hrrr_profiles` exactly. Same columns, same indexes. The only reason to have a separate table is operational isolation — don't contaminate the 10M-row HRRR partition with 8M-row HRDPS rows, and let you drop/reprocess HRDPS without touching HRRR. Use daily partitioning on `valid_time` just like `hrrr_profiles`. **Step 3.5: Partition-pruning migration** Add an `Oban.Plugins.Cron` entry for `HrdpsPartitionCleanup` similar to `HrrrPartitionCleanup` if one exists. **Step 3.6: Commit** --- ## Task 4: `HrdpsGridWorker` — hourly compute over Canadian grid **Files:** - Create: `lib/microwaveprop/workers/hrdps_grid_worker.ex` - Test: `test/microwaveprop/workers/hrdps_grid_worker_test.exs` - Modify: `lib/microwaveprop/propagation/grid.ex` (add `canadian_points/0`) **Step 4.1: Define the Canadian grid** At HRDPS 2.5 km resolution, Canada (42N-83N, -141 to -52W) has ~1.5M cells. Too many to score every hour. Instead: - Sample at 0.125° (14 km) to match HRRR's output grid → ~100K points - For the subset of points that are INSIDE HRRR's CONUS footprint, prefer HRRR; OUTSIDE, use HRDPS - Store the "outside HRRR" mask as `Grid.hrdps_only_points/0` **Step 4.2: Worker perform/1** Mirror `PropagationGridWorker.process_forecast_hour/4` but with `HrdpsClient.fetch_grid/3` and `HrdpsNativeClient.fetch_native_duct_grid/4`. **Step 4.3: Scoring** Call `Propagation.score_grid_point/4` — it's already projection-agnostic. It takes a profile map and valid_time and produces per-band scores. No changes needed to the scorer. **Step 4.4: Upsert into `propagation_scores`** `grid_scores` is already indexed by `(lat, lon, valid_time, band_mhz)`. HRDPS and HRRR both write into the same table — the unique key prevents collisions because HRDPS cells have different (lat, lon) than HRRR cells at the shared 0.125° resolution IF you don't re-score HRRR's cells from HRDPS. Use the `hrdps_only_points/0` mask to guarantee no overlap. **Step 4.5: TDD cycle, commit** --- ## Task 5: Oban queue + cron **Files:** - Modify: `config/config.exs`, `config/runtime.exs` **Step 5.1: Add `hrdps` queue** ```elixir queues: [ # ... existing hrdps: 1 # HRDPS fetch is sequential like HRRR (memory pressure) ] ``` **Step 5.2: Add cron** ```elixir # HRDPS runs 00/06/12/18Z. Aligned with RDPS cadence. {"35 */6 * * *", Microwaveprop.Workers.HrdpsGridWorker} ``` Note: 35 past the hour, not 5, because PropagationGridWorker (HRRR) fires at 5 past. Stagger by 30 minutes so they don't compete for the DB and the wgrib2 subprocess slot. **Step 5.3: `mix compile` sanity-check** **Step 5.4: Commit** --- ## Task 6: Map overlay — extend prop map to cover Canada **Files:** - Modify: `lib/microwaveprop_web/live/map_live.ex` — `@initial_bounds`, timeline - Modify: `assets/js/propagation_map_hook.ts` — heatmap rendering bounds - Modify: `lib/microwaveprop/propagation/grid.ex` — export `north_america_points/0` that returns HRRR ∪ HRDPS **Step 6.1: Write failing LiveView test** ```elixir test "/map renders scores for a Canadian grid cell (53N, -113W)" do {:ok, view, html} = live(conn, ~p"/map") # after PropagationGridWorker (HRRR) and HrdpsGridWorker (HRDPS) have both written scores assert html =~ "53.0" # or whatever assertion proves the HRDPS cell made it to the client end ``` **Step 6.2: Widen `@initial_bounds` in map_live.ex** From the CONUS-only `(23.5, 49.5, -124.5, -66.9)` to `(23.5, 83.0, -141.0, -52.0)`. Zoom out default to show all of North America. **Step 6.3: Verify front-end renders Canadian points** ```bash mix phx.server # Open http://localhost:4000/map # Confirm colored cells north of 49N after the first HRDPS run completes ``` **Step 6.4: Commit** --- ## Task 7: Terrain + QSO path analysis north of the border **Files:** - Modify: `lib/microwaveprop/terrain/terrain_analysis.ex` (if needed) - Modify: `lib/microwaveprop/workers/terrain_profile_worker.ex` - Possibly: mount SRTM data covering Canada onto the production pods **Step 7.1: Check SRTM coverage** SRTM1 covers 60°S–60°N. Canadian radiosondes and stations above 60°N (Yellowknife, Whitehorse, Inuvik, Resolute) have NO SRTM coverage. The terrain analysis pipeline needs a fallback — likely CDEM (Canadian Digital Elevation Model) or ASTER GDEM. **Decision gate with your human partner**: Does the feature need to work above 60°N? If yes, this is a multi-day sub-project (Task 7 becomes a whole new plan). If no — scope to 42–60°N and explicitly document that Inuvik paths won't get terrain analysis. **Step 7.2 onward**: Out of scope for this plan. Write a follow-up plan for Arctic terrain if your human partner wants coverage. **Step 7.3: Commit the scope decision to `docs/plans/` or a memory file** --- ## Task 8: Freshness monitoring + observability **Files:** - Modify: `lib/microwaveprop/propagation/freshness_monitor.ex` - Modify: `lib/microwaveprop_web/live/status_live.ex` **Step 8.1: Extend FreshnessMonitor to track HRDPS run time** Add a `hrdps_last_run_at` field to the monitor state, surface it on `/status`. **Step 8.2: Alert when HRDPS run_time is > 4 hours stale** HRDPS runs every 6 hours, publishes ~90 min after, so a healthy pipeline never exceeds ~2.5 hours stale. 4 hours = something broken. **Step 8.3: Commit** --- ## Task 9: Precommit, docs, PR **Step 9.1: `mix precommit`** Must show 0 failures. **Step 9.2: Update `CLAUDE.md`** - Add HRDPS to `### Key Data Sources` - Add `HrdpsGridWorker` + `HrdpsFetchWorker` to the background queue table - Add a note on the CONUS → North America scope change **Step 9.3: Update `memory/reference_hrdps_canada.md`** Mark the plan as executed — include the final ingestion rate, queue config, and any projection gotchas that surfaced. **Step 9.4: Commit, push to `github` and `origin`, let Flux deploy** Do NOT push to `dokku`. --- ## Known risks + gotchas - **Projection**: HRDPS rotated lat/lon is the #1 source of bugs. Burn a day in Task 1.6 validating a dozen points against UWYO or ASOS surface obs before moving on. A 10 km projection offset is invisible on the propagation map but destroys terrain-path analysis. - **Memory pressure**: The existing HRRR worker hits `:system_memory_high_watermark` alarms at ~4 GB. HRDPS grid is 3× larger in cell count. Plan for 8 GB per pod for the prop namespace, or offload HRDPS fetch to a dedicated node via Kubernetes node affinity. - **Publish time**: CMC's publish delay can be erratic — 90 min on good days, 3+ hours under load. Build retry-with-backoff into `HrdpsGridWorker.perform/1` so the cron doesn't fire, find no data, and dead-letter immediately. - **Arctic terrain**: SRTM stops at 60°N. Accept or fix — don't silently serve wrong diffraction losses to stations in Yellowknife. - **HRRR/HRDPS overlap**: The two models disagree on the US/Canadian border. Pick a rule (prefer HRRR south of 49°N, prefer HRDPS north) and encode it in `Grid.point_source/2`. - **Scope creep**: This plan is already ~5 days of work. Resist adding "while we're at it, let's also fix the native-level ducting for rotated grids" side-quests. Ship the minimum viable Canadian prop grid first. ## Rough effort estimate | Task | Estimate | |---|---| | Pre-flight research | 2 hours | | Task 1: HrdpsClient | 1 day (projection debug dominates) | | Task 2: HrdpsNativeClient | 0.5 day (or skipped if no native) | | Task 3: hrdps_profiles schema | 2 hours | | Task 4: HrdpsGridWorker | 1 day | | Task 5: Cron wiring | 1 hour | | Task 6: Map overlay | 0.5 day | | Task 7: Terrain coverage decision | 2 hours (punt if Arctic gets out-of-scope) | | Task 8: Freshness monitoring | 2 hours | | Task 9: Precommit + docs | 2 hours | | **Total** | **~5 days** | Compare to RDPS plan (~2 days) and the UWYO Canadian soundings (~4 hours, already done). Do HRDPS only when the RDPS 10 km coverage feels limiting, not before.