# HRDPS Canadian Propagation Grid Implementation Plan (SUPERSEDED)

> **SUPERSEDED 2026-04-29 by `docs/plans/2026-04-29-hrdps-canadian-prop-grid.md`.** A live probe of MSC Datamart on 2026-04-29 found that ECCC reorganized the URL structure (the `/model_hrdps/` flat root in the pre-flight steps below 404s now) and that HRDPS publishes one variable per GRIB2 file rather than HRRR's bundled wrfsfcf/wrfprsf format — so the Stage 1 idx + byte-range fetch shape this plan inherits from HrrrClient does not apply. Use the 2026-04-29 plan instead.

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Extend the propagation scoring grid to cover all of Canada at HRDPS's 2.5 km resolution — the Canadian analog to HRRR. This lights up the `/map` for VE stations, gives terrain-diffraction analysis for VE QSOs, and unlocks cross-border path analysis for KP/W7-to-VE7 tropo paths.

**Architecture:** Build a parallel NWP ingestion pipeline (`HrdpsClient` + `HrdpsGridWorker` + `hrdps_profiles` table) that matches the HRRR structure one-for-one, with three major differences: rotated lat/lon projection, CMC file naming (`CMC_hrdps_continental_*_ps2.5km_*_{FFF}.grib2`), and a different native-level grid spec for duct analysis. A new `propagation_scores` region tag distinguishes HRRR-sourced vs HRDPS-sourced cells so the front-end can render them uniformly.

**Tech Stack:** Elixir 1.15, Req, Nx (optional for duct analysis), Postgres, Oban, wgrib2, proj (for projection conversions if needed)

**Prerequisite:** Task 6 of the RDPS plan (`2026-04-13-rdps-vertical-profiles.md`) — or at minimum a working pattern for "score grid_data from a non-HRRR source". If you're starting HRDPS fresh, Task 6 below stands alone but will duplicate some work from the RDPS plan.

---

## Pre-flight research (1–2 hours)

**Step R.1: Confirm HRDPS datamart URL pattern**

```bash
curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/ 2>&1 | head -30
curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/00/000/ 2>&1 | head -30
```

Expected: file list with entries like `CMC_hrdps_continental_TMP_TGL_2_ps2.5km_{YYYYMMDDHH}_P{FFF}-00.grib2`.

**Step R.2: Download one sample + inspect with wgrib2**

```bash
wgrib2 CMC_hrdps_continental_TMP_TGL_2_ps2.5km_2026041300_P000-00.grib2 -grid -v
```

Expected: confirms grid is rotated lat/lon, gives dimensions (typically ~2500×1800), and lists all variables.

**Step R.3: Confirm pressure-level files exist**

HRDPS publishes surface + pressure-level files separately (like HRRR wrfsfc vs wrfprs). Find the pressure-level variant:

```bash
curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/00/000/ | grep -i 'ISBL'
```

If separate files per pressure level, the ingestion must fan out like `HrrrClient.pressure_messages/0`. If a single combined file, one download gets everything.

**Step R.4: Decide projection strategy**

wgrib2 can extract points from rotated lat/lon directly with `-lon LON LAT`. For extracting the full grid as a flat array, you need `proj` or a wgrib2 `-new_grid` reprojection. **Recommended**: never reproject — always extract point-by-point for the grid points we care about. Slower than bulk extraction but avoids projection bugs.

**Step R.5: Confirm run frequency and forecast range**

HRDPS: 4× daily (00/06/12/18Z), forecasts to 48h. Verify via datamart. This is half HRRR's run frequency (hourly) and 2.7× longer (48h vs 18h).

**Decision gate:** After R.5, confirm with your human partner which forecast range to ingest. Recommendation: start with f00–f24 to match the "current + 1 day" use case. Extending to f48 doubles storage and compute and probably isn't worth it for amateur radio planning.

---

## Task 1: `HrdpsClient` — surface + pressure fetch

**Files:**
- Create: `lib/microwaveprop/weather/hrdps_client.ex`
- Test: `test/microwaveprop/weather/hrdps_client_test.exs`
- Fixture: `test/support/fixtures/hrdps_sample_sfc.grib2` + `test/support/fixtures/hrdps_sample_prs.grib2`

**Step 1.1: Write failing URL builder test**

```elixir
test "builds correct CMC datamart URL for surface temp" do
  url = HrdpsClient.grib_url(~D[2026-04-13], 12, 6, :surface, :tmp)
  assert url =~ "/model_hrdps/continental/2.5km/12/006/CMC_hrdps_continental_TMP_TGL_2_ps2.5km_2026041312_P006-00.grib2"
end
```

Adjust the format string after pre-flight research confirms the exact naming.

**Step 1.2–1.4: Write → fail → implement → pass**

Reuse `HrrrClient.parse_idx/1`, `HrrrClient.byte_ranges_for_messages/2`, and `HrrrClient.download_grib_ranges/2` — they're format-agnostic. Only the URL pattern and the message set differ.

**Step 1.5: Implement `fetch_grid/3`**

```elixir
@spec fetch_grid([{float(), float()}], DateTime.t(), keyword()) ::
        {:ok, %{{float(), float()} => map()}} | {:error, term()}
def fetch_grid(points, run_time, opts \\ [])
```

Same signature as `HrrrClient.fetch_grid/3` so `PropagationGridWorker.process_forecast_hour/4` can call either one polymorphically in Task 6.

**Step 1.6: Nail the projection**

Write a focused test: pick a known Canadian airport (CYYR = Goose Bay, 53.32°N, 60.42°W) and verify the HRDPS profile at that point matches the UWYO radiosonde for the same `valid_time` within a reasonable tolerance (surface temp ±3°C, dewpoint ±4°C). If they don't match, the projection is wrong and you need to debug wgrib2's `-lon` extraction vs the CMC rotated grid.

**Step 1.7: Commit**

---

## Task 2: Native-level duct metrics via `HrdpsNativeClient`

**Files:**
- Create: `lib/microwaveprop/weather/hrdps_native_client.ex`
- Test: `test/microwaveprop/weather/hrdps_native_client_test.exs`

**Step 2.1: Confirm HRDPS publishes native hybrid levels**

HRDPS uses CMC's GEM dynamical core with hybrid sigma-pressure levels (62 of them). The datamart file naming for native levels is `*_HY_*` or within `model_hrdps/continental/2.5km/native/` — confirm in pre-flight.

**If native levels are NOT published**: this task is impossible; skip it and fall back to pressure-level duct analysis (coarser — 31 levels vs 62 — but still works). The map's ducting detection quality will be slightly worse over VE airspace than over CONUS, which is acceptable for a first pass.

**If native levels ARE published**: mirror `HrrrNativeClient.fetch_native_duct_grid/4`. Use `Microwaveprop.Propagation.Duct.analyze/1` which is projection-agnostic — it just needs a list of levels with temperature, pressure, and specific humidity.

**Step 2.2–2.5: TDD cycle as above**

**Step 2.6: Commit**

---

## Task 3: `hrdps_profiles` table + schema

**Files:**
- Create: `priv/repo/migrations/{timestamp}_create_hrdps_profiles.exs`
- Create: `lib/microwaveprop/weather/hrdps_profile.ex`
- Test: `test/microwaveprop/weather/hrdps_profile_test.exs`

**Step 3.1–3.4**: Mirror `hrrr_profiles` exactly. Same columns, same indexes. The only reason to have a separate table is operational isolation — don't contaminate the 10M-row HRRR partition with 8M-row HRDPS rows, and let you drop/reprocess HRDPS without touching HRRR.

Use daily partitioning on `valid_time` just like `hrrr_profiles`.

**Step 3.5: Partition-pruning migration**

Add an `Oban.Plugins.Cron` entry for `HrdpsPartitionCleanup` similar to `HrrrPartitionCleanup` if one exists.

**Step 3.6: Commit**

---

## Task 4: `HrdpsGridWorker` — hourly compute over Canadian grid

**Files:**
- Create: `lib/microwaveprop/workers/hrdps_grid_worker.ex`
- Test: `test/microwaveprop/workers/hrdps_grid_worker_test.exs`
- Modify: `lib/microwaveprop/propagation/grid.ex` (add `canadian_points/0`)

**Step 4.1: Define the Canadian grid**

At HRDPS 2.5 km resolution, Canada (42N-83N, -141 to -52W) has ~1.5M cells. Too many to score every hour. Instead:
- Sample at 0.125° (14 km) to match HRRR's output grid → ~100K points
- For the subset of points that are INSIDE HRRR's CONUS footprint, prefer HRRR; OUTSIDE, use HRDPS
- Store the "outside HRRR" mask as `Grid.hrdps_only_points/0`

**Step 4.2: Worker perform/1**

Mirror `PropagationGridWorker.process_forecast_hour/4` but with `HrdpsClient.fetch_grid/3` and `HrdpsNativeClient.fetch_native_duct_grid/4`.

**Step 4.3: Scoring**

Call `Propagation.score_grid_point/4` — it's already projection-agnostic. It takes a profile map and valid_time and produces per-band scores. No changes needed to the scorer.

**Step 4.4: Upsert into `propagation_scores`**

`grid_scores` is already indexed by `(lat, lon, valid_time, band_mhz)`. HRDPS and HRRR both write into the same table — the unique key prevents collisions because HRDPS cells have different (lat, lon) than HRRR cells at the shared 0.125° resolution IF you don't re-score HRRR's cells from HRDPS. Use the `hrdps_only_points/0` mask to guarantee no overlap.

**Step 4.5: TDD cycle, commit**

---

## Task 5: Oban queue + cron

**Files:**
- Modify: `config/config.exs`, `config/runtime.exs`

**Step 5.1: Add `hrdps` queue**

```elixir
queues: [
  # ... existing
  hrdps: 1  # HRDPS fetch is sequential like HRRR (memory pressure)
]
```

**Step 5.2: Add cron**

```elixir
# HRDPS runs 00/06/12/18Z. Aligned with RDPS cadence.
{"35 */6 * * *", Microwaveprop.Workers.HrdpsGridWorker}
```

Note: 35 past the hour, not 5, because PropagationGridWorker (HRRR) fires at 5 past. Stagger by 30 minutes so they don't compete for the DB and the wgrib2 subprocess slot.

**Step 5.3: `mix compile` sanity-check**

**Step 5.4: Commit**

---

## Task 6: Map overlay — extend prop map to cover Canada

**Files:**
- Modify: `lib/microwaveprop_web/live/map_live.ex` — `@initial_bounds`, timeline
- Modify: `assets/js/propagation_map_hook.ts` — heatmap rendering bounds
- Modify: `lib/microwaveprop/propagation/grid.ex` — export `north_america_points/0` that returns HRRR ∪ HRDPS

**Step 6.1: Write failing LiveView test**

```elixir
test "/map renders scores for a Canadian grid cell (53N, -113W)" do
  {:ok, view, html} = live(conn, ~p"/map")
  # after PropagationGridWorker (HRRR) and HrdpsGridWorker (HRDPS) have both written scores
  assert html =~ "53.0"  # or whatever assertion proves the HRDPS cell made it to the client
end
```

**Step 6.2: Widen `@initial_bounds` in map_live.ex**

From the CONUS-only `(23.5, 49.5, -124.5, -66.9)` to `(23.5, 83.0, -141.0, -52.0)`. Zoom out default to show all of North America.

**Step 6.3: Verify front-end renders Canadian points**

```bash
mix phx.server
# Open http://localhost:4000/map
# Confirm colored cells north of 49N after the first HRDPS run completes
```

**Step 6.4: Commit**

---

## Task 7: Terrain + QSO path analysis north of the border

**Files:**
- Modify: `lib/microwaveprop/terrain/terrain_analysis.ex` (if needed)
- Modify: `lib/microwaveprop/workers/terrain_profile_worker.ex`
- Possibly: mount SRTM data covering Canada onto the production pods

**Step 7.1: Check SRTM coverage**

SRTM1 covers 60°S–60°N. Canadian radiosondes and stations above 60°N (Yellowknife, Whitehorse, Inuvik, Resolute) have NO SRTM coverage. The terrain analysis pipeline needs a fallback — likely CDEM (Canadian Digital Elevation Model) or ASTER GDEM.

**Decision gate with your human partner**: Does the feature need to work above 60°N? If yes, this is a multi-day sub-project (Task 7 becomes a whole new plan). If no — scope to 42–60°N and explicitly document that Inuvik paths won't get terrain analysis.

**Step 7.2 onward**: Out of scope for this plan. Write a follow-up plan for Arctic terrain if your human partner wants coverage.

**Step 7.3: Commit the scope decision to `docs/plans/` or a memory file**

---

## Task 8: Freshness monitoring + observability

**Files:**
- Modify: `lib/microwaveprop/propagation/freshness_monitor.ex`
- Modify: `lib/microwaveprop_web/live/status_live.ex`

**Step 8.1: Extend FreshnessMonitor to track HRDPS run time**

Add a `hrdps_last_run_at` field to the monitor state, surface it on `/status`.

**Step 8.2: Alert when HRDPS run_time is > 4 hours stale**

HRDPS runs every 6 hours, publishes ~90 min after, so a healthy pipeline never exceeds ~2.5 hours stale. 4 hours = something broken.

**Step 8.3: Commit**

---

## Task 9: Precommit, docs, PR

**Step 9.1: `mix precommit`**

Must show 0 failures.

**Step 9.2: Update `CLAUDE.md`**

- Add HRDPS to `### Key Data Sources`
- Add `HrdpsGridWorker` + `HrdpsFetchWorker` to the background queue table
- Add a note on the CONUS → North America scope change

**Step 9.3: Update `memory/reference_hrdps_canada.md`**

Mark the plan as executed — include the final ingestion rate, queue config, and any projection gotchas that surfaced.

**Step 9.4: Commit, push to `github` and `origin`, let Flux deploy**

Do NOT push to `dokku`.

---

## Known risks + gotchas

- **Projection**: HRDPS rotated lat/lon is the #1 source of bugs. Burn a day in Task 1.6 validating a dozen points against UWYO or ASOS surface obs before moving on. A 10 km projection offset is invisible on the propagation map but destroys terrain-path analysis.
- **Memory pressure**: The existing HRRR worker hits `:system_memory_high_watermark` alarms at ~4 GB. HRDPS grid is 3× larger in cell count. Plan for 8 GB per pod for the prop namespace, or offload HRDPS fetch to a dedicated node via Kubernetes node affinity.
- **Publish time**: CMC's publish delay can be erratic — 90 min on good days, 3+ hours under load. Build retry-with-backoff into `HrdpsGridWorker.perform/1` so the cron doesn't fire, find no data, and dead-letter immediately.
- **Arctic terrain**: SRTM stops at 60°N. Accept or fix — don't silently serve wrong diffraction losses to stations in Yellowknife.
- **HRRR/HRDPS overlap**: The two models disagree on the US/Canadian border. Pick a rule (prefer HRRR south of 49°N, prefer HRDPS north) and encode it in `Grid.point_source/2`.
- **Scope creep**: This plan is already ~5 days of work. Resist adding "while we're at it, let's also fix the native-level ducting for rotated grids" side-quests. Ship the minimum viable Canadian prop grid first.

## Rough effort estimate

| Task | Estimate |
|---|---|
| Pre-flight research | 2 hours |
| Task 1: HrdpsClient | 1 day (projection debug dominates) |
| Task 2: HrdpsNativeClient | 0.5 day (or skipped if no native) |
| Task 3: hrdps_profiles schema | 2 hours |
| Task 4: HrdpsGridWorker | 1 day |
| Task 5: Cron wiring | 1 hour |
| Task 6: Map overlay | 0.5 day |
| Task 7: Terrain coverage decision | 2 hours (punt if Arctic gets out-of-scope) |
| Task 8: Freshness monitoring | 2 hours |
| Task 9: Precommit + docs | 2 hours |
| **Total** | **~5 days** |

Compare to RDPS plan (~2 days) and the UWYO Canadian soundings (~4 hours, already done). Do HRDPS only when the RDPS 10 km coverage feels limiting, not before.
