prop/docs/plans/2026-04-13-hrdps-canadian-prop-grid.md

HRDPS Canadian Propagation Grid Implementation Plan (SUPERSEDED)

SUPERSEDED 2026-04-29 by docs/plans/2026-04-29-hrdps-canadian-prop-grid.md. A live probe of MSC Datamart on 2026-04-29 found that ECCC reorganized the URL structure (the /model_hrdps/ flat root in the pre-flight steps below 404s now) and that HRDPS publishes one variable per GRIB2 file rather than HRRR's bundled wrfsfcf/wrfprsf format — so the Stage 1 idx + byte-range fetch shape this plan inherits from HrrrClient does not apply. Use the 2026-04-29 plan instead.

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Extend the propagation scoring grid to cover all of Canada at HRDPS's 2.5 km resolution — the Canadian analog to HRRR. This lights up the /map for VE stations, gives terrain-diffraction analysis for VE QSOs, and unlocks cross-border path analysis for KP/W7-to-VE7 tropo paths.

Architecture: Build a parallel NWP ingestion pipeline (HrdpsClient + HrdpsGridWorker + hrdps_profiles table) that matches the HRRR structure one-for-one, with three major differences: rotated lat/lon projection, CMC file naming (CMC_hrdps_continental_*_ps2.5km_*_{FFF}.grib2), and a different native-level grid spec for duct analysis. A new propagation_scores region tag distinguishes HRRR-sourced vs HRDPS-sourced cells so the front-end can render them uniformly.

Tech Stack: Elixir 1.15, Req, Nx (optional for duct analysis), Postgres, Oban, wgrib2, proj (for projection conversions if needed)

Prerequisite: Task 6 of the RDPS plan (2026-04-13-rdps-vertical-profiles.md) — or at minimum a working pattern for "score grid_data from a non-HRRR source". If you're starting HRDPS fresh, Task 6 below stands alone but will duplicate some work from the RDPS plan.

---

Pre-flight research (1–2 hours)

Step R.1: Confirm HRDPS datamart URL pattern

curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/ 2>&1 | head -30
curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/00/000/ 2>&1 | head -30

Expected: file list with entries like CMC_hrdps_continental_TMP_TGL_2_ps2.5km_{YYYYMMDDHH}_P{FFF}-00.grib2.

Step R.2: Download one sample + inspect with wgrib2

wgrib2 CMC_hrdps_continental_TMP_TGL_2_ps2.5km_2026041300_P000-00.grib2 -grid -v

Expected: confirms grid is rotated lat/lon, gives dimensions (typically ~2500×1800), and lists all variables.

Step R.3: Confirm pressure-level files exist

HRDPS publishes surface + pressure-level files separately (like HRRR wrfsfc vs wrfprs). Find the pressure-level variant:

curl -s https://dd.weather.gc.ca/model_hrdps/continental/2.5km/00/000/ | grep -i 'ISBL'

If separate files per pressure level, the ingestion must fan out like HrrrClient.pressure_messages/0. If a single combined file, one download gets everything.

Step R.4: Decide projection strategy

wgrib2 can extract points from rotated lat/lon directly with -lon LON LAT. For extracting the full grid as a flat array, you need proj or a wgrib2 -new_grid reprojection. Recommended: never reproject — always extract point-by-point for the grid points we care about. Slower than bulk extraction but avoids projection bugs.

Step R.5: Confirm run frequency and forecast range

HRDPS: 4× daily (00/06/12/18Z), forecasts to 48h. Verify via datamart. This is half HRRR's run frequency (hourly) and 2.7× longer (48h vs 18h).

Decision gate: After R.5, confirm with your human partner which forecast range to ingest. Recommendation: start with f00–f24 to match the "current + 1 day" use case. Extending to f48 doubles storage and compute and probably isn't worth it for amateur radio planning.

---

Task 1: HrdpsClient — surface + pressure fetch

Files:

• Create: lib/microwaveprop/weather/hrdps_client.ex
• Test: test/microwaveprop/weather/hrdps_client_test.exs
• Fixture: test/support/fixtures/hrdps_sample_sfc.grib2 + test/support/fixtures/hrdps_sample_prs.grib2

Step 1.1: Write failing URL builder test

test "builds correct CMC datamart URL for surface temp" do
  url = HrdpsClient.grib_url(~D[2026-04-13], 12, 6, :surface, :tmp)
  assert url =~ "/model_hrdps/continental/2.5km/12/006/CMC_hrdps_continental_TMP_TGL_2_ps2.5km_2026041312_P006-00.grib2"
end

Adjust the format string after pre-flight research confirms the exact naming.

Step 1.2–1.4: Write → fail → implement → pass

Reuse HrrrClient.parse_idx/1, HrrrClient.byte_ranges_for_messages/2, and HrrrClient.download_grib_ranges/2 — they're format-agnostic. Only the URL pattern and the message set differ.

Step 1.5: Implement fetch_grid/3

@spec fetch_grid([{float(), float()}], DateTime.t(), keyword()) ::
        {:ok, %{{float(), float()} => map()}} | {:error, term()}
def fetch_grid(points, run_time, opts \\ [])

Same signature as HrrrClient.fetch_grid/3 so PropagationGridWorker.process_forecast_hour/4 can call either one polymorphically in Task 6.

Step 1.6: Nail the projection

Write a focused test: pick a known Canadian airport (CYYR = Goose Bay, 53.32°N, 60.42°W) and verify the HRDPS profile at that point matches the UWYO radiosonde for the same valid_time within a reasonable tolerance (surface temp ±3°C, dewpoint ±4°C). If they don't match, the projection is wrong and you need to debug wgrib2's -lon extraction vs the CMC rotated grid.

Step 1.7: Commit

---

Task 2: Native-level duct metrics via HrdpsNativeClient

Files:

• Create: lib/microwaveprop/weather/hrdps_native_client.ex
• Test: test/microwaveprop/weather/hrdps_native_client_test.exs

Step 2.1: Confirm HRDPS publishes native hybrid levels

HRDPS uses CMC's GEM dynamical core with hybrid sigma-pressure levels (62 of them). The datamart file naming for native levels is *_HY_* or within model_hrdps/continental/2.5km/native/ — confirm in pre-flight.

If native levels are NOT published: this task is impossible; skip it and fall back to pressure-level duct analysis (coarser — 31 levels vs 62 — but still works). The map's ducting detection quality will be slightly worse over VE airspace than over CONUS, which is acceptable for a first pass.

If native levels ARE published: mirror HrrrNativeClient.fetch_native_duct_grid/4. Use Microwaveprop.Propagation.Duct.analyze/1 which is projection-agnostic — it just needs a list of levels with temperature, pressure, and specific humidity.

Step 2.2–2.5: TDD cycle as above

Step 2.6: Commit

---

Task 3: hrdps_profiles table + schema

Files:

• Create: priv/repo/migrations/{timestamp}_create_hrdps_profiles.exs
• Create: lib/microwaveprop/weather/hrdps_profile.ex
• Test: test/microwaveprop/weather/hrdps_profile_test.exs

Step 3.1–3.4: Mirror hrrr_profiles exactly. Same columns, same indexes. The only reason to have a separate table is operational isolation — don't contaminate the 10M-row HRRR partition with 8M-row HRDPS rows, and let you drop/reprocess HRDPS without touching HRRR.

Use daily partitioning on valid_time just like hrrr_profiles.

Step 3.5: Partition-pruning migration

Add an Oban.Plugins.Cron entry for HrdpsPartitionCleanup similar to HrrrPartitionCleanup if one exists.

Step 3.6: Commit

---

Task 4: HrdpsGridWorker — hourly compute over Canadian grid

Files:

• Create: lib/microwaveprop/workers/hrdps_grid_worker.ex
• Test: test/microwaveprop/workers/hrdps_grid_worker_test.exs
• Modify: lib/microwaveprop/propagation/grid.ex (add canadian_points/0)

Step 4.1: Define the Canadian grid

At HRDPS 2.5 km resolution, Canada (42N-83N, -141 to -52W) has ~1.5M cells. Too many to score every hour. Instead:

• Sample at 0.125° (14 km) to match HRRR's output grid → ~100K points
• For the subset of points that are INSIDE HRRR's CONUS footprint, prefer HRRR; OUTSIDE, use HRDPS
• Store the "outside HRRR" mask as Grid.hrdps_only_points/0

Step 4.2: Worker perform/1

Mirror PropagationGridWorker.process_forecast_hour/4 but with HrdpsClient.fetch_grid/3 and HrdpsNativeClient.fetch_native_duct_grid/4.

Step 4.3: Scoring

Call Propagation.score_grid_point/4 — it's already projection-agnostic. It takes a profile map and valid_time and produces per-band scores. No changes needed to the scorer.

Step 4.4: Upsert into propagation_scores

grid_scores is already indexed by (lat, lon, valid_time, band_mhz). HRDPS and HRRR both write into the same table — the unique key prevents collisions because HRDPS cells have different (lat, lon) than HRRR cells at the shared 0.125° resolution IF you don't re-score HRRR's cells from HRDPS. Use the hrdps_only_points/0 mask to guarantee no overlap.

Step 4.5: TDD cycle, commit

---

Task 5: Oban queue + cron

Files:

• Modify: config/config.exs, config/runtime.exs

Step 5.1: Add hrdps queue

queues: [
  # ... existing
  hrdps: 1  # HRDPS fetch is sequential like HRRR (memory pressure)
]

Step 5.2: Add cron

# HRDPS runs 00/06/12/18Z. Aligned with RDPS cadence.
{"35 */6 * * *", Microwaveprop.Workers.HrdpsGridWorker}

Note: 35 past the hour, not 5, because PropagationGridWorker (HRRR) fires at 5 past. Stagger by 30 minutes so they don't compete for the DB and the wgrib2 subprocess slot.

Step 5.3: mix compile sanity-check

Step 5.4: Commit

---

Task 6: Map overlay — extend prop map to cover Canada

Files:

• Modify: lib/microwaveprop_web/live/map_live.ex — @initial_bounds, timeline
• Modify: assets/js/propagation_map_hook.ts — heatmap rendering bounds
• Modify: lib/microwaveprop/propagation/grid.ex — export north_america_points/0 that returns HRRR ∪ HRDPS

Step 6.1: Write failing LiveView test

test "/map renders scores for a Canadian grid cell (53N, -113W)" do
  {:ok, view, html} = live(conn, ~p"/map")
  # after PropagationGridWorker (HRRR) and HrdpsGridWorker (HRDPS) have both written scores
  assert html =~ "53.0"  # or whatever assertion proves the HRDPS cell made it to the client
end

Step 6.2: Widen @initial_bounds in map_live.ex

From the CONUS-only (23.5, 49.5, -124.5, -66.9) to (23.5, 83.0, -141.0, -52.0). Zoom out default to show all of North America.

Step 6.3: Verify front-end renders Canadian points

mix phx.server
# Open http://localhost:4000/map
# Confirm colored cells north of 49N after the first HRDPS run completes

Step 6.4: Commit

---

Task 7: Terrain + QSO path analysis north of the border

Files:

• Modify: lib/microwaveprop/terrain/terrain_analysis.ex (if needed)
• Modify: lib/microwaveprop/workers/terrain_profile_worker.ex
• Possibly: mount SRTM data covering Canada onto the production pods

Step 7.1: Check SRTM coverage

SRTM1 covers 60°S–60°N. Canadian radiosondes and stations above 60°N (Yellowknife, Whitehorse, Inuvik, Resolute) have NO SRTM coverage. The terrain analysis pipeline needs a fallback — likely CDEM (Canadian Digital Elevation Model) or ASTER GDEM.

Decision gate with your human partner: Does the feature need to work above 60°N? If yes, this is a multi-day sub-project (Task 7 becomes a whole new plan). If no — scope to 42–60°N and explicitly document that Inuvik paths won't get terrain analysis.

Step 7.2 onward: Out of scope for this plan. Write a follow-up plan for Arctic terrain if your human partner wants coverage.

Step 7.3: Commit the scope decision to docs/plans/ or a memory file

---

Task 8: Freshness monitoring + observability

Files:

• Modify: lib/microwaveprop/propagation/freshness_monitor.ex
• Modify: lib/microwaveprop_web/live/status_live.ex

Step 8.1: Extend FreshnessMonitor to track HRDPS run time

Add a hrdps_last_run_at field to the monitor state, surface it on /status.

Step 8.2: Alert when HRDPS run_time is > 4 hours stale

HRDPS runs every 6 hours, publishes ~90 min after, so a healthy pipeline never exceeds ~2.5 hours stale. 4 hours = something broken.

Step 8.3: Commit

---

Task 9: Precommit, docs, PR

Step 9.1: mix precommit

Must show 0 failures.

Step 9.2: Update CLAUDE.md

• Add HRDPS to ### Key Data Sources
• Add HrdpsGridWorker + HrdpsFetchWorker to the background queue table
• Add a note on the CONUS → North America scope change

Step 9.3: Update memory/reference_hrdps_canada.md

Mark the plan as executed — include the final ingestion rate, queue config, and any projection gotchas that surfaced.

Step 9.4: Commit, push to github and origin, let Flux deploy

Do NOT push to dokku.

---

Known risks + gotchas

• Projection: HRDPS rotated lat/lon is the #1 source of bugs. Burn a day in Task 1.6 validating a dozen points against UWYO or ASOS surface obs before moving on. A 10 km projection offset is invisible on the propagation map but destroys terrain-path analysis.
• Memory pressure: The existing HRRR worker hits :system_memory_high_watermark alarms at ~4 GB. HRDPS grid is 3× larger in cell count. Plan for 8 GB per pod for the prop namespace, or offload HRDPS fetch to a dedicated node via Kubernetes node affinity.
• Publish time: CMC's publish delay can be erratic — 90 min on good days, 3+ hours under load. Build retry-with-backoff into HrdpsGridWorker.perform/1 so the cron doesn't fire, find no data, and dead-letter immediately.
• Arctic terrain: SRTM stops at 60°N. Accept or fix — don't silently serve wrong diffraction losses to stations in Yellowknife.
• HRRR/HRDPS overlap: The two models disagree on the US/Canadian border. Pick a rule (prefer HRRR south of 49°N, prefer HRDPS north) and encode it in Grid.point_source/2.
• Scope creep: This plan is already ~5 days of work. Resist adding "while we're at it, let's also fix the native-level ducting for rotated grids" side-quests. Ship the minimum viable Canadian prop grid first.

Rough effort estimate

Task	Estimate
Pre-flight research	2 hours
Task 1: HrdpsClient	1 day (projection debug dominates)
Task 2: HrdpsNativeClient	0.5 day (or skipped if no native)
Task 3: hrdps_profiles schema	2 hours
Task 4: HrdpsGridWorker	1 day
Task 5: Cron wiring	1 hour
Task 6: Map overlay	0.5 day
Task 7: Terrain coverage decision	2 hours (punt if Arctic gets out-of-scope)
Task 8: Freshness monitoring	2 hours
Task 9: Precommit + docs	2 hours
Total	~5 days

Compare to RDPS plan (~2 days) and the UWYO Canadian soundings (~4 hours, already done). Do HRDPS only when the RDPS 10 km coverage feels limiting, not before.