prop/docs/plans/2026-04-13-rdps-vertical-profiles.md

RDPS Vertical Profiles (Canadian Ducting Outside HRRR Footprint) Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Extend the refractivity/ducting analysis beyond HRRR's Lambert CONUS footprint by ingesting ECCC's Regional Deterministic Prediction System (RDPS) vertical profiles. This gives us forecast ducting across all of Canada + northern-tier US gaps without the full HRDPS ingestion complexity.

Architecture: RDPS publishes a dedicated "vertical profiles" subset at 10 km resolution, 84h forecast, 4×/day. The files contain pressure-level T/Td/height — exactly the shape SoundingParams.derive/1 + Propagation.Duct.analyze/1 already consume. Fetch one forecast cycle per run, decode, store as time-indexed profile rows in a new rdps_profiles table (parallel to hrrr_profiles but on a 10 km grid), and surface it through a new /weather-ca map or extend the existing weather map timeline to "fall back to RDPS outside the HRRR footprint".

Tech Stack: Elixir 1.15, Req (HTTP), Nx optional, Postgres, Oban, wgrib2 (already installed for HRRR native duct grid)

---

Pre-flight: URL and format research

Before writing any code, confirm the RDPS vertical-profiles format. MSC docs are thin; the cheapest way is to hit the datamart directly.

Files to touch: None — this is read-only research.

Step R.1: Find today's run directory

Run:

curl -s https://dd.weather.gc.ca/model_gem_regional/10km/grib2/00/000/ | grep -oE 'href="[^"]+\.grib2"' | head -20

Expected: file list including a *vertical*profile*.grib2 or similar naming.

Step R.2: Confirm vertical-profiles subdirectory

Actual RDPS directory on dd.weather.gc.ca is /model_gem_regional/10km/grib2/{HH}/{FFF}/ — where HH is 00/06/12/18 and FFF is forecast hour. The vertical-profile subset may live under a different path; check:

curl -s https://dd.weather.gc.ca/ensemble/rdps/grib2/ 2>/dev/null
curl -s https://dd.weather.gc.ca/20260413/WXO-DD/model_gem_regional/ 2>/dev/null | grep -oE 'href="[^"]+/?'

If no dedicated vertical-profiles subset exists, fall back to pulling TMP/DEPR/HGT at all 31 pressure levels of the regular RDPS pressure-level files (same approach HrrrClient.fetch_grid/3 uses for HRRR).

Step R.3: Download one sample file and inspect with wgrib2

wgrib2 /tmp/sample.grib2 -var -v | head -30

Confirm it contains TMP, DEPR (or SPFH), HGT at 1000, 925, 850, 700, 500, 400, 300, 250, 200 mb (minimum).

Decision after R.3: If RDPS ships a ready-made vertical-profiles product with structured per-cell levels, continue with Task 1. If it's just the full pressure-level grid (like HRRR), this plan is still viable but the first task becomes "adapt HrrrClient pressure-level extraction to RDPS rotated lat/lon projection" — still simpler than full HRDPS.

---

Task 1: New RdpsClient module for GRIB2 fetch + decode

Files:

• Create: lib/microwaveprop/weather/rdps_client.ex
• Test: test/microwaveprop/weather/rdps_client_test.exs
• Fixture: test/support/fixtures/rdps_sample.grib2 (capture one real ~50 MB file via curl, commit it)

Step 1.1: Write failing test for URL builder

test "builds the correct datamart URL for a run/forecast hour/pressure level" do
  url = RdpsClient.grib_url(~D[2026-04-13], 12, 6, :tmp, 850)
  assert url =~ "/20260413/WXO-DD/model_gem_regional/10km/grib2/12/006/"
  assert url =~ "TMP_ISBL_850"
end

Step 1.2: Verify test fails

mix test test/microwaveprop/weather/rdps_client_test.exs -v

Expected: FAIL with UndefinedFunctionError.

Step 1.3: Implement grib_url/5

Match the actual dd.weather.gc.ca pattern discovered in pre-flight research. Copy the index-file + byte-range download pattern from HrrrClient.fetch_product/4 — RDPS GRIB2 files also ship .idx sidecars.

Step 1.4: Run, confirm PASS

Step 1.5: Write failing test for fetch_cell_profile/3

Given a lat/lon and run time, return %{"pres", "hght", "tmpc", "dwpc"} maps for all standard pressure levels, matching the shape SoundingParams.derive/1 expects. Use the committed fixture GRIB2 via Req.Test stub or directly from disk.

Step 1.6: Implement fetch_cell_profile/3 using wgrib2 subprocess

Reuse Microwaveprop.Weather.Grib2.Wgrib2.extract_points/3 — the module already wraps wgrib2 for HRRR extraction and supports arbitrary GRIB2 files.

Key nuance — rotated lat/lon projection: RDPS uses rotated_latitude_longitude, not Lambert conformal. wgrib2 handles this internally for -lon, -match ":(TMP|DEPR|HGT):(1000 mb|925 mb|850 mb|700 mb|500 mb|400 mb|300 mb|250 mb|200 mb)" extraction — verify by comparing a known point against UWYO sounding for Winnipeg/YWG.

Step 1.7: Commit

git add lib/microwaveprop/weather/rdps_client.ex test/microwaveprop/weather/rdps_client_test.exs test/support/fixtures/rdps_sample.grib2
git commit -m "Add RdpsClient for Canadian pressure-level weather fetch"

---

Task 2: rdps_profiles schema + migration

Files:

• Create: priv/repo/migrations/{timestamp}_create_rdps_profiles.exs (use mix ecto.gen.migration create_rdps_profiles)
• Create: lib/microwaveprop/weather/rdps_profile.ex
• Test: test/microwaveprop/weather/rdps_profile_test.exs

Step 2.1: Write failing changeset test

test "requires valid_time, lat, lon, and profile" do
  changeset = RdpsProfile.changeset(%RdpsProfile{}, %{})
  refute changeset.valid?
  assert %{valid_time: [_], lat: [_], lon: [_], profile: [_]} = errors_on(changeset)
end

Step 2.2: Generate migration

mix ecto.gen.migration create_rdps_profiles

Columns (mirror hrrr_profiles minus the native-duct fields RDPS doesn't provide):

• id binary_id
• valid_time utc_datetime
• run_time utc_datetime
• lat float
• lon float
• profile jsonb (array of pressure-level maps)
• surface_pressure_mb, surface_temp_c, surface_dewpoint_c floats
• surface_refractivity, min_refractivity_gradient floats
• ducting_detected boolean, duct_characteristics jsonb
• pwat_mm, hpbl_m floats (if RDPS publishes them; otherwise null)
• unique index on (lat, lon, valid_time)
• timestamps

Step 2.3: Implement schema module

Copy shape from lib/microwaveprop/weather/hrrr_profile.ex, strip anything RDPS doesn't supply.

Step 2.4: Run migration against dev DB and verify

mix ecto.migrate
psql -h localhost -d microwaveprop_dev -c "\d rdps_profiles"

Step 2.5: Run schema test, verify PASS

Step 2.6: Commit

---

Task 3: RdpsFetchWorker — one cell per job, queue-limited

Files:

• Create: lib/microwaveprop/workers/rdps_fetch_worker.ex
• Test: test/microwaveprop/workers/rdps_fetch_worker_test.exs

Step 3.1: Write failing test

Worker takes %{"lat" => lat, "lon" => lon, "valid_time" => iso}, stubs RdpsClient.fetch_cell_profile/3, asserts an RdpsProfile row is upserted with derived SoundingParams fields filled in.

Step 3.2: Implement worker

• use Oban.Worker, queue: :rdps, max_attempts: 3, unique: ...
• perform/1 calls RdpsClient.fetch_cell_profile, runs SoundingParams.derive/1 on the returned profile, upserts via Weather.upsert_rdps_profile/1 (add this function).

Step 3.3: Verify test passes

Step 3.4: Commit

---

Task 4: RdpsGridWorker — hourly-ish batch driver

Files:

• Create: lib/microwaveprop/workers/rdps_grid_worker.ex
• Test: test/microwaveprop/workers/rdps_grid_worker_test.exs

Step 4.1: Write failing test

Test that perform/1 enqueues one RdpsFetchWorker per grid point in a Canadian bounding box (south, north, west, east) = (42, 83, -141, -52) sampled at 0.5° (≈1500 points).

Step 4.2: Implement

Mirror PropagationGridWorker.perform/1 — compute run_time (2 hours ago nearest RDPS synoptic hour 00/06/12/18), loop over Canadian grid, enqueue. Don't couple to HRRR; RDPS runs on a different cadence (every 6 hours vs HRRR every hour) and this keeps the two pipelines independent.

Step 4.3: Verify

Step 4.4: Commit

---

Task 5: Oban cron wiring + queue limits

Files:

• Modify: config/config.exs (dev + shared)
• Modify: config/runtime.exs (prod Oban config)

Step 5.1: Add rdps queue

queues: [
  # ... existing
  rdps: 2
]

Dev: 2. Prod: 2 per pod × 3 pods = 6 workers, plus a global_limit if MSC starts rate-limiting.

Step 5.2: Add cron

# RDPS runs 00/06/12/18Z. Data typically publishes ~90 minutes after synoptic.
{"30 */6 * * *", Microwaveprop.Workers.RdpsGridWorker}

Step 5.3: mix compile sanity-check

Step 5.4: Commit

---

Task 6: Score integration — fall back to RDPS outside HRRR footprint

Files:

• Modify: lib/microwaveprop/propagation.ex:~244 (scores_at/3)
• Modify: lib/microwaveprop/workers/propagation_grid_worker.ex:~74 (process_forecast_hour/4)
• Test: test/microwaveprop/propagation/rdps_fallback_test.exs

Step 6.1: Write failing test

For a lat/lon inside HRRR's CONUS footprint, Propagation.score_grid_point/4 uses the HRRR profile. For a lat/lon in northern Canada (65N, -90W), it falls back to the RdpsProfile for the same valid_time.

Step 6.2: Add rdps_profile_at/3 helper in Weather

def rdps_profile_at(lat, lon, valid_time) do
  # nearest-neighbor lookup in rdps_profiles
end

Step 6.3: Modify process_forecast_hour/4 to also score RDPS points

Two-pass: first the HRRR CONUS grid as today, then a second pass over RDPS cells that fall outside HRRR's Lambert domain. Extend grid_data to include both sources, mark each point with source: :hrrr or source: :rdps in factors.

Step 6.4: Verify tests + precommit

Step 6.5: Commit

---

Task 7: Map overlay — extend CONUS prop map north

Files:

• Modify: lib/microwaveprop/propagation/grid.ex — conus_points/0 → north_america_points/0 or similar
• Modify: assets/js/propagation_map_hook.ts — extend bounds, label RDPS-sourced cells

Step 7.1: Decide UX

Pick one:

• Option A: Silent extension — the existing /map just shows scores north of 49N without visual distinction.
• Option B: Dashed line on the map at the HRRR/RDPS boundary, tooltip notes data source.

Default to Option A unless your human partner prefers B.

Step 7.2: Extend grid bounds

Change the conus_points/0 generator to include cells in (42, 83, -141, -52) for RDPS. Keep CONUS cells on the current 0.125° grid, add RDPS cells at 0.5° (coarser).

Step 7.3: Verify /map renders Canadian cells

Manual: mix phx.server, open http://localhost:4000/map, zoom to Ontario/Quebec, confirm colored cells appear.

Step 7.4: Commit

---

Task 8: Precommit + PR

Step 8.1: mix precommit

Must show 0 failures.

Step 8.2: Update CLAUDE.md with the new pipeline

Add an RdpsFetchWorker/RdpsGridWorker row to the Background Job Queues table and a bullet under ## Key Data Sources for RDPS.

Step 8.3: Update memory/reference_hrdps_canada.md

Add a note: "RDPS vertical profiles (10 km) ingestion landed YYYY-MM-DD — gives Canadian ducting without the full HRDPS rewrite."

Step 8.4: Commit and push to both github and origin remotes

Do NOT push to dokku. Let Flux roll out the new image from the CI build.

---

Notes / risks

• Rotated lat/lon projection: wgrib2 handles it correctly for point extraction but caution needed if the team ever decides to store full-grid binary rasters.
• File size: RDPS pressure-level files are ~50 MB each. At 4 runs/day × ~10 forecast hours × ~5 variables × ~10 pressure levels, total download ≈ 2 GB/day. Not a problem on a home lab with skippy.w5isp.com cache, but worth caching aggressively.
• Data latency: RDPS publishes ~90 minutes after synoptic; the cron is already aligned.
• Fallback when RDPS is unavailable: scores_at/3 should gracefully degrade to HRRR-only if rdps_profiles is empty for the requested valid_time.