prop/docs/plans/2026-04-13-rdps-vertical-profiles.md

# RDPS Vertical Profiles (Canadian Ducting Outside HRRR Footprint) Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Extend the refractivity/ducting analysis beyond HRRR's Lambert CONUS footprint by ingesting ECCC's Regional Deterministic Prediction System (RDPS) vertical profiles. This gives us forecast ducting across all of Canada + northern-tier US gaps without the full HRDPS ingestion complexity.

**Architecture:** RDPS publishes a dedicated "vertical profiles" subset at 10 km resolution, 84h forecast, 4×/day. The files contain pressure-level T/Td/height — exactly the shape `SoundingParams.derive/1` + `Propagation.Duct.analyze/1` already consume. Fetch one forecast cycle per run, decode, store as time-indexed profile rows in a new `rdps_profiles` table (parallel to `hrrr_profiles` but on a 10 km grid), and surface it through a new `/weather-ca` map or extend the existing weather map timeline to "fall back to RDPS outside the HRRR footprint".

**Tech Stack:** Elixir 1.15, Req (HTTP), Nx optional, Postgres, Oban, wgrib2 (already installed for HRRR native duct grid)

---

## Pre-flight: URL and format research

Before writing any code, confirm the RDPS vertical-profiles format. MSC docs are thin; the cheapest way is to hit the datamart directly.

**Files to touch:** None — this is read-only research.

**Step R.1: Find today's run directory**

Run:

```bash
curl -s https://dd.weather.gc.ca/model_gem_regional/10km/grib2/00/000/ | grep -oE 'href="[^"]+\.grib2"' | head -20
```

Expected: file list including a `*vertical*profile*.grib2` or similar naming.

**Step R.2: Confirm vertical-profiles subdirectory**

Actual RDPS directory on dd.weather.gc.ca is `/model_gem_regional/10km/grib2/{HH}/{FFF}/` — where HH is 00/06/12/18 and FFF is forecast hour. The vertical-profile subset may live under a different path; check:

```bash
curl -s https://dd.weather.gc.ca/ensemble/rdps/grib2/ 2>/dev/null
curl -s https://dd.weather.gc.ca/20260413/WXO-DD/model_gem_regional/ 2>/dev/null | grep -oE 'href="[^"]+/?'
```

If no dedicated vertical-profiles subset exists, fall back to pulling TMP/DEPR/HGT at all 31 pressure levels of the regular RDPS pressure-level files (same approach `HrrrClient.fetch_grid/3` uses for HRRR).

**Step R.3: Download one sample file and inspect with wgrib2**

```bash
wgrib2 /tmp/sample.grib2 -var -v | head -30
```

Confirm it contains TMP, DEPR (or SPFH), HGT at 1000, 925, 850, 700, 500, 400, 300, 250, 200 mb (minimum).

**Decision after R.3:** If RDPS ships a ready-made vertical-profiles product with structured per-cell levels, continue with Task 1. If it's just the full pressure-level grid (like HRRR), this plan is still viable but the first task becomes "adapt HrrrClient pressure-level extraction to RDPS rotated lat/lon projection" — still simpler than full HRDPS.

---

## Task 1: New `RdpsClient` module for GRIB2 fetch + decode

**Files:**
- Create: `lib/microwaveprop/weather/rdps_client.ex`
- Test: `test/microwaveprop/weather/rdps_client_test.exs`
- Fixture: `test/support/fixtures/rdps_sample.grib2` (capture one real ~50 MB file via curl, commit it)

**Step 1.1: Write failing test for URL builder**

```elixir
test "builds the correct datamart URL for a run/forecast hour/pressure level" do
  url = RdpsClient.grib_url(~D[2026-04-13], 12, 6, :tmp, 850)
  assert url =~ "/20260413/WXO-DD/model_gem_regional/10km/grib2/12/006/"
  assert url =~ "TMP_ISBL_850"
end
```

**Step 1.2: Verify test fails**

```
mix test test/microwaveprop/weather/rdps_client_test.exs -v
```

Expected: FAIL with UndefinedFunctionError.

**Step 1.3: Implement `grib_url/5`**

Match the actual dd.weather.gc.ca pattern discovered in pre-flight research. Copy the index-file + byte-range download pattern from `HrrrClient.fetch_product/4` — RDPS GRIB2 files also ship `.idx` sidecars.

**Step 1.4: Run, confirm PASS**

**Step 1.5: Write failing test for `fetch_cell_profile/3`**

Given a lat/lon and run time, return `%{"pres", "hght", "tmpc", "dwpc"}` maps for all standard pressure levels, matching the shape `SoundingParams.derive/1` expects. Use the committed fixture GRIB2 via `Req.Test` stub or directly from disk.

**Step 1.6: Implement fetch_cell_profile/3 using wgrib2 subprocess**

Reuse `Microwaveprop.Weather.Grib2.Wgrib2.extract_points/3` — the module already wraps wgrib2 for HRRR extraction and supports arbitrary GRIB2 files.

**Key nuance — rotated lat/lon projection:** RDPS uses rotated_latitude_longitude, not Lambert conformal. wgrib2 handles this internally for `-lon`, `-match ":(TMP|DEPR|HGT):(1000 mb|925 mb|850 mb|700 mb|500 mb|400 mb|300 mb|250 mb|200 mb)"` extraction — verify by comparing a known point against UWYO sounding for Winnipeg/YWG.

**Step 1.7: Commit**

```bash
git add lib/microwaveprop/weather/rdps_client.ex test/microwaveprop/weather/rdps_client_test.exs test/support/fixtures/rdps_sample.grib2
git commit -m "Add RdpsClient for Canadian pressure-level weather fetch"
```

---

## Task 2: `rdps_profiles` schema + migration

**Files:**
- Create: `priv/repo/migrations/{timestamp}_create_rdps_profiles.exs` (use `mix ecto.gen.migration create_rdps_profiles`)
- Create: `lib/microwaveprop/weather/rdps_profile.ex`
- Test: `test/microwaveprop/weather/rdps_profile_test.exs`

**Step 2.1: Write failing changeset test**

```elixir
test "requires valid_time, lat, lon, and profile" do
  changeset = RdpsProfile.changeset(%RdpsProfile{}, %{})
  refute changeset.valid?
  assert %{valid_time: [_], lat: [_], lon: [_], profile: [_]} = errors_on(changeset)
end
```

**Step 2.2: Generate migration**

```bash
mix ecto.gen.migration create_rdps_profiles
```

Columns (mirror `hrrr_profiles` minus the native-duct fields RDPS doesn't provide):
- `id` binary_id
- `valid_time` utc_datetime
- `run_time` utc_datetime
- `lat` float
- `lon` float
- `profile` jsonb (array of pressure-level maps)
- `surface_pressure_mb`, `surface_temp_c`, `surface_dewpoint_c` floats
- `surface_refractivity`, `min_refractivity_gradient` floats
- `ducting_detected` boolean, `duct_characteristics` jsonb
- `pwat_mm`, `hpbl_m` floats (if RDPS publishes them; otherwise null)
- unique index on `(lat, lon, valid_time)`
- timestamps

**Step 2.3: Implement schema module**

Copy shape from `lib/microwaveprop/weather/hrrr_profile.ex`, strip anything RDPS doesn't supply.

**Step 2.4: Run migration against dev DB and verify**

```bash
mix ecto.migrate
psql -h localhost -d prop_dev -c "\d rdps_profiles"
```

**Step 2.5: Run schema test, verify PASS**

**Step 2.6: Commit**

---

## Task 3: `RdpsFetchWorker` — one cell per job, queue-limited

**Files:**
- Create: `lib/microwaveprop/workers/rdps_fetch_worker.ex`
- Test: `test/microwaveprop/workers/rdps_fetch_worker_test.exs`

**Step 3.1: Write failing test**

Worker takes `%{"lat" => lat, "lon" => lon, "valid_time" => iso}`, stubs `RdpsClient.fetch_cell_profile/3`, asserts an `RdpsProfile` row is upserted with derived SoundingParams fields filled in.

**Step 3.2: Implement worker**

- `use Oban.Worker, queue: :rdps, max_attempts: 3, unique: ...`
- `perform/1` calls `RdpsClient.fetch_cell_profile`, runs `SoundingParams.derive/1` on the returned profile, upserts via `Weather.upsert_rdps_profile/1` (add this function).

**Step 3.3: Verify test passes**

**Step 3.4: Commit**

---

## Task 4: `RdpsGridWorker` — hourly-ish batch driver

**Files:**
- Create: `lib/microwaveprop/workers/rdps_grid_worker.ex`
- Test: `test/microwaveprop/workers/rdps_grid_worker_test.exs`

**Step 4.1: Write failing test**

Test that `perform/1` enqueues one `RdpsFetchWorker` per grid point in a Canadian bounding box `(south, north, west, east) = (42, 83, -141, -52)` sampled at 0.5° (≈1500 points).

**Step 4.2: Implement**

Mirror `PropagationGridWorker.perform/1` — compute run_time (2 hours ago nearest RDPS synoptic hour 00/06/12/18), loop over Canadian grid, enqueue. Don't couple to HRRR; RDPS runs on a different cadence (every 6 hours vs HRRR every hour) and this keeps the two pipelines independent.

**Step 4.3: Verify**

**Step 4.4: Commit**

---

## Task 5: Oban cron wiring + queue limits

**Files:**
- Modify: `config/config.exs` (dev + shared)
- Modify: `config/runtime.exs` (prod Oban config)

**Step 5.1: Add `rdps` queue**

```elixir
queues: [
  # ... existing
  rdps: 2
]
```

Dev: 2. Prod: 2 per pod × 3 pods = 6 workers, plus a global_limit if MSC starts rate-limiting.

**Step 5.2: Add cron**

```elixir
# RDPS runs 00/06/12/18Z. Data typically publishes ~90 minutes after synoptic.
{"30 */6 * * *", Microwaveprop.Workers.RdpsGridWorker}
```

**Step 5.3: `mix compile` sanity-check**

**Step 5.4: Commit**

---

## Task 6: Score integration — fall back to RDPS outside HRRR footprint

**Files:**
- Modify: `lib/microwaveprop/propagation.ex:~244` (`scores_at/3`)
- Modify: `lib/microwaveprop/workers/propagation_grid_worker.ex:~74` (`process_forecast_hour/4`)
- Test: `test/microwaveprop/propagation/rdps_fallback_test.exs`

**Step 6.1: Write failing test**

For a lat/lon inside HRRR's CONUS footprint, `Propagation.score_grid_point/4` uses the HRRR profile. For a lat/lon in northern Canada (65N, -90W), it falls back to the `RdpsProfile` for the same valid_time.

**Step 6.2: Add `rdps_profile_at/3` helper in `Weather`**

```elixir
def rdps_profile_at(lat, lon, valid_time) do
  # nearest-neighbor lookup in rdps_profiles
end
```

**Step 6.3: Modify `process_forecast_hour/4` to also score RDPS points**

Two-pass: first the HRRR CONUS grid as today, then a second pass over RDPS cells that fall outside HRRR's Lambert domain. Extend `grid_data` to include both sources, mark each point with `source: :hrrr` or `source: :rdps` in factors.

**Step 6.4: Verify tests + precommit**

**Step 6.5: Commit**

---

## Task 7: Map overlay — extend CONUS prop map north

**Files:**
- Modify: `lib/microwaveprop/propagation/grid.ex` — `conus_points/0` → `north_america_points/0` or similar
- Modify: `assets/js/propagation_map_hook.ts` — extend bounds, label RDPS-sourced cells

**Step 7.1: Decide UX**

Pick one:
- **Option A**: Silent extension — the existing /map just shows scores north of 49N without visual distinction.
- **Option B**: Dashed line on the map at the HRRR/RDPS boundary, tooltip notes data source.

Default to Option A unless your human partner prefers B.

**Step 7.2: Extend grid bounds**

Change the `conus_points/0` generator to include cells in `(42, 83, -141, -52)` for RDPS. Keep CONUS cells on the current 0.125° grid, add RDPS cells at 0.5° (coarser).

**Step 7.3: Verify `/map` renders Canadian cells**

Manual: `mix phx.server`, open http://localhost:4000/map, zoom to Ontario/Quebec, confirm colored cells appear.

**Step 7.4: Commit**

---

## Task 8: Precommit + PR

**Step 8.1: `mix precommit`**

Must show 0 failures.

**Step 8.2: Update `CLAUDE.md` with the new pipeline**

Add an `RdpsFetchWorker`/`RdpsGridWorker` row to the Background Job Queues table and a bullet under `## Key Data Sources` for RDPS.

**Step 8.3: Update `memory/reference_hrdps_canada.md`**

Add a note: "RDPS vertical profiles (10 km) ingestion landed YYYY-MM-DD — gives Canadian ducting without the full HRDPS rewrite."

**Step 8.4: Commit and push to both `github` and `origin` remotes**

Do NOT push to `dokku`. Let Flux roll out the new image from the CI build.

---

## Notes / risks

- **Rotated lat/lon projection**: wgrib2 handles it correctly for point extraction but caution needed if the team ever decides to store full-grid binary rasters.
- **File size**: RDPS pressure-level files are ~50 MB each. At 4 runs/day × ~10 forecast hours × ~5 variables × ~10 pressure levels, total download ≈ 2 GB/day. Not a problem on a home lab with skippy.w5isp.com cache, but worth caching aggressively.
- **Data latency**: RDPS publishes ~90 minutes after synoptic; the cron is already aligned.
- **Fallback when RDPS is unavailable**: `scores_at/3` should gracefully degrade to HRRR-only if `rdps_profiles` is empty for the requested `valid_time`.