# Extended-Horizon Propagation Forecasts — Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Extend propagation forecast horizon from HRRR's 18-hour cap to ~10 days, so users can plan VHF/microwave activity for upcoming contest weekends, DXpeditions, and band openings. Handoff is: HRRR drives 0–18 h (what we have today), a coarser global model drives 18–240 h, with the visible seam at ~18 hours being the point where resolution drops from 3 km to ~13 km.

**Architecture:** New ingestion pipeline for a global-model NWP source running alongside `HrrrFetchWorker` / `PropagationGridWorker`. Global-model profiles land in a separate binary-file store (`ScoresFileGlobal`) so the hourly HRRR chain isn't touched. The map's forecast timeline stretches from hour 0 to hour 240, with a visual band at the ~18 h boundary showing resolution drop.

**Tech Stack:** Existing (Oban, Ecto, Phoenix LiveView, ProfilesFile/ScoresFile binary storage, wgrib2). New: global-model GRIB2 client and field extractor. No new Python / netCDF dependencies — GFS and ECMWF HRES open-data are both GRIB2.

---

## Source comparison

| Attribute | HRRR (current) | GFS (proposed) | ECMWF IFS HRES |
|---|---|---|---|
| Horizon | 18 h | 384 h | 240 h |
| Cadence | Hourly runs, hourly output | 4 runs/day (00/06/12/18 UTC), 1 h output through f120, 3 h f120-f240, 12 h f240-f384 | 4 runs/day, 3 h output through f144, 6 h f144-f240 |
| Horizontal res | 3 km | 0.25° (~28 km) | 0.25° open-data; 0.1° subscription |
| Vertical | 50 hybrid sigma + 40 pressure levels | 31 pressure levels, 10 hPa top | 37 pressure levels |
| Latency | ~55 min after run start | ~3 h 20 min after run start | ~4-5 h after run start |
| Cost | Free, AWS S3 | Free, AWS S3 + NCEP | Free 0.25° (since 2024), via ECMWF open-data S3 |
| Surface PWAT, HPBL | Yes | PWAT yes, HPBL yes | PWAT yes, BLH yes |
| Refractivity gradient | Reconstructable from 13 levels | Reconstructable from pressure levels (coarser than HRRR's 13) | Reconstructable from 37 levels (comparable to HRRR pressure-level) |

**Recommendation: start with GFS**, add ECMWF in a follow-up if forecast skill warrants. Reasons:
- GFS open-data access is identical in shape to HRRR (byte-range GRIB2 on S3 with a `.idx` index file), so the client reuses 80% of `HrrrClient`.
- GFS's 384 h horizon is longer than ECMWF HRES's 240 h, and ham-radio use cases care about 3-7 day planning more than 10-day.
- ECMWF open-data 0.25° coverage only goes back to January 2024, so no historical-recalibration option; GFS archive is deep.
- A second ingest pipeline (ECMWF) can layer on later without breaking the first.

---

## Scope

In scope:
1. GFS fetch + parse into the existing HRRR profile shape
2. Forecast scoring for the 18-240 h window (coarser grid)
3. Map UI: extended forecast timeline slider with a visual seam at the resolution change
4. Separate binary-file storage so HRRR and GFS stay independent

Out of scope (follow-ups):
- ECMWF IFS HRES ingest
- GEFS ensemble spread (uncertainty bands)
- ECMWF AIFS AI-forecasts
- Handling the resolution change with anything fancier than a visual seam

---

## Data pipeline

GFS open-data on AWS: `s3://noaa-gfs-bdp-pds/gfs.YYYYMMDD/HH/atmos/gfs.tHHz.pgrb2.0p25.fNNN`

Fields needed per forecast hour (same as HRRR extraction):
- Surface: TMP_2m, DPT_2m, PRES_surface, UGRD/VGRD_10m, PWAT_entire_atmosphere, HPBL
- Pressure levels: TMP / RH / HGT at 1000, 925, 850, 700, 500 mb (5 levels = enough to reconstruct dN/dh with acceptable noise at the 28 km grid scale)
- Derived: surface N, dN/dh min, ducting_detected (via `SoundingParams.derive/1`)

Byte-range HTTP: pull the `.idx` file first, find offsets for needed fields, issue one ranged request per field. GFS `.idx` format is the same as HRRR's.

---

## Task list

### Task 1: `Microwaveprop.Weather.GfsClient` — fetch + decode

**Files:**
- Create: `lib/microwaveprop/weather/gfs_client.ex`
- Create: `test/microwaveprop/weather/gfs_client_test.exs`

**Step 1: Write the failing test**

```elixir
test "builds GFS S3 URL for a given run + forecast hour" do
  assert GfsClient.s3_url(~U[2026-04-18 12:00:00Z], 42) ==
    "https://noaa-gfs-bdp-pds.s3.amazonaws.com/gfs.20260418/12/atmos/gfs.t12z.pgrb2.0p25.f042"
end

test "fetches pressure + surface fields for a grid point" do
  Req.Test.stub(GfsClient, fn conn ->
    conn |> Plug.Conn.put_resp_content_type("application/octet-stream") |> Plug.Conn.send_resp(200, fixture_grib())
  end)

  assert {:ok, profile} = GfsClient.fetch_profile(~U[2026-04-18 12:00:00Z], 42, 32.9, -97.0)
  assert profile.surface_temp_c == 25.0
  assert profile.pwat_mm > 0
end
```

**Step 2: Implement**

Port the byte-range + wgrib2 decoding from `HrrrClient`. Differences:
- `.idx` field names match GFS conventions (`:TMP:2 m above ground:` not `:TMP:2m above ground:` — subtle whitespace changes)
- Pressure levels: 5 instead of 13 — match GFS naming
- Grid: 1440 × 721 (0.25° × 0.25° global) so the `wgrib2 -lon` extraction targets a different grid

**Step 3: Commit.**

### Task 2: `gfs_profiles` table + `GfsProfile` schema

**Files:**
- Create: `priv/repo/migrations/YYYYMMDD_create_gfs_profiles.exs`
- Create: `lib/microwaveprop/weather/gfs_profile.ex`

**Step 1: Write migration**

Same shape as `hrrr_profiles`:
```elixir
create table(:gfs_profiles, primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :run_time, :utc_datetime, null: false
  add :forecast_hour, :integer, null: false
  add :valid_time, :utc_datetime, null: false
  add :lat, :float, null: false
  add :lon, :float, null: false
  add :profile, {:array, :map}
  add :surface_temp_c, :float
  add :surface_dewpoint_c, :float
  add :surface_pressure_mb, :float
  add :pwat_mm, :float
  add :hpbl_m, :float
  add :surface_refractivity, :float
  add :min_refractivity_gradient, :float
  add :ducting_detected, :boolean, default: false
  timestamps(type: :utc_datetime)
end
create unique_index(:gfs_profiles, [:lat, :lon, :valid_time, :run_time])
create index(:gfs_profiles, [:valid_time])
```

**Step 2: Commit.**

### Task 3: `Microwaveprop.Workers.GfsFetchWorker`

**Files:**
- Create: `lib/microwaveprop/workers/gfs_fetch_worker.ex`
- Create: `test/microwaveprop/workers/gfs_fetch_worker_test.exs`

**Step 1: Test — worker fetches one forecast hour + upserts a grid of profiles**

```elixir
test "fetches GFS f042 and upserts 20 CONUS grid-point profiles" do
  Req.Test.stub(GfsClient, &gfs_grib_stub/1)

  assert {:ok, %{upserted: n}} = GfsFetchWorker.perform(%Oban.Job{
    args: %{"run_time" => "2026-04-18T12:00:00Z", "forecast_hour" => 42}
  })
  assert n > 0
end
```

**Step 2: Implement** — mirror `HrrrFetchWorker` but:
- Targets the 0.25° CONUS subset (grid points at 0.25° spacing, not 0.125°)
- Chain from f018 → f240 via `Oban.Pro.Workers.Chain` (one job per forecast hour)
- Only starts new chains when `run_time + 3h20m < now` (GFS latency)

**Step 3: Commit.**

### Task 4: `GfsGridChainWorker` — scheduled chain launcher

**Files:**
- Create: `lib/microwaveprop/workers/gfs_grid_chain_worker.ex`
- Update: `config/runtime.exs` to add a cron entry

```elixir
# Four runs per day, 3.5 hours after each GFS cycle
# (00/06/12/18 UTC). Each chain runs f018 → f240 hourly to f120 and
# 3-hourly after.
{"30 3,9,15,21 * * *", Microwaveprop.Workers.GfsGridChainWorker}
```

**Step 3: Commit.**

### Task 5: Extend `ScoresFile` → per-source files

**Files:**
- Modify: `lib/microwaveprop/propagation/scores_file.ex`
- Create: `lib/microwaveprop/propagation/gfs_scores_file.ex`

Keep HRRR scores in the current file (no migration pain). Add a parallel GFS scores file at `/data/propagation/gfs/<band_mhz>/<valid_time>.bin`. The map reads both: HRRR for valid_time ≤ now + 18h, GFS otherwise.

### Task 6: Extend `PropagationGridWorker` → `GfsPropagationGridWorker`

Same scoring loop, but reads `GfsProfile` instead of `HrrrProfile`, writes to `GfsScoresFile`.

### Task 7: Map UI — extended timeline

**Files:**
- Modify: `assets/js/propagation_map_hook.ts`
- Modify: `lib/microwaveprop_web/live/map_live.ex`

Timeline slider now covers 0–240 h. A visual seam (vertical rule + "18h / 28 km" label) marks the HRRR→GFS resolution drop. Client-side color interpolation dampens slightly past the seam so users can see at a glance which forecast side they're reading.

### Task 8: Forecast page additions

**Files:**
- Modify: `lib/microwaveprop_web/live/path_live.ex`

Path calculator's 18-hour forecast sparkline extends to 240 h. Visual break at the seam. Forecast for a single point comes from `Propagation.point_forecast/3` — extend that function to read GFS scores for valid_time > now+18h.

### Task 9: Tests — end-to-end

Fixture GFS GRIB2 file committed at `test/fixtures/gfs_sample.grib2` (take one hour's worth of real f042 data, subset to a ~5×5 CONUS box, keep <100 KB). Workers and client tests use that fixture via Req.Test.

---

## Cutover strategy

1. Deploy ingest + storage (Tasks 1–6) silently. `GfsGridChainWorker` runs but nothing reads the results yet.
2. Verify 48 h of GFS runs land successfully. Compare GFS f018 to HRRR f018 at a few grid points as a sanity check — they should agree within a few K / a few mb / a few N-units.
3. Roll out map UI (Task 7) behind a feature flag (`:enable_extended_forecast` in `Application.compile_env`). Admin-only at first.
4. Open to all users once the seam looks clean and the forecast scores past f018 don't visually thrash.

---

## Open questions

- **Cold-start handling.** First-time deploy has no GFS data; the map would show a blank past f018 until a full chain runs. Acceptable for a one-time transition, but worth a "Data loading..." placeholder.
- **Storage cost.** GFS CONUS at 0.25° = 272 × 217 = 59,024 grid points. Per forecast hour = 59k × (float64 × 12 fields) ≈ 5.5 MB. Times 240 h per run × 4 runs/day = 5.3 GB/day raw. Keep only the latest run per cycle (drop the previous): 1.3 GB at any moment. Still worth confirming before shipping.
- **Model bias at the seam.** HRRR f018 and GFS f018 disagree by systematic amounts (HRRR is nested, has diurnal cycle bias corrections GFS lacks). A naive join may produce a visible discontinuity at the 18 h mark. Mitigation: fade between HRRR f017 and GFS f018 with a weighted blend over a 1-hour window.
- **ECMWF as a follow-up.** Once GFS works, adding ECMWF is a repeat of Tasks 1–6 with a different S3 bucket and `.idx` format. At that point we can start ensemble-averaging the two, or let users toggle which source drives the extended forecast.

---

## Rollout milestones

- **Week 1:** Tasks 1–3 (client, schema, single-hour worker)
- **Week 2:** Tasks 4–6 (chain, storage, scoring)
- **Week 3:** Tasks 7–8 (UI)
- **Week 4:** Monitoring, tuning the seam blend, open to users.

Total: ~4 weeks of part-time work if no blockers. The longest single task is the client (Task 1) — GFS `.idx` quirks and field-name tolerance will take a few hours of iteration against real data.