prop/docs/plans/2026-04-18-extended-horizon-forecasts.md

Extended-Horizon Propagation Forecasts — Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Extend propagation forecast horizon from HRRR's 18-hour cap to ~10 days, so users can plan VHF/microwave activity for upcoming contest weekends, DXpeditions, and band openings. Handoff is: HRRR drives 0–18 h (what we have today), a coarser global model drives 18–240 h, with the visible seam at ~18 hours being the point where resolution drops from 3 km to ~13 km.

Architecture: New ingestion pipeline for a global-model NWP source running alongside HrrrFetchWorker / PropagationGridWorker. Global-model profiles land in a separate binary-file store (ScoresFileGlobal) so the hourly HRRR chain isn't touched. The map's forecast timeline stretches from hour 0 to hour 240, with a visual band at the ~18 h boundary showing resolution drop.

Tech Stack: Existing (Oban, Ecto, Phoenix LiveView, ProfilesFile/ScoresFile binary storage, wgrib2). New: global-model GRIB2 client and field extractor. No new Python / netCDF dependencies — GFS and ECMWF HRES open-data are both GRIB2.

---

Source comparison

Attribute	HRRR (current)	GFS (proposed)	ECMWF IFS HRES
Horizon	18 h	384 h	240 h
Cadence	Hourly runs, hourly output	4 runs/day (00/06/12/18 UTC), 1 h output through f120, 3 h f120-f240, 12 h f240-f384	4 runs/day, 3 h output through f144, 6 h f144-f240
Horizontal res	3 km	0.25° (~28 km)	0.25° open-data; 0.1° subscription
Vertical	50 hybrid sigma + 40 pressure levels	31 pressure levels, 10 hPa top	37 pressure levels
Latency	~55 min after run start	~3 h 20 min after run start	~4-5 h after run start
Cost	Free, AWS S3	Free, AWS S3 + NCEP	Free 0.25° (since 2024), via ECMWF open-data S3
Surface PWAT, HPBL	Yes	PWAT yes, HPBL yes	PWAT yes, BLH yes
Refractivity gradient	Reconstructable from 13 levels	Reconstructable from pressure levels (coarser than HRRR's 13)	Reconstructable from 37 levels (comparable to HRRR pressure-level)

Recommendation: start with GFS, add ECMWF in a follow-up if forecast skill warrants. Reasons:

• GFS open-data access is identical in shape to HRRR (byte-range GRIB2 on S3 with a .idx index file), so the client reuses 80% of HrrrClient.
• GFS's 384 h horizon is longer than ECMWF HRES's 240 h, and ham-radio use cases care about 3-7 day planning more than 10-day.
• ECMWF open-data 0.25° coverage only goes back to January 2024, so no historical-recalibration option; GFS archive is deep.
• A second ingest pipeline (ECMWF) can layer on later without breaking the first.

---

Scope

In scope:

1. GFS fetch + parse into the existing HRRR profile shape
2. Forecast scoring for the 18-240 h window (coarser grid)
3. Map UI: extended forecast timeline slider with a visual seam at the resolution change
4. Separate binary-file storage so HRRR and GFS stay independent

Out of scope (follow-ups):

• ECMWF IFS HRES ingest
• GEFS ensemble spread (uncertainty bands)
• ECMWF AIFS AI-forecasts
• Handling the resolution change with anything fancier than a visual seam

---

Data pipeline

GFS open-data on AWS: s3://noaa-gfs-bdp-pds/gfs.YYYYMMDD/HH/atmos/gfs.tHHz.pgrb2.0p25.fNNN

Fields needed per forecast hour (same as HRRR extraction):

• Surface: TMP_2m, DPT_2m, PRES_surface, UGRD/VGRD_10m, PWAT_entire_atmosphere, HPBL
• Pressure levels: TMP / RH / HGT at 1000, 925, 850, 700, 500 mb (5 levels = enough to reconstruct dN/dh with acceptable noise at the 28 km grid scale)
• Derived: surface N, dN/dh min, ducting_detected (via SoundingParams.derive/1)

Byte-range HTTP: pull the .idx file first, find offsets for needed fields, issue one ranged request per field. GFS .idx format is the same as HRRR's.

---

Task list

Task 1: Microwaveprop.Weather.GfsClient — fetch + decode

Files:

• Create: lib/microwaveprop/weather/gfs_client.ex
• Create: test/microwaveprop/weather/gfs_client_test.exs

Step 1: Write the failing test

test "builds GFS S3 URL for a given run + forecast hour" do
  assert GfsClient.s3_url(~U[2026-04-18 12:00:00Z], 42) ==
    "https://noaa-gfs-bdp-pds.s3.amazonaws.com/gfs.20260418/12/atmos/gfs.t12z.pgrb2.0p25.f042"
end

test "fetches pressure + surface fields for a grid point" do
  Req.Test.stub(GfsClient, fn conn ->
    conn |> Plug.Conn.put_resp_content_type("application/octet-stream") |> Plug.Conn.send_resp(200, fixture_grib())
  end)

  assert {:ok, profile} = GfsClient.fetch_profile(~U[2026-04-18 12:00:00Z], 42, 32.9, -97.0)
  assert profile.surface_temp_c == 25.0
  assert profile.pwat_mm > 0
end

Step 2: Implement

Port the byte-range + wgrib2 decoding from HrrrClient. Differences:

• .idx field names match GFS conventions (:TMP:2 m above ground: not :TMP:2m above ground: — subtle whitespace changes)
• Pressure levels: 5 instead of 13 — match GFS naming
• Grid: 1440 × 721 (0.25° × 0.25° global) so the wgrib2 -lon extraction targets a different grid

Step 3: Commit.

Task 2: gfs_profiles table + GfsProfile schema

Files:

• Create: priv/repo/migrations/YYYYMMDD_create_gfs_profiles.exs
• Create: lib/microwaveprop/weather/gfs_profile.ex

Step 1: Write migration

Same shape as hrrr_profiles:

create table(:gfs_profiles, primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :run_time, :utc_datetime, null: false
  add :forecast_hour, :integer, null: false
  add :valid_time, :utc_datetime, null: false
  add :lat, :float, null: false
  add :lon, :float, null: false
  add :profile, {:array, :map}
  add :surface_temp_c, :float
  add :surface_dewpoint_c, :float
  add :surface_pressure_mb, :float
  add :pwat_mm, :float
  add :hpbl_m, :float
  add :surface_refractivity, :float
  add :min_refractivity_gradient, :float
  add :ducting_detected, :boolean, default: false
  timestamps(type: :utc_datetime)
end
create unique_index(:gfs_profiles, [:lat, :lon, :valid_time, :run_time])
create index(:gfs_profiles, [:valid_time])

Step 2: Commit.

Task 3: Microwaveprop.Workers.GfsFetchWorker

Files:

• Create: lib/microwaveprop/workers/gfs_fetch_worker.ex
• Create: test/microwaveprop/workers/gfs_fetch_worker_test.exs

Step 1: Test — worker fetches one forecast hour + upserts a grid of profiles

test "fetches GFS f042 and upserts 20 CONUS grid-point profiles" do
  Req.Test.stub(GfsClient, &gfs_grib_stub/1)

  assert {:ok, %{upserted: n}} = GfsFetchWorker.perform(%Oban.Job{
    args: %{"run_time" => "2026-04-18T12:00:00Z", "forecast_hour" => 42}
  })
  assert n > 0
end

Step 2: Implement — mirror HrrrFetchWorker but:

• Targets the 0.25° CONUS subset (grid points at 0.25° spacing, not 0.125°)
• Chain from f018 → f240 via Oban.Pro.Workers.Chain (one job per forecast hour)
• Only starts new chains when run_time + 3h20m < now (GFS latency)

Step 3: Commit.

Task 4: GfsGridChainWorker — scheduled chain launcher

Files:

• Create: lib/microwaveprop/workers/gfs_grid_chain_worker.ex
• Update: config/runtime.exs to add a cron entry

# Four runs per day, 3.5 hours after each GFS cycle
# (00/06/12/18 UTC). Each chain runs f018 → f240 hourly to f120 and
# 3-hourly after.
{"30 3,9,15,21 * * *", Microwaveprop.Workers.GfsGridChainWorker}

Step 3: Commit.

Task 5: Extend ScoresFile → per-source files

Files:

• Modify: lib/microwaveprop/propagation/scores_file.ex
• Create: lib/microwaveprop/propagation/gfs_scores_file.ex

Keep HRRR scores in the current file (no migration pain). Add a parallel GFS scores file at /data/propagation/gfs/<band_mhz>/<valid_time>.bin. The map reads both: HRRR for valid_time ≤ now + 18h, GFS otherwise.

Task 6: Extend PropagationGridWorker → GfsPropagationGridWorker

Same scoring loop, but reads GfsProfile instead of HrrrProfile, writes to GfsScoresFile.

Task 7: Map UI — extended timeline

Files:

• Modify: assets/js/propagation_map_hook.ts
• Modify: lib/microwaveprop_web/live/map_live.ex

Timeline slider now covers 0–240 h. A visual seam (vertical rule + "18h / 28 km" label) marks the HRRR→GFS resolution drop. Client-side color interpolation dampens slightly past the seam so users can see at a glance which forecast side they're reading.

Task 8: Forecast page additions

Files:

• Modify: lib/microwaveprop_web/live/path_live.ex

Path calculator's 18-hour forecast sparkline extends to 240 h. Visual break at the seam. Forecast for a single point comes from Propagation.point_forecast/3 — extend that function to read GFS scores for valid_time > now+18h.

Task 9: Tests — end-to-end

Fixture GFS GRIB2 file committed at test/fixtures/gfs_sample.grib2 (take one hour's worth of real f042 data, subset to a ~5×5 CONUS box, keep <100 KB). Workers and client tests use that fixture via Req.Test.

---

Cutover strategy

1. Deploy ingest + storage (Tasks 1–6) silently. GfsGridChainWorker runs but nothing reads the results yet.
2. Verify 48 h of GFS runs land successfully. Compare GFS f018 to HRRR f018 at a few grid points as a sanity check — they should agree within a few K / a few mb / a few N-units.
3. Roll out map UI (Task 7) behind a feature flag (:enable_extended_forecast in Application.compile_env). Admin-only at first.
4. Open to all users once the seam looks clean and the forecast scores past f018 don't visually thrash.

---

Open questions

• Cold-start handling. First-time deploy has no GFS data; the map would show a blank past f018 until a full chain runs. Acceptable for a one-time transition, but worth a "Data loading..." placeholder.
• Storage cost. GFS CONUS at 0.25° = 272 × 217 = 59,024 grid points. Per forecast hour = 59k × (float64 × 12 fields) ≈ 5.5 MB. Times 240 h per run × 4 runs/day = 5.3 GB/day raw. Keep only the latest run per cycle (drop the previous): 1.3 GB at any moment. Still worth confirming before shipping.
• Model bias at the seam. HRRR f018 and GFS f018 disagree by systematic amounts (HRRR is nested, has diurnal cycle bias corrections GFS lacks). A naive join may produce a visible discontinuity at the 18 h mark. Mitigation: fade between HRRR f017 and GFS f018 with a weighted blend over a 1-hour window.
• ECMWF as a follow-up. Once GFS works, adding ECMWF is a repeat of Tasks 1–6 with a different S3 bucket and .idx format. At that point we can start ensemble-averaging the two, or let users toggle which source drives the extended forecast.

---

Rollout milestones

• Week 1: Tasks 1–3 (client, schema, single-hour worker)
• Week 2: Tasks 4–6 (chain, storage, scoring)
• Week 3: Tasks 7–8 (UI)
• Week 4: Monitoring, tuning the seam blend, open to users.

Total: ~4 weeks of part-time work if no blockers. The longest single task is the client (Task 1) — GFS .idx quirks and field-name tolerance will take a few hours of iteration against real data.