From f40e88e3050a7f98b68d51638baeec4399661ef5 Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Sat, 18 Apr 2026 12:35:07 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20plan=20for=20GFS-driven=20extended-hori?= =?UTF-8?q?zon=20forecasts=20(18=20h=20=E2=86=92=20240=20h)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit HRRR caps at f018, so beyond ~18 hours the map and path calculator go blank. This plan scopes a GFS open-data ingest that runs alongside HRRR to drive the 18-240 hour forecast window, with a visible seam on the timeline where the resolution drops from 3 km to 0.25°. Scope: GFS 0.25° CONUS subset from AWS S3, four runs/day on the standard cycle, chained fetch/score workers that mirror the HRRR pipeline shape, per-source binary score files so HRRR and GFS stay independent. ECMWF IFS HRES listed as a follow-up once GFS works. --- .../2026-04-18-extended-horizon-forecasts.md | 232 ++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 docs/plans/2026-04-18-extended-horizon-forecasts.md diff --git a/docs/plans/2026-04-18-extended-horizon-forecasts.md b/docs/plans/2026-04-18-extended-horizon-forecasts.md new file mode 100644 index 00000000..b63b494a --- /dev/null +++ b/docs/plans/2026-04-18-extended-horizon-forecasts.md @@ -0,0 +1,232 @@ +# Extended-Horizon Propagation Forecasts — Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Extend propagation forecast horizon from HRRR's 18-hour cap to ~10 days, so users can plan VHF/microwave activity for upcoming contest weekends, DXpeditions, and band openings. Handoff is: HRRR drives 0–18 h (what we have today), a coarser global model drives 18–240 h, with the visible seam at ~18 hours being the point where resolution drops from 3 km to ~13 km. + +**Architecture:** New ingestion pipeline for a global-model NWP source running alongside `HrrrFetchWorker` / `PropagationGridWorker`. Global-model profiles land in a separate binary-file store (`ScoresFileGlobal`) so the hourly HRRR chain isn't touched. The map's forecast timeline stretches from hour 0 to hour 240, with a visual band at the ~18 h boundary showing resolution drop. + +**Tech Stack:** Existing (Oban, Ecto, Phoenix LiveView, ProfilesFile/ScoresFile binary storage, wgrib2). New: global-model GRIB2 client and field extractor. No new Python / netCDF dependencies — GFS and ECMWF HRES open-data are both GRIB2. + +--- + +## Source comparison + +| Attribute | HRRR (current) | GFS (proposed) | ECMWF IFS HRES | +|---|---|---|---| +| Horizon | 18 h | 384 h | 240 h | +| Cadence | Hourly runs, hourly output | 4 runs/day (00/06/12/18 UTC), 1 h output through f120, 3 h f120-f240, 12 h f240-f384 | 4 runs/day, 3 h output through f144, 6 h f144-f240 | +| Horizontal res | 3 km | 0.25° (~28 km) | 0.25° open-data; 0.1° subscription | +| Vertical | 50 hybrid sigma + 40 pressure levels | 31 pressure levels, 10 hPa top | 37 pressure levels | +| Latency | ~55 min after run start | ~3 h 20 min after run start | ~4-5 h after run start | +| Cost | Free, AWS S3 | Free, AWS S3 + NCEP | Free 0.25° (since 2024), via ECMWF open-data S3 | +| Surface PWAT, HPBL | Yes | PWAT yes, HPBL yes | PWAT yes, BLH yes | +| Refractivity gradient | Reconstructable from 13 levels | Reconstructable from pressure levels (coarser than HRRR's 13) | Reconstructable from 37 levels (comparable to HRRR pressure-level) | + +**Recommendation: start with GFS**, add ECMWF in a follow-up if forecast skill warrants. Reasons: +- GFS open-data access is identical in shape to HRRR (byte-range GRIB2 on S3 with a `.idx` index file), so the client reuses 80% of `HrrrClient`. +- GFS's 384 h horizon is longer than ECMWF HRES's 240 h, and ham-radio use cases care about 3-7 day planning more than 10-day. +- ECMWF open-data 0.25° coverage only goes back to January 2024, so no historical-recalibration option; GFS archive is deep. +- A second ingest pipeline (ECMWF) can layer on later without breaking the first. + +--- + +## Scope + +In scope: +1. GFS fetch + parse into the existing HRRR profile shape +2. Forecast scoring for the 18-240 h window (coarser grid) +3. Map UI: extended forecast timeline slider with a visual seam at the resolution change +4. Separate binary-file storage so HRRR and GFS stay independent + +Out of scope (follow-ups): +- ECMWF IFS HRES ingest +- GEFS ensemble spread (uncertainty bands) +- ECMWF AIFS AI-forecasts +- Handling the resolution change with anything fancier than a visual seam + +--- + +## Data pipeline + +GFS open-data on AWS: `s3://noaa-gfs-bdp-pds/gfs.YYYYMMDD/HH/atmos/gfs.tHHz.pgrb2.0p25.fNNN` + +Fields needed per forecast hour (same as HRRR extraction): +- Surface: TMP_2m, DPT_2m, PRES_surface, UGRD/VGRD_10m, PWAT_entire_atmosphere, HPBL +- Pressure levels: TMP / RH / HGT at 1000, 925, 850, 700, 500 mb (5 levels = enough to reconstruct dN/dh with acceptable noise at the 28 km grid scale) +- Derived: surface N, dN/dh min, ducting_detected (via `SoundingParams.derive/1`) + +Byte-range HTTP: pull the `.idx` file first, find offsets for needed fields, issue one ranged request per field. GFS `.idx` format is the same as HRRR's. + +--- + +## Task list + +### Task 1: `Microwaveprop.Weather.GfsClient` — fetch + decode + +**Files:** +- Create: `lib/microwaveprop/weather/gfs_client.ex` +- Create: `test/microwaveprop/weather/gfs_client_test.exs` + +**Step 1: Write the failing test** + +```elixir +test "builds GFS S3 URL for a given run + forecast hour" do + assert GfsClient.s3_url(~U[2026-04-18 12:00:00Z], 42) == + "https://noaa-gfs-bdp-pds.s3.amazonaws.com/gfs.20260418/12/atmos/gfs.t12z.pgrb2.0p25.f042" +end + +test "fetches pressure + surface fields for a grid point" do + Req.Test.stub(GfsClient, fn conn -> + conn |> Plug.Conn.put_resp_content_type("application/octet-stream") |> Plug.Conn.send_resp(200, fixture_grib()) + end) + + assert {:ok, profile} = GfsClient.fetch_profile(~U[2026-04-18 12:00:00Z], 42, 32.9, -97.0) + assert profile.surface_temp_c == 25.0 + assert profile.pwat_mm > 0 +end +``` + +**Step 2: Implement** + +Port the byte-range + wgrib2 decoding from `HrrrClient`. Differences: +- `.idx` field names match GFS conventions (`:TMP:2 m above ground:` not `:TMP:2m above ground:` — subtle whitespace changes) +- Pressure levels: 5 instead of 13 — match GFS naming +- Grid: 1440 × 721 (0.25° × 0.25° global) so the `wgrib2 -lon` extraction targets a different grid + +**Step 3: Commit.** + +### Task 2: `gfs_profiles` table + `GfsProfile` schema + +**Files:** +- Create: `priv/repo/migrations/YYYYMMDD_create_gfs_profiles.exs` +- Create: `lib/microwaveprop/weather/gfs_profile.ex` + +**Step 1: Write migration** + +Same shape as `hrrr_profiles`: +```elixir +create table(:gfs_profiles, primary_key: false) do + add :id, :binary_id, primary_key: true + add :run_time, :utc_datetime, null: false + add :forecast_hour, :integer, null: false + add :valid_time, :utc_datetime, null: false + add :lat, :float, null: false + add :lon, :float, null: false + add :profile, {:array, :map} + add :surface_temp_c, :float + add :surface_dewpoint_c, :float + add :surface_pressure_mb, :float + add :pwat_mm, :float + add :hpbl_m, :float + add :surface_refractivity, :float + add :min_refractivity_gradient, :float + add :ducting_detected, :boolean, default: false + timestamps(type: :utc_datetime) +end +create unique_index(:gfs_profiles, [:lat, :lon, :valid_time, :run_time]) +create index(:gfs_profiles, [:valid_time]) +``` + +**Step 2: Commit.** + +### Task 3: `Microwaveprop.Workers.GfsFetchWorker` + +**Files:** +- Create: `lib/microwaveprop/workers/gfs_fetch_worker.ex` +- Create: `test/microwaveprop/workers/gfs_fetch_worker_test.exs` + +**Step 1: Test — worker fetches one forecast hour + upserts a grid of profiles** + +```elixir +test "fetches GFS f042 and upserts 20 CONUS grid-point profiles" do + Req.Test.stub(GfsClient, &gfs_grib_stub/1) + + assert {:ok, %{upserted: n}} = GfsFetchWorker.perform(%Oban.Job{ + args: %{"run_time" => "2026-04-18T12:00:00Z", "forecast_hour" => 42} + }) + assert n > 0 +end +``` + +**Step 2: Implement** — mirror `HrrrFetchWorker` but: +- Targets the 0.25° CONUS subset (grid points at 0.25° spacing, not 0.125°) +- Chain from f018 → f240 via `Oban.Pro.Workers.Chain` (one job per forecast hour) +- Only starts new chains when `run_time + 3h20m < now` (GFS latency) + +**Step 3: Commit.** + +### Task 4: `GfsGridChainWorker` — scheduled chain launcher + +**Files:** +- Create: `lib/microwaveprop/workers/gfs_grid_chain_worker.ex` +- Update: `config/runtime.exs` to add a cron entry + +```elixir +# Four runs per day, 3.5 hours after each GFS cycle +# (00/06/12/18 UTC). Each chain runs f018 → f240 hourly to f120 and +# 3-hourly after. +{"30 3,9,15,21 * * *", Microwaveprop.Workers.GfsGridChainWorker} +``` + +**Step 3: Commit.** + +### Task 5: Extend `ScoresFile` → per-source files + +**Files:** +- Modify: `lib/microwaveprop/propagation/scores_file.ex` +- Create: `lib/microwaveprop/propagation/gfs_scores_file.ex` + +Keep HRRR scores in the current file (no migration pain). Add a parallel GFS scores file at `/data/propagation/gfs//.bin`. The map reads both: HRRR for valid_time ≤ now + 18h, GFS otherwise. + +### Task 6: Extend `PropagationGridWorker` → `GfsPropagationGridWorker` + +Same scoring loop, but reads `GfsProfile` instead of `HrrrProfile`, writes to `GfsScoresFile`. + +### Task 7: Map UI — extended timeline + +**Files:** +- Modify: `assets/js/propagation_map_hook.ts` +- Modify: `lib/microwaveprop_web/live/map_live.ex` + +Timeline slider now covers 0–240 h. A visual seam (vertical rule + "18h / 28 km" label) marks the HRRR→GFS resolution drop. Client-side color interpolation dampens slightly past the seam so users can see at a glance which forecast side they're reading. + +### Task 8: Forecast page additions + +**Files:** +- Modify: `lib/microwaveprop_web/live/path_live.ex` + +Path calculator's 18-hour forecast sparkline extends to 240 h. Visual break at the seam. Forecast for a single point comes from `Propagation.point_forecast/3` — extend that function to read GFS scores for valid_time > now+18h. + +### Task 9: Tests — end-to-end + +Fixture GFS GRIB2 file committed at `test/fixtures/gfs_sample.grib2` (take one hour's worth of real f042 data, subset to a ~5×5 CONUS box, keep <100 KB). Workers and client tests use that fixture via Req.Test. + +--- + +## Cutover strategy + +1. Deploy ingest + storage (Tasks 1–6) silently. `GfsGridChainWorker` runs but nothing reads the results yet. +2. Verify 48 h of GFS runs land successfully. Compare GFS f018 to HRRR f018 at a few grid points as a sanity check — they should agree within a few K / a few mb / a few N-units. +3. Roll out map UI (Task 7) behind a feature flag (`:enable_extended_forecast` in `Application.compile_env`). Admin-only at first. +4. Open to all users once the seam looks clean and the forecast scores past f018 don't visually thrash. + +--- + +## Open questions + +- **Cold-start handling.** First-time deploy has no GFS data; the map would show a blank past f018 until a full chain runs. Acceptable for a one-time transition, but worth a "Data loading..." placeholder. +- **Storage cost.** GFS CONUS at 0.25° = 272 × 217 = 59,024 grid points. Per forecast hour = 59k × (float64 × 12 fields) ≈ 5.5 MB. Times 240 h per run × 4 runs/day = 5.3 GB/day raw. Keep only the latest run per cycle (drop the previous): 1.3 GB at any moment. Still worth confirming before shipping. +- **Model bias at the seam.** HRRR f018 and GFS f018 disagree by systematic amounts (HRRR is nested, has diurnal cycle bias corrections GFS lacks). A naive join may produce a visible discontinuity at the 18 h mark. Mitigation: fade between HRRR f017 and GFS f018 with a weighted blend over a 1-hour window. +- **ECMWF as a follow-up.** Once GFS works, adding ECMWF is a repeat of Tasks 1–6 with a different S3 bucket and `.idx` format. At that point we can start ensemble-averaging the two, or let users toggle which source drives the extended forecast. + +--- + +## Rollout milestones + +- **Week 1:** Tasks 1–3 (client, schema, single-hour worker) +- **Week 2:** Tasks 4–6 (chain, storage, scoring) +- **Week 3:** Tasks 7–8 (UI) +- **Week 4:** Monitoring, tuning the seam blend, open to users. + +Total: ~4 weeks of part-time work if no blockers. The longest single task is the client (Task 1) — GFS `.idx` quirks and field-name tolerance will take a few hours of iteration against real data.