Graham McIntire
f40e88e305
HRRR caps at f018, so beyond ~18 hours the map and path calculator go blank. This plan scopes a GFS open-data ingest that runs alongside HRRR to drive the 18-240 hour forecast window, with a visible seam on the timeline where the resolution drops from 3 km to 0.25°. Scope: GFS 0.25° CONUS subset from AWS S3, four runs/day on the standard cycle, chained fetch/score workers that mirror the HRRR pipeline shape, per-source binary score files so HRRR and GFS stay independent. ECMWF IFS HRES listed as a follow-up once GFS works.
232 lines
11 KiB
Markdown
232 lines
11 KiB
Markdown
# Extended-Horizon Propagation Forecasts — Plan
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** Extend propagation forecast horizon from HRRR's 18-hour cap to ~10 days, so users can plan VHF/microwave activity for upcoming contest weekends, DXpeditions, and band openings. Handoff is: HRRR drives 0–18 h (what we have today), a coarser global model drives 18–240 h, with the visible seam at ~18 hours being the point where resolution drops from 3 km to ~13 km.
|
||
|
||
**Architecture:** New ingestion pipeline for a global-model NWP source running alongside `HrrrFetchWorker` / `PropagationGridWorker`. Global-model profiles land in a separate binary-file store (`ScoresFileGlobal`) so the hourly HRRR chain isn't touched. The map's forecast timeline stretches from hour 0 to hour 240, with a visual band at the ~18 h boundary showing resolution drop.
|
||
|
||
**Tech Stack:** Existing (Oban, Ecto, Phoenix LiveView, ProfilesFile/ScoresFile binary storage, wgrib2). New: global-model GRIB2 client and field extractor. No new Python / netCDF dependencies — GFS and ECMWF HRES open-data are both GRIB2.
|
||
|
||
---
|
||
|
||
## Source comparison
|
||
|
||
| Attribute | HRRR (current) | GFS (proposed) | ECMWF IFS HRES |
|
||
|---|---|---|---|
|
||
| Horizon | 18 h | 384 h | 240 h |
|
||
| Cadence | Hourly runs, hourly output | 4 runs/day (00/06/12/18 UTC), 1 h output through f120, 3 h f120-f240, 12 h f240-f384 | 4 runs/day, 3 h output through f144, 6 h f144-f240 |
|
||
| Horizontal res | 3 km | 0.25° (~28 km) | 0.25° open-data; 0.1° subscription |
|
||
| Vertical | 50 hybrid sigma + 40 pressure levels | 31 pressure levels, 10 hPa top | 37 pressure levels |
|
||
| Latency | ~55 min after run start | ~3 h 20 min after run start | ~4-5 h after run start |
|
||
| Cost | Free, AWS S3 | Free, AWS S3 + NCEP | Free 0.25° (since 2024), via ECMWF open-data S3 |
|
||
| Surface PWAT, HPBL | Yes | PWAT yes, HPBL yes | PWAT yes, BLH yes |
|
||
| Refractivity gradient | Reconstructable from 13 levels | Reconstructable from pressure levels (coarser than HRRR's 13) | Reconstructable from 37 levels (comparable to HRRR pressure-level) |
|
||
|
||
**Recommendation: start with GFS**, add ECMWF in a follow-up if forecast skill warrants. Reasons:
|
||
- GFS open-data access is identical in shape to HRRR (byte-range GRIB2 on S3 with a `.idx` index file), so the client reuses 80% of `HrrrClient`.
|
||
- GFS's 384 h horizon is longer than ECMWF HRES's 240 h, and ham-radio use cases care about 3-7 day planning more than 10-day.
|
||
- ECMWF open-data 0.25° coverage only goes back to January 2024, so no historical-recalibration option; GFS archive is deep.
|
||
- A second ingest pipeline (ECMWF) can layer on later without breaking the first.
|
||
|
||
---
|
||
|
||
## Scope
|
||
|
||
In scope:
|
||
1. GFS fetch + parse into the existing HRRR profile shape
|
||
2. Forecast scoring for the 18-240 h window (coarser grid)
|
||
3. Map UI: extended forecast timeline slider with a visual seam at the resolution change
|
||
4. Separate binary-file storage so HRRR and GFS stay independent
|
||
|
||
Out of scope (follow-ups):
|
||
- ECMWF IFS HRES ingest
|
||
- GEFS ensemble spread (uncertainty bands)
|
||
- ECMWF AIFS AI-forecasts
|
||
- Handling the resolution change with anything fancier than a visual seam
|
||
|
||
---
|
||
|
||
## Data pipeline
|
||
|
||
GFS open-data on AWS: `s3://noaa-gfs-bdp-pds/gfs.YYYYMMDD/HH/atmos/gfs.tHHz.pgrb2.0p25.fNNN`
|
||
|
||
Fields needed per forecast hour (same as HRRR extraction):
|
||
- Surface: TMP_2m, DPT_2m, PRES_surface, UGRD/VGRD_10m, PWAT_entire_atmosphere, HPBL
|
||
- Pressure levels: TMP / RH / HGT at 1000, 925, 850, 700, 500 mb (5 levels = enough to reconstruct dN/dh with acceptable noise at the 28 km grid scale)
|
||
- Derived: surface N, dN/dh min, ducting_detected (via `SoundingParams.derive/1`)
|
||
|
||
Byte-range HTTP: pull the `.idx` file first, find offsets for needed fields, issue one ranged request per field. GFS `.idx` format is the same as HRRR's.
|
||
|
||
---
|
||
|
||
## Task list
|
||
|
||
### Task 1: `Microwaveprop.Weather.GfsClient` — fetch + decode
|
||
|
||
**Files:**
|
||
- Create: `lib/microwaveprop/weather/gfs_client.ex`
|
||
- Create: `test/microwaveprop/weather/gfs_client_test.exs`
|
||
|
||
**Step 1: Write the failing test**
|
||
|
||
```elixir
|
||
test "builds GFS S3 URL for a given run + forecast hour" do
|
||
assert GfsClient.s3_url(~U[2026-04-18 12:00:00Z], 42) ==
|
||
"https://noaa-gfs-bdp-pds.s3.amazonaws.com/gfs.20260418/12/atmos/gfs.t12z.pgrb2.0p25.f042"
|
||
end
|
||
|
||
test "fetches pressure + surface fields for a grid point" do
|
||
Req.Test.stub(GfsClient, fn conn ->
|
||
conn |> Plug.Conn.put_resp_content_type("application/octet-stream") |> Plug.Conn.send_resp(200, fixture_grib())
|
||
end)
|
||
|
||
assert {:ok, profile} = GfsClient.fetch_profile(~U[2026-04-18 12:00:00Z], 42, 32.9, -97.0)
|
||
assert profile.surface_temp_c == 25.0
|
||
assert profile.pwat_mm > 0
|
||
end
|
||
```
|
||
|
||
**Step 2: Implement**
|
||
|
||
Port the byte-range + wgrib2 decoding from `HrrrClient`. Differences:
|
||
- `.idx` field names match GFS conventions (`:TMP:2 m above ground:` not `:TMP:2m above ground:` — subtle whitespace changes)
|
||
- Pressure levels: 5 instead of 13 — match GFS naming
|
||
- Grid: 1440 × 721 (0.25° × 0.25° global) so the `wgrib2 -lon` extraction targets a different grid
|
||
|
||
**Step 3: Commit.**
|
||
|
||
### Task 2: `gfs_profiles` table + `GfsProfile` schema
|
||
|
||
**Files:**
|
||
- Create: `priv/repo/migrations/YYYYMMDD_create_gfs_profiles.exs`
|
||
- Create: `lib/microwaveprop/weather/gfs_profile.ex`
|
||
|
||
**Step 1: Write migration**
|
||
|
||
Same shape as `hrrr_profiles`:
|
||
```elixir
|
||
create table(:gfs_profiles, primary_key: false) do
|
||
add :id, :binary_id, primary_key: true
|
||
add :run_time, :utc_datetime, null: false
|
||
add :forecast_hour, :integer, null: false
|
||
add :valid_time, :utc_datetime, null: false
|
||
add :lat, :float, null: false
|
||
add :lon, :float, null: false
|
||
add :profile, {:array, :map}
|
||
add :surface_temp_c, :float
|
||
add :surface_dewpoint_c, :float
|
||
add :surface_pressure_mb, :float
|
||
add :pwat_mm, :float
|
||
add :hpbl_m, :float
|
||
add :surface_refractivity, :float
|
||
add :min_refractivity_gradient, :float
|
||
add :ducting_detected, :boolean, default: false
|
||
timestamps(type: :utc_datetime)
|
||
end
|
||
create unique_index(:gfs_profiles, [:lat, :lon, :valid_time, :run_time])
|
||
create index(:gfs_profiles, [:valid_time])
|
||
```
|
||
|
||
**Step 2: Commit.**
|
||
|
||
### Task 3: `Microwaveprop.Workers.GfsFetchWorker`
|
||
|
||
**Files:**
|
||
- Create: `lib/microwaveprop/workers/gfs_fetch_worker.ex`
|
||
- Create: `test/microwaveprop/workers/gfs_fetch_worker_test.exs`
|
||
|
||
**Step 1: Test — worker fetches one forecast hour + upserts a grid of profiles**
|
||
|
||
```elixir
|
||
test "fetches GFS f042 and upserts 20 CONUS grid-point profiles" do
|
||
Req.Test.stub(GfsClient, &gfs_grib_stub/1)
|
||
|
||
assert {:ok, %{upserted: n}} = GfsFetchWorker.perform(%Oban.Job{
|
||
args: %{"run_time" => "2026-04-18T12:00:00Z", "forecast_hour" => 42}
|
||
})
|
||
assert n > 0
|
||
end
|
||
```
|
||
|
||
**Step 2: Implement** — mirror `HrrrFetchWorker` but:
|
||
- Targets the 0.25° CONUS subset (grid points at 0.25° spacing, not 0.125°)
|
||
- Chain from f018 → f240 via `Oban.Pro.Workers.Chain` (one job per forecast hour)
|
||
- Only starts new chains when `run_time + 3h20m < now` (GFS latency)
|
||
|
||
**Step 3: Commit.**
|
||
|
||
### Task 4: `GfsGridChainWorker` — scheduled chain launcher
|
||
|
||
**Files:**
|
||
- Create: `lib/microwaveprop/workers/gfs_grid_chain_worker.ex`
|
||
- Update: `config/runtime.exs` to add a cron entry
|
||
|
||
```elixir
|
||
# Four runs per day, 3.5 hours after each GFS cycle
|
||
# (00/06/12/18 UTC). Each chain runs f018 → f240 hourly to f120 and
|
||
# 3-hourly after.
|
||
{"30 3,9,15,21 * * *", Microwaveprop.Workers.GfsGridChainWorker}
|
||
```
|
||
|
||
**Step 3: Commit.**
|
||
|
||
### Task 5: Extend `ScoresFile` → per-source files
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop/propagation/scores_file.ex`
|
||
- Create: `lib/microwaveprop/propagation/gfs_scores_file.ex`
|
||
|
||
Keep HRRR scores in the current file (no migration pain). Add a parallel GFS scores file at `/data/propagation/gfs/<band_mhz>/<valid_time>.bin`. The map reads both: HRRR for valid_time ≤ now + 18h, GFS otherwise.
|
||
|
||
### Task 6: Extend `PropagationGridWorker` → `GfsPropagationGridWorker`
|
||
|
||
Same scoring loop, but reads `GfsProfile` instead of `HrrrProfile`, writes to `GfsScoresFile`.
|
||
|
||
### Task 7: Map UI — extended timeline
|
||
|
||
**Files:**
|
||
- Modify: `assets/js/propagation_map_hook.ts`
|
||
- Modify: `lib/microwaveprop_web/live/map_live.ex`
|
||
|
||
Timeline slider now covers 0–240 h. A visual seam (vertical rule + "18h / 28 km" label) marks the HRRR→GFS resolution drop. Client-side color interpolation dampens slightly past the seam so users can see at a glance which forecast side they're reading.
|
||
|
||
### Task 8: Forecast page additions
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop_web/live/path_live.ex`
|
||
|
||
Path calculator's 18-hour forecast sparkline extends to 240 h. Visual break at the seam. Forecast for a single point comes from `Propagation.point_forecast/3` — extend that function to read GFS scores for valid_time > now+18h.
|
||
|
||
### Task 9: Tests — end-to-end
|
||
|
||
Fixture GFS GRIB2 file committed at `test/fixtures/gfs_sample.grib2` (take one hour's worth of real f042 data, subset to a ~5×5 CONUS box, keep <100 KB). Workers and client tests use that fixture via Req.Test.
|
||
|
||
---
|
||
|
||
## Cutover strategy
|
||
|
||
1. Deploy ingest + storage (Tasks 1–6) silently. `GfsGridChainWorker` runs but nothing reads the results yet.
|
||
2. Verify 48 h of GFS runs land successfully. Compare GFS f018 to HRRR f018 at a few grid points as a sanity check — they should agree within a few K / a few mb / a few N-units.
|
||
3. Roll out map UI (Task 7) behind a feature flag (`:enable_extended_forecast` in `Application.compile_env`). Admin-only at first.
|
||
4. Open to all users once the seam looks clean and the forecast scores past f018 don't visually thrash.
|
||
|
||
---
|
||
|
||
## Open questions
|
||
|
||
- **Cold-start handling.** First-time deploy has no GFS data; the map would show a blank past f018 until a full chain runs. Acceptable for a one-time transition, but worth a "Data loading..." placeholder.
|
||
- **Storage cost.** GFS CONUS at 0.25° = 272 × 217 = 59,024 grid points. Per forecast hour = 59k × (float64 × 12 fields) ≈ 5.5 MB. Times 240 h per run × 4 runs/day = 5.3 GB/day raw. Keep only the latest run per cycle (drop the previous): 1.3 GB at any moment. Still worth confirming before shipping.
|
||
- **Model bias at the seam.** HRRR f018 and GFS f018 disagree by systematic amounts (HRRR is nested, has diurnal cycle bias corrections GFS lacks). A naive join may produce a visible discontinuity at the 18 h mark. Mitigation: fade between HRRR f017 and GFS f018 with a weighted blend over a 1-hour window.
|
||
- **ECMWF as a follow-up.** Once GFS works, adding ECMWF is a repeat of Tasks 1–6 with a different S3 bucket and `.idx` format. At that point we can start ensemble-averaging the two, or let users toggle which source drives the extended forecast.
|
||
|
||
---
|
||
|
||
## Rollout milestones
|
||
|
||
- **Week 1:** Tasks 1–3 (client, schema, single-hour worker)
|
||
- **Week 2:** Tasks 4–6 (chain, storage, scoring)
|
||
- **Week 3:** Tasks 7–8 (UI)
|
||
- **Week 4:** Monitoring, tuning the seam blend, open to users.
|
||
|
||
Total: ~4 weeks of part-time work if no blockers. The longest single task is the client (Task 1) — GFS `.idx` quirks and field-name tolerance will take a few hours of iteration against real data.
|