Graham McIntire
aab3c3736c
HRRR publishes f21-f48 at 3-hourly intervals in addition to the hourly f01-f18 we already fetch. This extends the grid_tasks seed from 19 rows (1 analysis + 18 forecast) to 29 rows (1 analysis + 18 hourly + 10 3-hourly), and bumps the UI forecast window constant from 18h to 48h so the map timeline and path calculator forecast chart automatically show the extended horizon. No Rust logic changes needed — the pipeline is data-driven and u8 forecast_hour already handles f48.
362 lines
12 KiB
Markdown
362 lines
12 KiB
Markdown
# Extend HRRR Forecast Horizon to 48 Hours — Implementation Plan
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** Extend the propagation forecast from 18 hours to 48 hours using HRRR's own extended forecast hours (f21–f48 at 3-hourly intervals), without adding a new data source.
|
||
|
||
**Architecture:** The system is data-driven — whatever forecast hours are seeded into `grid_tasks`, Rust processes. Two constants gate the current 18-hour cap: `GridTaskEnqueuer.@max_forecast_hour` and `Propagation.@hrrr_forecast_horizon_hours`. Changing these, plus adjusting a few validation limits and comment strings, is the entire change. No Rust code changes needed.
|
||
|
||
**Tech Stack:** Elixir (Oban, Ecto, Phoenix LiveView), Rust (prop_grid_rs), HRRR NOAA S3
|
||
|
||
**Relationship to prior plan:** `docs/plans/2026-04-18-extended-horizon-forecasts.md` covers adding GFS as a separate data source for 10-day forecasts. This plan is the zero-new-infrastructure quick win — HRRR already publishes f21–f48, we just don't fetch them. The GFS plan is still the right long-term approach for multi-day planning; this extends the existing hourly chain at near-zero cost.
|
||
|
||
---
|
||
|
||
## HRRR forecast hour availability
|
||
|
||
| Range | Interval | Count | Notes |
|
||
|---|---|---|---|
|
||
| f00 | — | 1 | Analysis (Elixir-owned, unchanged) |
|
||
| f01–f18 | 1-hourly | 18 | What we fetch today |
|
||
| f21–f48 | 3-hourly | 10 | New: f21, f24, f27, f30, f33, f36, f39, f42, f45, f48 |
|
||
|
||
Total: 1 analysis + 18 hourly + 10 3-hourly = 29 `grid_tasks` rows per cycle (up from 19).
|
||
|
||
Accuracy degrades beyond f18 — HRRR is a 3 km convection-allowing model; small-scale features (ducting, refractivity gradients) lose fidelity at longer lead times. The 3-hourly output spacing reflects this. Accepted trade-off per user request.
|
||
|
||
---
|
||
|
||
### Task 1: Define the forecast hour list and update `GridTaskEnqueuer`
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop/propagation/grid_task_enqueuer.ex`
|
||
|
||
**Changes:**
|
||
|
||
Replace the `@max_forecast_hour 18` module attribute with a list of all forecast hours Rust should process:
|
||
|
||
```elixir
|
||
# Before (line 19):
|
||
@max_forecast_hour 18
|
||
|
||
# After:
|
||
@forecast_hours [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48]
|
||
```
|
||
|
||
Update `seed_with_analysis/2` (line 70) — the `for fh <- 1..@max_forecast_hour` comprehension:
|
||
|
||
```elixir
|
||
# Before (line 70):
|
||
for fh <- 1..@max_forecast_hour do
|
||
|
||
# After:
|
||
for fh <- @forecast_hours do
|
||
```
|
||
|
||
Update `seed/2` (line 103) — same pattern:
|
||
|
||
```elixir
|
||
# Before (line 103):
|
||
for fh <- 1..@max_forecast_hour do
|
||
|
||
# After:
|
||
for fh <- @forecast_hours do
|
||
```
|
||
|
||
Update `seed_current_hour/2` (lines 174, 192) — the clamp ceiling:
|
||
|
||
```elixir
|
||
# Before (line 174):
|
||
# Forecast hour is clamped to `1..@max_forecast_hour`.
|
||
# (line 176):
|
||
# `run_time + @max_forecast_hour`, we cap there.
|
||
# (line 192):
|
||
|> min(@max_forecast_hour)
|
||
|
||
# After (line 174):
|
||
# Forecast hour is clamped to the available forecast hour list.
|
||
# (line 176):
|
||
# the max forecast hour, we cap there.
|
||
# (line 192):
|
||
|> min(Enum.max(@forecast_hours))
|
||
```
|
||
|
||
Update doc comments (lines 36–37, 92):
|
||
|
||
```elixir
|
||
# Before (line 36-37):
|
||
# Seed one kind='analysis' row (f00) plus 18 kind='forecast' rows
|
||
# (f01..f18) for `run_time`.
|
||
|
||
# After:
|
||
# Seed one kind='analysis' row (f00) plus forecast rows
|
||
# (f01..f18 hourly, f21..f48 3-hourly) for `run_time`.
|
||
|
||
# Before (line 92):
|
||
# Legacy: seed only kind='forecast' rows (f01..f18).
|
||
|
||
# After:
|
||
# Legacy: seed only kind='forecast' rows.
|
||
```
|
||
|
||
**Step 1: Update the test to expect the new forecast hour list**
|
||
|
||
File: `test/microwaveprop/propagation/grid_task_enqueuer_test.exs:22`
|
||
|
||
```elixir
|
||
# Before:
|
||
assert rows |> Enum.map(& &1.forecast_hour) |> Enum.sort() == Enum.to_list(1..18)
|
||
|
||
# After:
|
||
@forecast_hours [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48]
|
||
assert rows |> Enum.map(& &1.forecast_hour) |> Enum.sort() == @forecast_hours
|
||
```
|
||
|
||
Update the worker test at `test/microwaveprop/workers/propagation_grid_worker_test.exs`:
|
||
|
||
```elixir
|
||
# Line 7 (comment):
|
||
# Before: inserts 19 grid_tasks rows (1 analysis f00 + 18 forecast f01..f18)
|
||
# After: inserts grid_tasks rows (1 analysis f00 + forecast f01..f18 hourly, f21..f48 3-hourly)
|
||
|
||
# Line 70:
|
||
# Before: test "inserts 1 analysis + 18 forecast grid_tasks rows; no Oban fan-out" do
|
||
# After: test "inserts 1 analysis + forecast grid_tasks rows; no Oban fan-out" do
|
||
|
||
# Line 120:
|
||
# Before: assert kinds == %{"analysis" => 1, "forecast" => 18}
|
||
# After: assert kinds == %{"analysis" => 1, "forecast" => 28}
|
||
|
||
# Line 125:
|
||
# Before: assert forecast_fhs == Enum.to_list(1..18)
|
||
# After: assert forecast_fhs == @forecast_hours (with the module attribute defined above)
|
||
```
|
||
|
||
**Step 2: Run tests to verify they fail**
|
||
|
||
```bash
|
||
mix test test/microwaveprop/propagation/grid_task_enqueuer_test.exs test/microwaveprop/workers/propagation_grid_worker_test.exs
|
||
```
|
||
|
||
Expected: tests fail because they still expect 18 forecast hours.
|
||
|
||
**Step 3: Apply the `GridTaskEnqueuer` changes**
|
||
|
||
Make the code changes listed above.
|
||
|
||
**Step 4: Run tests to verify they pass**
|
||
|
||
```bash
|
||
mix test test/microwaveprop/propagation/grid_task_enqueuer_test.exs test/microwaveprop/workers/propagation_grid_worker_test.exs
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
**Step 5: Commit**
|
||
|
||
```bash
|
||
git add lib/microwaveprop/propagation/grid_task_enqueuer.ex test/microwaveprop/propagation/grid_task_enqueuer_test.exs test/microwaveprop/workers/propagation_grid_worker_test.exs
|
||
git commit -m "feat: extend grid_tasks seeding to 48h forecast horizon"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: Update propagation forecast horizon constant
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop/propagation.ex`
|
||
|
||
**Changes:**
|
||
|
||
```elixir
|
||
# Before (line 261):
|
||
@hrrr_forecast_horizon_hours 18
|
||
|
||
# After:
|
||
@hrrr_forecast_horizon_hours 48
|
||
```
|
||
|
||
Update the comment on lines 258–260:
|
||
|
||
```elixir
|
||
# Before:
|
||
# HRRR forecast horizon: f00..f18 covers the next 18 hours from cycle
|
||
# time. Anything beyond that in the score store is a leftover from a
|
||
# stale cycle and clutters the timeline without adding information.
|
||
|
||
# After:
|
||
# HRRR forecast horizon: f00..f48 covers the next 48 hours from cycle
|
||
# time (f01-f18 hourly, f21-f48 3-hourly). Anything beyond that in the
|
||
# score store is a leftover from a stale cycle and clutters the
|
||
# timeline without adding information.
|
||
```
|
||
|
||
Update the `@doc` for `hot_cache_window/0` (line 275):
|
||
|
||
```elixir
|
||
# Before:
|
||
through HRRR's 18-hour forecast horizon.
|
||
|
||
# After:
|
||
through HRRR's 48-hour forecast horizon.
|
||
```
|
||
|
||
This single constant change automatically extends:
|
||
- The map timeline (`available_valid_times/1` → `hot_cache_window/0`)
|
||
- The path calculator forecast chart (`point_forecast/3` → `forecast_window/2`)
|
||
- The score cache ETS retention window (`hot_cache_window/0` used by `NotifyListener`)
|
||
|
||
**Step 1: Make the change**
|
||
|
||
**Step 2: Run propagation tests**
|
||
|
||
```bash
|
||
mix test test/microwaveprop/propagation_test.exs
|
||
```
|
||
|
||
**Step 3: Commit**
|
||
|
||
```bash
|
||
git add lib/microwaveprop/propagation.ex
|
||
git commit -m "feat: extend forecast horizon window from 18h to 48h"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: Update validation limits and UI clamps
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop/propagation/run_timing.ex:38`
|
||
- Modify: `lib/microwaveprop_web/live/rover_live.ex:185`
|
||
|
||
**Changes:**
|
||
|
||
`run_timing.ex` — bump the validation ceiling and update doc:
|
||
|
||
```elixir
|
||
# Before (line 2-7 doc):
|
||
# `forecast_hour` ∈ 0..18.
|
||
|
||
# After:
|
||
# `forecast_hour` ∈ 0..48.
|
||
|
||
# Before (line 38):
|
||
|> validate_number(:forecast_hour, greater_than_or_equal_to: 0, less_than_or_equal_to: 18)
|
||
|
||
# After:
|
||
|> validate_number(:forecast_hour, greater_than_or_equal_to: 0, less_than_or_equal_to: 48)
|
||
```
|
||
|
||
`rover_live.ex` — the forecast hour selector clamp:
|
||
|
||
```elixir
|
||
# Before (line 185):
|
||
hour = v |> parse_int(@default_forecast_hour) |> clamp(0, 18)
|
||
|
||
# After:
|
||
hour = v |> parse_int(@default_forecast_hour) |> clamp(0, 48)
|
||
```
|
||
|
||
**Step 1: Make the changes**
|
||
|
||
No new tests needed — these are just ceiling bumps on existing validation. Existing tests exercise the validation path.
|
||
|
||
**Step 2: Run affected tests**
|
||
|
||
```bash
|
||
mix test test/microwaveprop/propagation/run_timing_test.exs test/microwaveprop_web/live/rover_live_test.exs
|
||
```
|
||
|
||
**Step 3: Commit**
|
||
|
||
```bash
|
||
git add lib/microwaveprop/propagation/run_timing.ex lib/microwaveprop_web/live/rover_live.ex
|
||
git commit -m "feat: bump forecast hour validation limits to 48h"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: Update comments and documentation strings
|
||
|
||
**Files:** ~15 files with "f01..f18", "18-hour forecast", or equivalent references in comments/docstrings.
|
||
|
||
These are cosmetic — no behavior change:
|
||
|
||
| File | What to update |
|
||
|---|---|
|
||
| `lib/microwaveprop/workers/propagation_grid_worker.ex:8,50` | "f01..f18" → "f01..f48" in comments/log |
|
||
| `lib/microwaveprop/weather/hrrr_client.ex:190` | Comment about f18 idx probe (no change needed — we still probe f18, it's still valid) |
|
||
| `lib/microwaveprop/propagation/path_compute.ex:9,73` | "18-hour forecast" → "48-hour forecast" |
|
||
| `lib/microwaveprop/propagation/score_cache.ex:175` | "18 h forward" → "48 h forward" |
|
||
| `lib/microwaveprop/weather.ex:931,985,1297` | Comment references to f01..f18 → f01..f48 |
|
||
| `lib/microwaveprop/weather/sounding_params.ex:24` | "f01..f18" → "f01..f48" |
|
||
| `lib/microwaveprop_web/router.ex:72` | "18-hour forecast" → "48-hour forecast" |
|
||
| `lib/microwaveprop_web/live/about_live.ex:81` | "18-hour forecast" → "48-hour forecast" |
|
||
| `lib/microwaveprop_web/live/api_docs_live.ex:841,845` | "18-hour" → "48-hour" |
|
||
| `lib/microwaveprop_web/live/skewt_live.ex:341` | "18 forecast hours" → "48 forecast hours" |
|
||
| `lib/microwaveprop_web/controllers/page_controller.ex:25` | "18-hour" → "48-hour" |
|
||
| `lib/microwaveprop_web/controllers/api/v1/score_controller.ex:8` | "18-hour" → "48-hour" |
|
||
| `rust/prop_grid_rs/src/pipeline.rs:1,72` | "f01..f18" → "f01..f48" in doc comments |
|
||
| `rust/prop_grid_rs/src/db.rs:20` | "f01..f18" → "f01..f48" in doc comment |
|
||
| `rust/prop_grid_rs/src/lib.rs:12` | "f01..f18" → "f01..f48" |
|
||
| `rust/prop_grid_rs/src/decoder.rs:2` | "f01..f18" → "f01..f48" |
|
||
| `rust/prop_grid_rs/src/sounding_params.rs:8` | "f01..f18" → "f01..f48" |
|
||
| `rust/prop_grid_rs/src/weather_scalar_file.rs:161` | "f01..f18" → "f01..f48" |
|
||
| `lib/microwaveprop/workers/hrdps_grid_worker.ex:25` | "18 forecast hours" → update context |
|
||
|
||
We update `path_live.ex`'s `forecast_chart` header on line 1133 — the dynamic string already adapts from `length(@forecast)`, but the alt text and surrounding prose says "18-Hour":
|
||
|
||
```elixir
|
||
# Line 1132-1133 in path_live.ex:
|
||
# "{length(@forecast)}-Hour Propagation Forecast"
|
||
# This is already dynamic — no change needed to the label itself.
|
||
# The section comment on line 907:
|
||
# Before: <%!-- 18-Hour Forecast --%>
|
||
# After: <%!-- 48-Hour Forecast --%>
|
||
```
|
||
|
||
**Step 1: Apply all comment/doc updates**
|
||
|
||
**Step 2: Run the full test suite**
|
||
|
||
```bash
|
||
mix test
|
||
```
|
||
|
||
**Step 3: Verify compilation is clean**
|
||
|
||
```bash
|
||
mix compile --warnings-as-errors
|
||
mix credo --strict
|
||
```
|
||
|
||
**Step 4: Commit**
|
||
|
||
```bash
|
||
git add -u
|
||
git commit -m "docs: update forecast horizon references from 18h to 48h"
|
||
```
|
||
|
||
---
|
||
|
||
## What does NOT need to change
|
||
|
||
- **Rust pipeline** (`pipeline.rs`, `db.rs`, `fetcher.rs`) — `forecast_hour: u8` handles f48 fine (max 255). The pipeline reads whatever `grid_tasks` rows are seeded and processes them identically regardless of forecast hour.
|
||
- **`HrrrClient.cycle_available?/1`** — still probes f18 idx (still the last hourly file, and a reliable signal that the cycle is fully published).
|
||
- **`ScoresFile`** — file format is per `(band, valid_time)`, agnostic to forecast hour. New hours just create new files.
|
||
- **`ProfilesFile`** — same, per-`valid_time` storage.
|
||
- **`retain_window` calls** — parameterized on `max_forecast_hour`, already dynamic; callers pass the constant.
|
||
- **`ScoreCache`** — horizon-aware via `hot_cache_window/0`, which reads the constant.
|
||
- **HRRR URL builder** (`hrrr_url/4`) — already handles any `forecast_hour` integer, pads to 2 digits.
|
||
- **`HrrrClient.cycle_available?`** — still probes f18 which is still in the set.
|
||
|
||
## Performance impact
|
||
|
||
Each forecast hour costs ~3–4 minutes of Rust pipeline wall time. The 10 additional 3-hourly steps add ~30–40 minutes to the hourly chain. Current chain runs ~60 minutes for 19 steps; this extends it to ~90–100 minutes. The f01–f18 results still land at the same latency; extended hours arrive later in the cycle.
|
||
|
||
## Rollout
|
||
|
||
1. Deploy the code change. The next hourly cron tick seeds 29 `grid_tasks` rows instead of 19.
|
||
2. Rust picks them up and processes f21–f48 alongside f01–f18.
|
||
3. Map timeline and path calculator forecast chart automatically show the extended horizon via the changed constant.
|
||
4. No migration, no feature flag, no downtime.
|