feat: extend HRRR forecast horizon from 18h to 48h
HRRR publishes f21-f48 at 3-hourly intervals in addition to the hourly f01-f18 we already fetch. This extends the grid_tasks seed from 19 rows (1 analysis + 18 forecast) to 29 rows (1 analysis + 18 hourly + 10 3-hourly), and bumps the UI forecast window constant from 18h to 48h so the map timeline and path calculator forecast chart automatically show the extended horizon. No Rust logic changes needed — the pipeline is data-driven and u8 forecast_hour already handles f48.
This commit is contained in:
parent
d67186176f
commit
aab3c3736c
29 changed files with 473 additions and 72 deletions
362
docs/plans/2026-06-10-extend-hrrr-forecast-to-48h.md
Normal file
362
docs/plans/2026-06-10-extend-hrrr-forecast-to-48h.md
Normal file
|
|
@ -0,0 +1,362 @@
|
||||||
|
# Extend HRRR Forecast Horizon to 48 Hours — Implementation Plan
|
||||||
|
|
||||||
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||||
|
|
||||||
|
**Goal:** Extend the propagation forecast from 18 hours to 48 hours using HRRR's own extended forecast hours (f21–f48 at 3-hourly intervals), without adding a new data source.
|
||||||
|
|
||||||
|
**Architecture:** The system is data-driven — whatever forecast hours are seeded into `grid_tasks`, Rust processes. Two constants gate the current 18-hour cap: `GridTaskEnqueuer.@max_forecast_hour` and `Propagation.@hrrr_forecast_horizon_hours`. Changing these, plus adjusting a few validation limits and comment strings, is the entire change. No Rust code changes needed.
|
||||||
|
|
||||||
|
**Tech Stack:** Elixir (Oban, Ecto, Phoenix LiveView), Rust (prop_grid_rs), HRRR NOAA S3
|
||||||
|
|
||||||
|
**Relationship to prior plan:** `docs/plans/2026-04-18-extended-horizon-forecasts.md` covers adding GFS as a separate data source for 10-day forecasts. This plan is the zero-new-infrastructure quick win — HRRR already publishes f21–f48, we just don't fetch them. The GFS plan is still the right long-term approach for multi-day planning; this extends the existing hourly chain at near-zero cost.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## HRRR forecast hour availability
|
||||||
|
|
||||||
|
| Range | Interval | Count | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| f00 | — | 1 | Analysis (Elixir-owned, unchanged) |
|
||||||
|
| f01–f18 | 1-hourly | 18 | What we fetch today |
|
||||||
|
| f21–f48 | 3-hourly | 10 | New: f21, f24, f27, f30, f33, f36, f39, f42, f45, f48 |
|
||||||
|
|
||||||
|
Total: 1 analysis + 18 hourly + 10 3-hourly = 29 `grid_tasks` rows per cycle (up from 19).
|
||||||
|
|
||||||
|
Accuracy degrades beyond f18 — HRRR is a 3 km convection-allowing model; small-scale features (ducting, refractivity gradients) lose fidelity at longer lead times. The 3-hourly output spacing reflects this. Accepted trade-off per user request.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Define the forecast hour list and update `GridTaskEnqueuer`
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `lib/microwaveprop/propagation/grid_task_enqueuer.ex`
|
||||||
|
|
||||||
|
**Changes:**
|
||||||
|
|
||||||
|
Replace the `@max_forecast_hour 18` module attribute with a list of all forecast hours Rust should process:
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Before (line 19):
|
||||||
|
@max_forecast_hour 18
|
||||||
|
|
||||||
|
# After:
|
||||||
|
@forecast_hours [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48]
|
||||||
|
```
|
||||||
|
|
||||||
|
Update `seed_with_analysis/2` (line 70) — the `for fh <- 1..@max_forecast_hour` comprehension:
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Before (line 70):
|
||||||
|
for fh <- 1..@max_forecast_hour do
|
||||||
|
|
||||||
|
# After:
|
||||||
|
for fh <- @forecast_hours do
|
||||||
|
```
|
||||||
|
|
||||||
|
Update `seed/2` (line 103) — same pattern:
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Before (line 103):
|
||||||
|
for fh <- 1..@max_forecast_hour do
|
||||||
|
|
||||||
|
# After:
|
||||||
|
for fh <- @forecast_hours do
|
||||||
|
```
|
||||||
|
|
||||||
|
Update `seed_current_hour/2` (lines 174, 192) — the clamp ceiling:
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Before (line 174):
|
||||||
|
# Forecast hour is clamped to `1..@max_forecast_hour`.
|
||||||
|
# (line 176):
|
||||||
|
# `run_time + @max_forecast_hour`, we cap there.
|
||||||
|
# (line 192):
|
||||||
|
|> min(@max_forecast_hour)
|
||||||
|
|
||||||
|
# After (line 174):
|
||||||
|
# Forecast hour is clamped to the available forecast hour list.
|
||||||
|
# (line 176):
|
||||||
|
# the max forecast hour, we cap there.
|
||||||
|
# (line 192):
|
||||||
|
|> min(Enum.max(@forecast_hours))
|
||||||
|
```
|
||||||
|
|
||||||
|
Update doc comments (lines 36–37, 92):
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Before (line 36-37):
|
||||||
|
# Seed one kind='analysis' row (f00) plus 18 kind='forecast' rows
|
||||||
|
# (f01..f18) for `run_time`.
|
||||||
|
|
||||||
|
# After:
|
||||||
|
# Seed one kind='analysis' row (f00) plus forecast rows
|
||||||
|
# (f01..f18 hourly, f21..f48 3-hourly) for `run_time`.
|
||||||
|
|
||||||
|
# Before (line 92):
|
||||||
|
# Legacy: seed only kind='forecast' rows (f01..f18).
|
||||||
|
|
||||||
|
# After:
|
||||||
|
# Legacy: seed only kind='forecast' rows.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 1: Update the test to expect the new forecast hour list**
|
||||||
|
|
||||||
|
File: `test/microwaveprop/propagation/grid_task_enqueuer_test.exs:22`
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Before:
|
||||||
|
assert rows |> Enum.map(& &1.forecast_hour) |> Enum.sort() == Enum.to_list(1..18)
|
||||||
|
|
||||||
|
# After:
|
||||||
|
@forecast_hours [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48]
|
||||||
|
assert rows |> Enum.map(& &1.forecast_hour) |> Enum.sort() == @forecast_hours
|
||||||
|
```
|
||||||
|
|
||||||
|
Update the worker test at `test/microwaveprop/workers/propagation_grid_worker_test.exs`:
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Line 7 (comment):
|
||||||
|
# Before: inserts 19 grid_tasks rows (1 analysis f00 + 18 forecast f01..f18)
|
||||||
|
# After: inserts grid_tasks rows (1 analysis f00 + forecast f01..f18 hourly, f21..f48 3-hourly)
|
||||||
|
|
||||||
|
# Line 70:
|
||||||
|
# Before: test "inserts 1 analysis + 18 forecast grid_tasks rows; no Oban fan-out" do
|
||||||
|
# After: test "inserts 1 analysis + forecast grid_tasks rows; no Oban fan-out" do
|
||||||
|
|
||||||
|
# Line 120:
|
||||||
|
# Before: assert kinds == %{"analysis" => 1, "forecast" => 18}
|
||||||
|
# After: assert kinds == %{"analysis" => 1, "forecast" => 28}
|
||||||
|
|
||||||
|
# Line 125:
|
||||||
|
# Before: assert forecast_fhs == Enum.to_list(1..18)
|
||||||
|
# After: assert forecast_fhs == @forecast_hours (with the module attribute defined above)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2: Run tests to verify they fail**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mix test test/microwaveprop/propagation/grid_task_enqueuer_test.exs test/microwaveprop/workers/propagation_grid_worker_test.exs
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: tests fail because they still expect 18 forecast hours.
|
||||||
|
|
||||||
|
**Step 3: Apply the `GridTaskEnqueuer` changes**
|
||||||
|
|
||||||
|
Make the code changes listed above.
|
||||||
|
|
||||||
|
**Step 4: Run tests to verify they pass**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mix test test/microwaveprop/propagation/grid_task_enqueuer_test.exs test/microwaveprop/workers/propagation_grid_worker_test.exs
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
**Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add lib/microwaveprop/propagation/grid_task_enqueuer.ex test/microwaveprop/propagation/grid_task_enqueuer_test.exs test/microwaveprop/workers/propagation_grid_worker_test.exs
|
||||||
|
git commit -m "feat: extend grid_tasks seeding to 48h forecast horizon"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: Update propagation forecast horizon constant
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `lib/microwaveprop/propagation.ex`
|
||||||
|
|
||||||
|
**Changes:**
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Before (line 261):
|
||||||
|
@hrrr_forecast_horizon_hours 18
|
||||||
|
|
||||||
|
# After:
|
||||||
|
@hrrr_forecast_horizon_hours 48
|
||||||
|
```
|
||||||
|
|
||||||
|
Update the comment on lines 258–260:
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Before:
|
||||||
|
# HRRR forecast horizon: f00..f18 covers the next 18 hours from cycle
|
||||||
|
# time. Anything beyond that in the score store is a leftover from a
|
||||||
|
# stale cycle and clutters the timeline without adding information.
|
||||||
|
|
||||||
|
# After:
|
||||||
|
# HRRR forecast horizon: f00..f48 covers the next 48 hours from cycle
|
||||||
|
# time (f01-f18 hourly, f21-f48 3-hourly). Anything beyond that in the
|
||||||
|
# score store is a leftover from a stale cycle and clutters the
|
||||||
|
# timeline without adding information.
|
||||||
|
```
|
||||||
|
|
||||||
|
Update the `@doc` for `hot_cache_window/0` (line 275):
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Before:
|
||||||
|
through HRRR's 18-hour forecast horizon.
|
||||||
|
|
||||||
|
# After:
|
||||||
|
through HRRR's 48-hour forecast horizon.
|
||||||
|
```
|
||||||
|
|
||||||
|
This single constant change automatically extends:
|
||||||
|
- The map timeline (`available_valid_times/1` → `hot_cache_window/0`)
|
||||||
|
- The path calculator forecast chart (`point_forecast/3` → `forecast_window/2`)
|
||||||
|
- The score cache ETS retention window (`hot_cache_window/0` used by `NotifyListener`)
|
||||||
|
|
||||||
|
**Step 1: Make the change**
|
||||||
|
|
||||||
|
**Step 2: Run propagation tests**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mix test test/microwaveprop/propagation_test.exs
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add lib/microwaveprop/propagation.ex
|
||||||
|
git commit -m "feat: extend forecast horizon window from 18h to 48h"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: Update validation limits and UI clamps
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `lib/microwaveprop/propagation/run_timing.ex:38`
|
||||||
|
- Modify: `lib/microwaveprop_web/live/rover_live.ex:185`
|
||||||
|
|
||||||
|
**Changes:**
|
||||||
|
|
||||||
|
`run_timing.ex` — bump the validation ceiling and update doc:
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Before (line 2-7 doc):
|
||||||
|
# `forecast_hour` ∈ 0..18.
|
||||||
|
|
||||||
|
# After:
|
||||||
|
# `forecast_hour` ∈ 0..48.
|
||||||
|
|
||||||
|
# Before (line 38):
|
||||||
|
|> validate_number(:forecast_hour, greater_than_or_equal_to: 0, less_than_or_equal_to: 18)
|
||||||
|
|
||||||
|
# After:
|
||||||
|
|> validate_number(:forecast_hour, greater_than_or_equal_to: 0, less_than_or_equal_to: 48)
|
||||||
|
```
|
||||||
|
|
||||||
|
`rover_live.ex` — the forecast hour selector clamp:
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Before (line 185):
|
||||||
|
hour = v |> parse_int(@default_forecast_hour) |> clamp(0, 18)
|
||||||
|
|
||||||
|
# After:
|
||||||
|
hour = v |> parse_int(@default_forecast_hour) |> clamp(0, 48)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 1: Make the changes**
|
||||||
|
|
||||||
|
No new tests needed — these are just ceiling bumps on existing validation. Existing tests exercise the validation path.
|
||||||
|
|
||||||
|
**Step 2: Run affected tests**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mix test test/microwaveprop/propagation/run_timing_test.exs test/microwaveprop_web/live/rover_live_test.exs
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add lib/microwaveprop/propagation/run_timing.ex lib/microwaveprop_web/live/rover_live.ex
|
||||||
|
git commit -m "feat: bump forecast hour validation limits to 48h"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: Update comments and documentation strings
|
||||||
|
|
||||||
|
**Files:** ~15 files with "f01..f18", "18-hour forecast", or equivalent references in comments/docstrings.
|
||||||
|
|
||||||
|
These are cosmetic — no behavior change:
|
||||||
|
|
||||||
|
| File | What to update |
|
||||||
|
|---|---|
|
||||||
|
| `lib/microwaveprop/workers/propagation_grid_worker.ex:8,50` | "f01..f18" → "f01..f48" in comments/log |
|
||||||
|
| `lib/microwaveprop/weather/hrrr_client.ex:190` | Comment about f18 idx probe (no change needed — we still probe f18, it's still valid) |
|
||||||
|
| `lib/microwaveprop/propagation/path_compute.ex:9,73` | "18-hour forecast" → "48-hour forecast" |
|
||||||
|
| `lib/microwaveprop/propagation/score_cache.ex:175` | "18 h forward" → "48 h forward" |
|
||||||
|
| `lib/microwaveprop/weather.ex:931,985,1297` | Comment references to f01..f18 → f01..f48 |
|
||||||
|
| `lib/microwaveprop/weather/sounding_params.ex:24` | "f01..f18" → "f01..f48" |
|
||||||
|
| `lib/microwaveprop_web/router.ex:72` | "18-hour forecast" → "48-hour forecast" |
|
||||||
|
| `lib/microwaveprop_web/live/about_live.ex:81` | "18-hour forecast" → "48-hour forecast" |
|
||||||
|
| `lib/microwaveprop_web/live/api_docs_live.ex:841,845` | "18-hour" → "48-hour" |
|
||||||
|
| `lib/microwaveprop_web/live/skewt_live.ex:341` | "18 forecast hours" → "48 forecast hours" |
|
||||||
|
| `lib/microwaveprop_web/controllers/page_controller.ex:25` | "18-hour" → "48-hour" |
|
||||||
|
| `lib/microwaveprop_web/controllers/api/v1/score_controller.ex:8` | "18-hour" → "48-hour" |
|
||||||
|
| `rust/prop_grid_rs/src/pipeline.rs:1,72` | "f01..f18" → "f01..f48" in doc comments |
|
||||||
|
| `rust/prop_grid_rs/src/db.rs:20` | "f01..f18" → "f01..f48" in doc comment |
|
||||||
|
| `rust/prop_grid_rs/src/lib.rs:12` | "f01..f18" → "f01..f48" |
|
||||||
|
| `rust/prop_grid_rs/src/decoder.rs:2` | "f01..f18" → "f01..f48" |
|
||||||
|
| `rust/prop_grid_rs/src/sounding_params.rs:8` | "f01..f18" → "f01..f48" |
|
||||||
|
| `rust/prop_grid_rs/src/weather_scalar_file.rs:161` | "f01..f18" → "f01..f48" |
|
||||||
|
| `lib/microwaveprop/workers/hrdps_grid_worker.ex:25` | "18 forecast hours" → update context |
|
||||||
|
|
||||||
|
We update `path_live.ex`'s `forecast_chart` header on line 1133 — the dynamic string already adapts from `length(@forecast)`, but the alt text and surrounding prose says "18-Hour":
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
# Line 1132-1133 in path_live.ex:
|
||||||
|
# "{length(@forecast)}-Hour Propagation Forecast"
|
||||||
|
# This is already dynamic — no change needed to the label itself.
|
||||||
|
# The section comment on line 907:
|
||||||
|
# Before: <%!-- 18-Hour Forecast --%>
|
||||||
|
# After: <%!-- 48-Hour Forecast --%>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 1: Apply all comment/doc updates**
|
||||||
|
|
||||||
|
**Step 2: Run the full test suite**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mix test
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3: Verify compilation is clean**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mix compile --warnings-as-errors
|
||||||
|
mix credo --strict
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 4: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add -u
|
||||||
|
git commit -m "docs: update forecast horizon references from 18h to 48h"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What does NOT need to change
|
||||||
|
|
||||||
|
- **Rust pipeline** (`pipeline.rs`, `db.rs`, `fetcher.rs`) — `forecast_hour: u8` handles f48 fine (max 255). The pipeline reads whatever `grid_tasks` rows are seeded and processes them identically regardless of forecast hour.
|
||||||
|
- **`HrrrClient.cycle_available?/1`** — still probes f18 idx (still the last hourly file, and a reliable signal that the cycle is fully published).
|
||||||
|
- **`ScoresFile`** — file format is per `(band, valid_time)`, agnostic to forecast hour. New hours just create new files.
|
||||||
|
- **`ProfilesFile`** — same, per-`valid_time` storage.
|
||||||
|
- **`retain_window` calls** — parameterized on `max_forecast_hour`, already dynamic; callers pass the constant.
|
||||||
|
- **`ScoreCache`** — horizon-aware via `hot_cache_window/0`, which reads the constant.
|
||||||
|
- **HRRR URL builder** (`hrrr_url/4`) — already handles any `forecast_hour` integer, pads to 2 digits.
|
||||||
|
- **`HrrrClient.cycle_available?`** — still probes f18 which is still in the set.
|
||||||
|
|
||||||
|
## Performance impact
|
||||||
|
|
||||||
|
Each forecast hour costs ~3–4 minutes of Rust pipeline wall time. The 10 additional 3-hourly steps add ~30–40 minutes to the hourly chain. Current chain runs ~60 minutes for 19 steps; this extends it to ~90–100 minutes. The f01–f18 results still land at the same latency; extended hours arrive later in the cycle.
|
||||||
|
|
||||||
|
## Rollout
|
||||||
|
|
||||||
|
1. Deploy the code change. The next hourly cron tick seeds 29 `grid_tasks` rows instead of 19.
|
||||||
|
2. Rust picks them up and processes f21–f48 alongside f01–f18.
|
||||||
|
3. Map timeline and path calculator forecast chart automatically show the extended horizon via the changed constant.
|
||||||
|
4. No migration, no feature flag, no downtime.
|
||||||
|
|
@ -255,10 +255,11 @@ defmodule Microwaveprop.Propagation do
|
||||||
the past, but always includes the most recent valid_time so there's
|
the past, but always includes the most recent valid_time so there's
|
||||||
always data to display.
|
always data to display.
|
||||||
"""
|
"""
|
||||||
# HRRR forecast horizon: f00..f18 covers the next 18 hours from cycle
|
# HRRR forecast horizon: f00..f48 covers the next 48 hours from cycle
|
||||||
# time. Anything beyond that in the score store is a leftover from a
|
# time (f01-f18 hourly, f21-f48 3-hourly). Anything beyond that in the
|
||||||
# stale cycle and clutters the timeline without adding information.
|
# score store is a leftover from a stale cycle and clutters the
|
||||||
@hrrr_forecast_horizon_hours 18
|
# timeline without adding information.
|
||||||
|
@hrrr_forecast_horizon_hours 48
|
||||||
|
|
||||||
@spec available_valid_times(non_neg_integer()) :: [DateTime.t()]
|
@spec available_valid_times(non_neg_integer()) :: [DateTime.t()]
|
||||||
def available_valid_times(band_mhz) do
|
def available_valid_times(band_mhz) do
|
||||||
|
|
@ -272,7 +273,7 @@ defmodule Microwaveprop.Propagation do
|
||||||
|
|
||||||
@doc """
|
@doc """
|
||||||
The active forecast window for the `/map` UI: one hour in the past
|
The active forecast window for the `/map` UI: one hour in the past
|
||||||
through HRRR's 18-hour forecast horizon. Used by `NotifyListener` to
|
through HRRR's 48-hour forecast horizon. Used by `NotifyListener` to
|
||||||
bound ETS growth — long-horizon GEFS `.prop` files on disk must not
|
bound ETS growth — long-horizon GEFS `.prop` files on disk must not
|
||||||
balloon `propagation_score_cache` past the memory the UI actually
|
balloon `propagation_score_cache` past the memory the UI actually
|
||||||
reads.
|
reads.
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,9 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuer do
|
||||||
|
|
||||||
require Logger
|
require Logger
|
||||||
|
|
||||||
@max_forecast_hour 18
|
# HRRR forecast hours: f01-f18 hourly, then f21-f48 3-hourly.
|
||||||
|
# Rust processes whatever rows are seeded — no code changes needed there.
|
||||||
|
@forecast_hours [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48]
|
||||||
|
|
||||||
# A `running` row older than this is an orphan: the Rust worker that
|
# A `running` row older than this is an orphan: the Rust worker that
|
||||||
# claimed it was SIGKILLed / OOMed / evicted before it could emit a
|
# claimed it was SIGKILLed / OOMed / evicted before it could emit a
|
||||||
|
|
@ -33,9 +35,10 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuer do
|
||||||
@max_reclaim_attempts 5
|
@max_reclaim_attempts 5
|
||||||
|
|
||||||
@doc """
|
@doc """
|
||||||
Seed one kind='analysis' row (f00) plus 18 kind='forecast' rows
|
Seed one kind='analysis' row (f00) plus kind='forecast' rows
|
||||||
(f01..f18) for `run_time`. This is the Phase 3 Stream A cutover
|
(f01..f18 hourly, f21..f48 3-hourly) for `run_time`. This is the
|
||||||
shape: Rust owns the entire chain end-to-end.
|
Phase 3 Stream A cutover shape: Rust owns the entire chain
|
||||||
|
end-to-end.
|
||||||
|
|
||||||
## Options
|
## Options
|
||||||
|
|
||||||
|
|
@ -67,7 +70,7 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuer do
|
||||||
}
|
}
|
||||||
|
|
||||||
forecast_rows =
|
forecast_rows =
|
||||||
for fh <- 1..@max_forecast_hour do
|
for fh <- @forecast_hours do
|
||||||
%{
|
%{
|
||||||
id: Ecto.UUID.bingenerate(),
|
id: Ecto.UUID.bingenerate(),
|
||||||
run_time: run_time,
|
run_time: run_time,
|
||||||
|
|
@ -89,7 +92,7 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuer do
|
||||||
end
|
end
|
||||||
|
|
||||||
@doc """
|
@doc """
|
||||||
Legacy: seed only kind='forecast' rows (f01..f18). Retained for
|
Legacy: seed only kind='forecast' rows. Retained for
|
||||||
tooling that needs to re-seed the forecast lane without touching
|
tooling that needs to re-seed the forecast lane without touching
|
||||||
the analysis row. `seed_with_analysis/2` is the production path.
|
the analysis row. `seed_with_analysis/2` is the production path.
|
||||||
"""
|
"""
|
||||||
|
|
@ -100,7 +103,7 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuer do
|
||||||
now = DateTime.truncate(DateTime.utc_now(), :microsecond)
|
now = DateTime.truncate(DateTime.utc_now(), :microsecond)
|
||||||
|
|
||||||
rows =
|
rows =
|
||||||
for fh <- 1..@max_forecast_hour do
|
for fh <- @forecast_hours do
|
||||||
%{
|
%{
|
||||||
id: Ecto.UUID.bingenerate(),
|
id: Ecto.UUID.bingenerate(),
|
||||||
run_time: run_time,
|
run_time: run_time,
|
||||||
|
|
@ -171,9 +174,9 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuer do
|
||||||
the f01..f48 chain isn't worth the wgrib2 cost (each HRDPS chain step
|
the f01..f48 chain isn't worth the wgrib2 cost (each HRDPS chain step
|
||||||
is ~20 min wall time at the production point density).
|
is ~20 min wall time at the production point density).
|
||||||
|
|
||||||
Forecast hour is clamped to `1..@max_forecast_hour`. When `now` lands
|
Forecast hour is clamped to the available forecast hour list. When
|
||||||
before `run_time + 1h`, we still seed `fh=1`; when it's past
|
`now` lands before `run_time + 1h`, we still seed `fh=1`; when it's
|
||||||
`run_time + @max_forecast_hour`, we cap there. The Rust worker rejects
|
past the max forecast hour, we cap there. The Rust worker rejects
|
||||||
`forecast_hour=0` (`F00Reserved`) so we never seed the analysis hour
|
`forecast_hour=0` (`F00Reserved`) so we never seed the analysis hour
|
||||||
through this path.
|
through this path.
|
||||||
"""
|
"""
|
||||||
|
|
@ -189,7 +192,7 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuer do
|
||||||
|> DateTime.diff(run_time, :second)
|
|> DateTime.diff(run_time, :second)
|
||||||
|> div(3600)
|
|> div(3600)
|
||||||
|> max(1)
|
|> max(1)
|
||||||
|> min(@max_forecast_hour)
|
|> min(Enum.max(@forecast_hours))
|
||||||
|
|
||||||
row = %{
|
row = %{
|
||||||
id: Ecto.UUID.bingenerate(),
|
id: Ecto.UUID.bingenerate(),
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@ defmodule Microwaveprop.Propagation.PathCompute do
|
||||||
same pipeline `/path` does: terrain analysis (ITU-R P.526), HRRR
|
same pipeline `/path` does: terrain analysis (ITU-R P.526), HRRR
|
||||||
profile lookup at 9 evenly-spaced points, sounding & ionosphere
|
profile lookup at 9 evenly-spaced points, sounding & ionosphere
|
||||||
readouts, native HRRR duct info, composite scoring, loss + power
|
readouts, native HRRR duct info, composite scoring, loss + power
|
||||||
budgets, and the 18-hour propagation forecast at the path midpoint.
|
budgets, and the 48-hour propagation forecast at the path midpoint.
|
||||||
|
|
||||||
Used by both:
|
Used by both:
|
||||||
* the live `MicrowavepropWeb.PathLive` page (live recompute)
|
* the live `MicrowavepropWeb.PathLive` page (live recompute)
|
||||||
|
|
@ -70,7 +70,7 @@ defmodule Microwaveprop.Propagation.PathCompute do
|
||||||
"Loading sounding & duct data",
|
"Loading sounding & duct data",
|
||||||
"Scoring propagation conditions",
|
"Scoring propagation conditions",
|
||||||
"Computing link & power budget",
|
"Computing link & power budget",
|
||||||
"Loading 18-hour forecast",
|
"Loading 48-hour forecast",
|
||||||
"Loading ionosphere data"
|
"Loading ionosphere data"
|
||||||
]
|
]
|
||||||
@total_stages length(@stages)
|
@total_stages length(@stages)
|
||||||
|
|
|
||||||
|
|
@ -298,7 +298,7 @@ defmodule Microwaveprop.Propagation.ProfilesFile do
|
||||||
|
|
||||||
@doc """
|
@doc """
|
||||||
Every persisted valid_time, sorted ascending. Used to build the
|
Every persisted valid_time, sorted ascending. Used to build the
|
||||||
/weather forecast timeline from the on-disk f00..f18 profile files.
|
/weather forecast timeline from the on-disk f00..f48 profile files.
|
||||||
"""
|
"""
|
||||||
@spec list_valid_times() :: [DateTime.t()]
|
@spec list_valid_times() :: [DateTime.t()]
|
||||||
def list_valid_times do
|
def list_valid_times do
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ defmodule Microwaveprop.Propagation.RunTiming do
|
||||||
how long that hour's fetch + score + persist cycle took.
|
how long that hour's fetch + score + persist cycle took.
|
||||||
|
|
||||||
Rows are keyed by `(run_time, forecast_hour)`. `run_time` is the HRRR
|
Rows are keyed by `(run_time, forecast_hour)`. `run_time` is the HRRR
|
||||||
model cycle and `forecast_hour` ∈ 0..18.
|
model cycle and `forecast_hour` ∈ 0..48.
|
||||||
"""
|
"""
|
||||||
use Ecto.Schema
|
use Ecto.Schema
|
||||||
|
|
||||||
|
|
@ -35,7 +35,7 @@ defmodule Microwaveprop.Propagation.RunTiming do
|
||||||
row
|
row
|
||||||
|> cast(attrs, @required ++ @optional)
|
|> cast(attrs, @required ++ @optional)
|
||||||
|> validate_required(@required)
|
|> validate_required(@required)
|
||||||
|> validate_number(:forecast_hour, greater_than_or_equal_to: 0, less_than_or_equal_to: 18)
|
|> validate_number(:forecast_hour, greater_than_or_equal_to: 0, less_than_or_equal_to: 48)
|
||||||
|> validate_number(:duration_ms, greater_than_or_equal_to: 0)
|
|> validate_number(:duration_ms, greater_than_or_equal_to: 0)
|
||||||
|> unique_constraint([:run_time, :forecast_hour])
|
|> unique_constraint([:run_time, :forecast_hour])
|
||||||
end
|
end
|
||||||
|
|
|
||||||
|
|
@ -172,7 +172,7 @@ defmodule Microwaveprop.Propagation.ScoreCache do
|
||||||
@doc """
|
@doc """
|
||||||
Delete every entry whose `valid_time` falls outside the inclusive
|
Delete every entry whose `valid_time` falls outside the inclusive
|
||||||
window `[past_cutoff, future_cutoff]`. Used to bound the cache to
|
window `[past_cutoff, future_cutoff]`. Used to bound the cache to
|
||||||
the active forecast horizon (past ~1 h + HRRR's 18 h forward) so
|
the active forecast horizon (past ~1 h + HRRR's 48 h forward) so
|
||||||
long-horizon GEFS valid_times never accumulate in ETS.
|
long-horizon GEFS valid_times never accumulate in ETS.
|
||||||
"""
|
"""
|
||||||
@spec prune_outside_window(DateTime.t(), DateTime.t()) :: non_neg_integer()
|
@spec prune_outside_window(DateTime.t(), DateTime.t()) :: non_neg_integer()
|
||||||
|
|
|
||||||
|
|
@ -928,7 +928,7 @@ defmodule Microwaveprop.Weather do
|
||||||
|
|
||||||
@doc """
|
@doc """
|
||||||
All persisted weather valid_times sorted ascending. The grid worker
|
All persisted weather valid_times sorted ascending. The grid worker
|
||||||
writes a ProfilesFile for every forecast hour (f00..f18), and the
|
writes a ProfilesFile for every forecast hour (f00..f18 hourly, f21..f48 3-hourly), and the
|
||||||
derived `ScalarFile` mirrors that for any hour that has been
|
derived `ScalarFile` mirrors that for any hour that has been
|
||||||
materialized. We union both so the timeline survives an aggressive
|
materialized. We union both so the timeline survives an aggressive
|
||||||
retention sweep on either side as long as one artifact remains.
|
retention sweep on either side as long as one artifact remains.
|
||||||
|
|
@ -982,7 +982,7 @@ defmodule Microwaveprop.Weather do
|
||||||
Returns `[]` if no profile file exists for that valid_time.
|
Returns `[]` if no profile file exists for that valid_time.
|
||||||
|
|
||||||
Deliberately does NOT write forecast-hour grids back into `GridCache`
|
Deliberately does NOT write forecast-hour grids back into `GridCache`
|
||||||
on a miss: caching 18 forecast hours × 92k points would add ~300 MB
|
on a miss: caching 48 forecast hours × 92k points would add ~300 MB
|
||||||
per pod. The ProfilesFile read is a single ~2 MB ETF decode per scrub,
|
per pod. The ProfilesFile read is a single ~2 MB ETF decode per scrub,
|
||||||
which is fast enough for a user click.
|
which is fast enough for a user click.
|
||||||
"""
|
"""
|
||||||
|
|
@ -1294,7 +1294,7 @@ defmodule Microwaveprop.Weather do
|
||||||
# unchanged. `surface_refractivity` and `min_refractivity_gradient`
|
# unchanged. `surface_refractivity` and `min_refractivity_gradient`
|
||||||
# aren't persisted by Rust — they flow through from the per-cell
|
# aren't persisted by Rust — they flow through from the per-cell
|
||||||
# sounding derivation; `refractivity_gradient` falls back to the
|
# sounding derivation; `refractivity_gradient` falls back to the
|
||||||
# `native_min_gradient` scalar the Rust f01..f18 pipeline does write.
|
# `native_min_gradient` scalar the Rust f01..f48 pipeline does write.
|
||||||
sounding = derive_sounding(profile[:profile])
|
sounding = derive_sounding(profile[:profile])
|
||||||
|
|
||||||
row = %{
|
row = %{
|
||||||
|
|
|
||||||
|
|
@ -21,7 +21,7 @@ defmodule Microwaveprop.Weather.SoundingParams do
|
||||||
* Legacy shape — string keys `"pres"` / `"hght"` / `"tmpc"` / `"dwpc"`
|
* Legacy shape — string keys `"pres"` / `"hght"` / `"tmpc"` / `"dwpc"`
|
||||||
(Elixir's HrrrClient, NARR, GEFS, UWYO clients).
|
(Elixir's HrrrClient, NARR, GEFS, UWYO clients).
|
||||||
* Rust shape — `"pres_mb"` / `"hght_m"` / `"tmpc"` / `"dwpc"` (string
|
* Rust shape — `"pres_mb"` / `"hght_m"` / `"tmpc"` / `"dwpc"` (string
|
||||||
or atom keys from the Rust hrrr_points worker and the f01..f18
|
or atom keys from the Rust hrrr_points worker and the f01..f48
|
||||||
ProfilesFile cells).
|
ProfilesFile cells).
|
||||||
|
|
||||||
Both are normalized at entry to the canonical legacy shape so the
|
Both are normalized at entry to the canonical legacy shape so the
|
||||||
|
|
|
||||||
|
|
@ -22,7 +22,7 @@ defmodule Microwaveprop.Workers.HrdpsGridWorker do
|
||||||
|
|
||||||
Seeds exactly one forecast row per fire — the one whose valid_time
|
Seeds exactly one forecast row per fire — the one whose valid_time
|
||||||
matches the current hour. The Canadian propagation/weather UI only
|
matches the current hour. The Canadian propagation/weather UI only
|
||||||
surfaces the present moment, so producing 18 forecast hours per cycle
|
surfaces the present moment, so producing forecast hours per cycle
|
||||||
was wasted wgrib2 work (~20 min/chain step at production point
|
was wasted wgrib2 work (~20 min/chain step at production point
|
||||||
density). If forecast hours come back into scope later, swap the
|
density). If forecast hours come back into scope later, swap the
|
||||||
enqueuer call back to `GridTaskEnqueuer.seed_with_analysis/2`.
|
enqueuer call back to `GridTaskEnqueuer.seed_with_analysis/2`.
|
||||||
|
|
|
||||||
|
|
@ -4,8 +4,8 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
|
||||||
|
|
||||||
Post-Phase-3-cutover, Elixir no longer runs any fetch/decode/score
|
Post-Phase-3-cutover, Elixir no longer runs any fetch/decode/score
|
||||||
work for the propagation grid. The hourly cron fires this worker
|
work for the propagation grid. The hourly cron fires this worker
|
||||||
with empty args, and it inserts 19 `grid_tasks` rows (1 analysis
|
with empty args, and it inserts grid_tasks rows (1 analysis
|
||||||
f00 + 18 forecast f01..f18) for Rust to drain. Rust owns everything
|
f00 + forecast f01..f18 hourly, f21..f48 3-hourly) for Rust to drain. Rust owns everything
|
||||||
from there: HRRR fetch, wgrib2 decode, native-level duct merge,
|
from there: HRRR fetch, wgrib2 decode, native-level duct merge,
|
||||||
NEXRAD composite, commercial-link degradation, band scoring, and
|
NEXRAD composite, commercial-link degradation, band scoring, and
|
||||||
both the ProfilesFile (MessagePack) and the per-band score files.
|
both the ProfilesFile (MessagePack) and the per-band score files.
|
||||||
|
|
@ -47,7 +47,9 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
|
||||||
|
|
||||||
run_time = pick_run_time(DateTime.utc_now())
|
run_time = pick_run_time(DateTime.utc_now())
|
||||||
|
|
||||||
Logger.info("PropagationGrid: seeding chain run_time=#{run_time} (f00 analysis + f01-f18 forecasts via grid_tasks)")
|
Logger.info(
|
||||||
|
"PropagationGrid: seeding chain run_time=#{run_time} (f00 analysis + f01-f18 hourly, f21-f48 3-hourly forecasts via grid_tasks)"
|
||||||
|
)
|
||||||
|
|
||||||
case GridTaskEnqueuer.seed_with_analysis(run_time) do
|
case GridTaskEnqueuer.seed_with_analysis(run_time) do
|
||||||
{:ok, count} ->
|
{:ok, count} ->
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,7 @@ defmodule MicrowavepropWeb.Api.V1.ScoreController do
|
||||||
* `GET /api/v1/scores?band=10000&lat=32.9&lon=-97.0&valid_time=ISO`
|
* `GET /api/v1/scores?band=10000&lat=32.9&lon=-97.0&valid_time=ISO`
|
||||||
returns the score + factor breakdown at a single grid point.
|
returns the score + factor breakdown at a single grid point.
|
||||||
* `GET /api/v1/forecast?band=10000&lat=32.9&lon=-97.0`
|
* `GET /api/v1/forecast?band=10000&lat=32.9&lon=-97.0`
|
||||||
returns the 18-hour score timeline at that point.
|
returns the 48-hour score timeline at that point.
|
||||||
* `GET /api/v1/scores/bands` returns the bands the engine scores for.
|
* `GET /api/v1/scores/bands` returns the bands the engine scores for.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -22,7 +22,7 @@ defmodule MicrowavepropWeb.PageController do
|
||||||
|
|
||||||
## Tools and maps
|
## Tools and maps
|
||||||
|
|
||||||
- [Propagation map](/map) — real-time CONUS propagation scores with 18-hour forecast overlay.
|
- [Propagation map](/map) — real-time CONUS propagation scores with 48-hour forecast overlay.
|
||||||
- [Weather map](/weather) — HRRR weather data overlay (temperature, dewpoint, refractivity, ducting).
|
- [Weather map](/weather) — HRRR weather data overlay (temperature, dewpoint, refractivity, ducting).
|
||||||
- [Weather map (Canada)](/weather-ca) — HRDPS/RDPS-based weather for Canadian coverage.
|
- [Weather map (Canada)](/weather-ca) — HRDPS/RDPS-based weather for Canadian coverage.
|
||||||
- [Contact map](/contacts/map) — geographic view of 58,000+ recorded QSOs.
|
- [Contact map](/contacts/map) — geographic view of 58,000+ recorded QSOs.
|
||||||
|
|
|
||||||
|
|
@ -78,7 +78,7 @@ defmodule MicrowavepropWeb.AboutLive do
|
||||||
water for every grid cell. ASOS surface obs, 12-hourly upper-air
|
water for every grid cell. ASOS surface obs, 12-hourly upper-air
|
||||||
soundings, and gridded IEMRE reanalysis fill in the gaps. A
|
soundings, and gridded IEMRE reanalysis fill in the gaps. A
|
||||||
10-factor composite score is written out for every 0.125° cell
|
10-factor composite score is written out for every 0.125° cell
|
||||||
across CONUS, for each hour of an 18-hour forecast.
|
across CONUS, for each hour of a 48-hour forecast.
|
||||||
</li>
|
</li>
|
||||||
<li>
|
<li>
|
||||||
<strong>The contacts.</strong>
|
<strong>The contacts.</strong>
|
||||||
|
|
|
||||||
|
|
@ -838,11 +838,11 @@ defmodule MicrowavepropWeb.ApiDocsLive do
|
||||||
</.endpoint>
|
</.endpoint>
|
||||||
|
|
||||||
<.endpoint id="forecast" method="GET" path="/forecast">
|
<.endpoint id="forecast" method="GET" path="/forecast">
|
||||||
<:summary>18-hour score timeline</:summary>
|
<:summary>48-hour score timeline</:summary>
|
||||||
<:body>
|
<:body>
|
||||||
<p>
|
<p>
|
||||||
Returns the score at one grid point across the available
|
Returns the score at one grid point across the available
|
||||||
forecast horizon (currently 18 hours). Same <code>band</code>
|
forecast horizon (currently 48 hours). Same <code>band</code>
|
||||||
+ <code>lat</code>
|
+ <code>lat</code>
|
||||||
+ <code>lon</code>
|
+ <code>lon</code>
|
||||||
parameters as <code>/scores</code>.
|
parameters as <code>/scores</code>.
|
||||||
|
|
|
||||||
|
|
@ -904,7 +904,7 @@ defmodule MicrowavepropWeb.PathLive do
|
||||||
<.ionosphere_readout readout={@result.ionosphere} band_mhz={@result.band_mhz} />
|
<.ionosphere_readout readout={@result.ionosphere} band_mhz={@result.band_mhz} />
|
||||||
<% end %>
|
<% end %>
|
||||||
|
|
||||||
<%!-- 18-Hour Forecast --%>
|
<%!-- 48-Hour Forecast --%>
|
||||||
<%= if @result.forecast != [] do %>
|
<%= if @result.forecast != [] do %>
|
||||||
<.forecast_chart forecast={@result.forecast} band_config={@result.band_config} />
|
<.forecast_chart forecast={@result.forecast} band_config={@result.band_config} />
|
||||||
<% end %>
|
<% end %>
|
||||||
|
|
|
||||||
|
|
@ -182,7 +182,7 @@ defmodule MicrowavepropWeb.RoverLive do
|
||||||
end
|
end
|
||||||
|
|
||||||
def handle_event("set_forecast_hour", %{"value" => v}, socket) do
|
def handle_event("set_forecast_hour", %{"value" => v}, socket) do
|
||||||
hour = v |> parse_int(@default_forecast_hour) |> clamp(0, 18)
|
hour = v |> parse_int(@default_forecast_hour) |> clamp(0, 48)
|
||||||
{:noreply, assign(socket, forecast_hour: hour)}
|
{:noreply, assign(socket, forecast_hour: hour)}
|
||||||
end
|
end
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -338,7 +338,7 @@ defmodule MicrowavepropWeb.SkewtLive do
|
||||||
end
|
end
|
||||||
|
|
||||||
defp available_valid_times(lat, lon) do
|
defp available_valid_times(lat, lon) do
|
||||||
# The chain produces analysis + 18 forecast hours every wall-clock
|
# The chain produces analysis + forecast hours every wall-clock
|
||||||
# hour. By the time the next chain finishes the previous chain's
|
# hour. By the time the next chain finishes the previous chain's
|
||||||
# analysis is up to 70 min old, and during a missed cycle it can be
|
# analysis is up to 70 min old, and during a missed cycle it can be
|
||||||
# ~2 h old. A 1-hour past cutoff would leave the page empty while
|
# ~2 h old. A 1-hour past cutoff would leave the page empty while
|
||||||
|
|
|
||||||
|
|
@ -69,7 +69,7 @@ defmodule MicrowavepropWeb.Router do
|
||||||
|
|
||||||
## Primary Resources
|
## Primary Resources
|
||||||
|
|
||||||
- [Propagation map](/map) — real-time CONUS propagation scores + 18-hour forecast.
|
- [Propagation map](/map) — real-time CONUS propagation scores + 48-hour forecast.
|
||||||
- [Algorithm documentation](/algo) — full methodology, factor weights, ITU-R references (also available as `text/markdown`).
|
- [Algorithm documentation](/algo) — full methodology, factor weights, ITU-R references (also available as `text/markdown`).
|
||||||
- [Contact database](/contacts) — 58,000+ historical QSOs used for calibration.
|
- [Contact database](/contacts) — 58,000+ historical QSOs used for calibration.
|
||||||
- [Contact map](/contacts/map) — geographic view of all recorded contacts.
|
- [Contact map](/contacts/map) — geographic view of all recorded contacts.
|
||||||
|
|
|
||||||
|
|
@ -17,7 +17,7 @@ pub const NOTIFY_CHANNEL: &str = "propagation_ready";
|
||||||
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
pub enum TaskKind {
|
pub enum TaskKind {
|
||||||
/// f01..f18 — fetch + decode + score the forecast-hour grid. This is
|
/// f01..f48 — fetch + decode + score the forecast-hour grid. This is
|
||||||
/// what Rust has owned since Phase 2 cutover.
|
/// what Rust has owned since Phase 2 cutover.
|
||||||
Forecast,
|
Forecast,
|
||||||
/// f00 enrichment — adds native-level duct merge, NEXRAD composite,
|
/// f00 enrichment — adds native-level duct merge, NEXRAD composite,
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
//! wgrib2 subprocess wrapper. 1:1 port of the hot paths in
|
//! wgrib2 subprocess wrapper. 1:1 port of the hot paths in
|
||||||
//! `lib/microwaveprop/weather/grib2/wgrib2.ex` used by f01..f18:
|
//! `lib/microwaveprop/weather/grib2/wgrib2.ex` used by f01..f48:
|
||||||
//! * `extract_grid` (write GRIB2 to tmp → `wgrib2 -lola … bin` → parse)
|
//! * `extract_grid` (write GRIB2 to tmp → `wgrib2 -lola … bin` → parse)
|
||||||
//! * inventory parsing from stdout
|
//! * inventory parsing from stdout
|
||||||
//! * `parse_lola_binary` (Fortran-unformatted IEEE 754 little-endian f32)
|
//! * `parse_lola_binary` (Fortran-unformatted IEEE 754 little-endian f32)
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,7 @@
|
||||||
//! tests can compare behavior 1:1. Module boundaries chosen so failures
|
//! tests can compare behavior 1:1. Module boundaries chosen so failures
|
||||||
//! during TDD fall into the smallest possible blast radius.
|
//! during TDD fall into the smallest possible blast radius.
|
||||||
//!
|
//!
|
||||||
//! Post-Phase-3: f00 analysis and f01..f18 forecasts both live here.
|
//! Post-Phase-3: f00 analysis and f01..f48 forecasts both live here.
|
||||||
//! Port targets:
|
//! Port targets:
|
||||||
//! band_config ← lib/microwaveprop/propagation/band_config.ex
|
//! band_config ← lib/microwaveprop/propagation/band_config.ex
|
||||||
//! region ← lib/microwaveprop/propagation/region.ex
|
//! region ← lib/microwaveprop/propagation/region.ex
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
//! End-to-end f01..f18 chain step: fetch surface + pressure GRIBs, decode,
|
//! End-to-end f01..f48 chain step: fetch surface + pressure GRIBs, decode,
|
||||||
//! build per-cell `Conditions`, score every band, write score-grid files
|
//! build per-cell `Conditions`, score every band, write score-grid files
|
||||||
//! (`<band>/<iso>.prop`) to the shared scores directory.
|
//! (`<band>/<iso>.prop`) to the shared scores directory.
|
||||||
//!
|
//!
|
||||||
|
|
@ -69,7 +69,7 @@ pub struct ChainStepStats {
|
||||||
pub point_count: u32,
|
pub point_count: u32,
|
||||||
pub band_count: u32,
|
pub band_count: u32,
|
||||||
/// Number of cells written to the per-valid_time profile file.
|
/// Number of cells written to the per-valid_time profile file.
|
||||||
/// Always `point_count` after the f01..f18 profile-file write was
|
/// Always `point_count` after the f01..f48 profile-file write was
|
||||||
/// added on 2026-04-25 — kept as a separate field so the log line
|
/// added on 2026-04-25 — kept as a separate field so the log line
|
||||||
/// is symmetric with the analysis step's `profile_cells_written`
|
/// is symmetric with the analysis step's `profile_cells_written`
|
||||||
/// in `RunStepStats`, and so a future failure mode that writes
|
/// in `RunStepStats`, and so a future failure mode that writes
|
||||||
|
|
@ -1026,7 +1026,7 @@ pub async fn run_analysis_step(
|
||||||
|
|
||||||
// Same wire format as Elixir's `ScalarFile`. See run_chain_step
|
// Same wire format as Elixir's `ScalarFile`. See run_chain_step
|
||||||
// (forecast hours) for the full rationale — both code paths go
|
// (forecast hours) for the full rationale — both code paths go
|
||||||
// through the same writer so f00 and f01..f18 land identical
|
// through the same writer so f00 and f01..f48 land identical
|
||||||
// chunked artifacts on NFS.
|
// chunked artifacts on NFS.
|
||||||
let scores_dir_for_scalars = scores_dir_owned.clone();
|
let scores_dir_for_scalars = scores_dir_owned.clone();
|
||||||
let scalar_future = {
|
let scalar_future = {
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,7 @@
|
||||||
//!
|
//!
|
||||||
//! The full Elixir module derives CAPE/LI/K-index/ducts — those are
|
//! The full Elixir module derives CAPE/LI/K-index/ducts — those are
|
||||||
//! used for contact enrichment and f00's native-duct merge, neither of
|
//! used for contact enrichment and f00's native-duct merge, neither of
|
||||||
//! which is in the Rust f01..f18 scope.
|
//! which is in the Rust f01..f48 scope.
|
||||||
|
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
pub struct Level {
|
pub struct Level {
|
||||||
|
|
|
||||||
|
|
@ -158,7 +158,7 @@ pub fn derive_row(
|
||||||
let inversion_strength = compute_inversion_strength(&sorted);
|
let inversion_strength = compute_inversion_strength(&sorted);
|
||||||
let inversion_base_m = compute_inversion_base_m(&sorted);
|
let inversion_base_m = compute_inversion_base_m(&sorted);
|
||||||
|
|
||||||
// Ducts: the f01..f18 pipeline only stores the duct-summary scalars
|
// Ducts: the f01..f48 pipeline only stores the duct-summary scalars
|
||||||
// (`max_duct_thickness_m`, `duct_count`, `best_duct_freq_ghz`) per
|
// (`max_duct_thickness_m`, `duct_count`, `best_duct_freq_ghz`) per
|
||||||
// cell. Duct base requires the full Duct list, which isn't threaded
|
// cell. Duct base requires the full Duct list, which isn't threaded
|
||||||
// through `CellValues` yet — matches the current Elixir behavior on
|
// through `CellValues` yet — matches the current Elixir behavior on
|
||||||
|
|
|
||||||
|
|
@ -6,9 +6,9 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuerTest do
|
||||||
alias Microwaveprop.Propagation.GridTaskEnqueuer
|
alias Microwaveprop.Propagation.GridTaskEnqueuer
|
||||||
alias Microwaveprop.Repo
|
alias Microwaveprop.Repo
|
||||||
|
|
||||||
test "seeds one row per fh=1..18" do
|
test "seeds one row per forecast hour" do
|
||||||
run_time = ~U[2026-04-19 15:00:00Z]
|
run_time = ~U[2026-04-19 15:00:00Z]
|
||||||
assert {:ok, 18} = GridTaskEnqueuer.seed(run_time)
|
assert {:ok, 28} = GridTaskEnqueuer.seed(run_time)
|
||||||
|
|
||||||
rows =
|
rows =
|
||||||
Repo.all(
|
Repo.all(
|
||||||
|
|
@ -18,8 +18,9 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuerTest do
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
assert length(rows) == 18
|
expected_fhs = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48]
|
||||||
assert rows |> Enum.map(& &1.forecast_hour) |> Enum.sort() == Enum.to_list(1..18)
|
assert length(rows) == 28
|
||||||
|
assert rows |> Enum.map(& &1.forecast_hour) |> Enum.sort() == expected_fhs
|
||||||
assert Enum.all?(rows, &(&1.status == "queued"))
|
assert Enum.all?(rows, &(&1.status == "queued"))
|
||||||
|
|
||||||
# Raw schemaless selects return NaiveDateTime; compare against the
|
# Raw schemaless selects return NaiveDateTime; compare against the
|
||||||
|
|
@ -34,17 +35,17 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuerTest do
|
||||||
|
|
||||||
test "is idempotent: reseeding the same run_time inserts nothing" do
|
test "is idempotent: reseeding the same run_time inserts nothing" do
|
||||||
run_time = ~U[2026-04-19 16:00:00Z]
|
run_time = ~U[2026-04-19 16:00:00Z]
|
||||||
assert {:ok, 18} = GridTaskEnqueuer.seed(run_time)
|
assert {:ok, 28} = GridTaskEnqueuer.seed(run_time)
|
||||||
assert {:ok, 0} = GridTaskEnqueuer.seed(run_time)
|
assert {:ok, 0} = GridTaskEnqueuer.seed(run_time)
|
||||||
|
|
||||||
count = Repo.one(from(g in "grid_tasks", where: g.run_time == ^run_time, select: count()))
|
count = Repo.one(from(g in "grid_tasks", where: g.run_time == ^run_time, select: count()))
|
||||||
|
|
||||||
assert count == 18
|
assert count == 28
|
||||||
end
|
end
|
||||||
|
|
||||||
test "never emits a row for fh=0 — Elixir keeps that chain step" do
|
test "never emits a row for fh=0 — Elixir keeps that chain step" do
|
||||||
run_time = ~U[2026-04-19 17:00:00Z]
|
run_time = ~U[2026-04-19 17:00:00Z]
|
||||||
assert {:ok, 18} = GridTaskEnqueuer.seed(run_time)
|
assert {:ok, 28} = GridTaskEnqueuer.seed(run_time)
|
||||||
|
|
||||||
f0_count = Repo.one(from(g in "grid_tasks", where: g.run_time == ^run_time and g.forecast_hour == 0, select: count()))
|
f0_count = Repo.one(from(g in "grid_tasks", where: g.run_time == ^run_time and g.forecast_hour == 0, select: count()))
|
||||||
|
|
||||||
|
|
@ -54,7 +55,7 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuerTest do
|
||||||
describe "source-aware seeding" do
|
describe "source-aware seeding" do
|
||||||
test "seed/2 defaults to source='hrrr'" do
|
test "seed/2 defaults to source='hrrr'" do
|
||||||
run_time = ~U[2026-04-29 12:00:00Z]
|
run_time = ~U[2026-04-29 12:00:00Z]
|
||||||
assert {:ok, 18} = GridTaskEnqueuer.seed(run_time)
|
assert {:ok, 28} = GridTaskEnqueuer.seed(run_time)
|
||||||
|
|
||||||
sources =
|
sources =
|
||||||
Repo.all(from(g in "grid_tasks", where: g.run_time == ^run_time, select: g.source))
|
Repo.all(from(g in "grid_tasks", where: g.run_time == ^run_time, select: g.source))
|
||||||
|
|
@ -64,27 +65,27 @@ defmodule Microwaveprop.Propagation.GridTaskEnqueuerTest do
|
||||||
|
|
||||||
test "seed_with_analysis/2 with source='hrdps' tags every row" do
|
test "seed_with_analysis/2 with source='hrdps' tags every row" do
|
||||||
run_time = ~U[2026-04-29 12:00:00Z]
|
run_time = ~U[2026-04-29 12:00:00Z]
|
||||||
assert {:ok, 19} = GridTaskEnqueuer.seed_with_analysis(run_time, source: "hrdps")
|
assert {:ok, 29} = GridTaskEnqueuer.seed_with_analysis(run_time, source: "hrdps")
|
||||||
|
|
||||||
sources =
|
sources =
|
||||||
Repo.all(from(g in "grid_tasks", where: g.run_time == ^run_time, select: g.source))
|
Repo.all(from(g in "grid_tasks", where: g.run_time == ^run_time, select: g.source))
|
||||||
|
|
||||||
assert length(sources) == 19
|
assert length(sources) == 29
|
||||||
assert Enum.uniq(sources) == ["hrdps"]
|
assert Enum.uniq(sources) == ["hrdps"]
|
||||||
end
|
end
|
||||||
|
|
||||||
test "HRRR and HRDPS chains for the same run_time coexist (4-column unique key)" do
|
test "HRRR and HRDPS chains for the same run_time coexist (4-column unique key)" do
|
||||||
run_time = ~U[2026-04-29 18:00:00Z]
|
run_time = ~U[2026-04-29 18:00:00Z]
|
||||||
assert {:ok, 19} = GridTaskEnqueuer.seed_with_analysis(run_time, source: "hrrr")
|
assert {:ok, 29} = GridTaskEnqueuer.seed_with_analysis(run_time, source: "hrrr")
|
||||||
assert {:ok, 19} = GridTaskEnqueuer.seed_with_analysis(run_time, source: "hrdps")
|
assert {:ok, 29} = GridTaskEnqueuer.seed_with_analysis(run_time, source: "hrdps")
|
||||||
|
|
||||||
total = Repo.one(from(g in "grid_tasks", where: g.run_time == ^run_time, select: count()))
|
total = Repo.one(from(g in "grid_tasks", where: g.run_time == ^run_time, select: count()))
|
||||||
assert total == 38
|
assert total == 58
|
||||||
end
|
end
|
||||||
|
|
||||||
test "reseeding the same run_time + source is idempotent" do
|
test "reseeding the same run_time + source is idempotent" do
|
||||||
run_time = ~U[2026-04-29 06:00:00Z]
|
run_time = ~U[2026-04-29 06:00:00Z]
|
||||||
assert {:ok, 19} = GridTaskEnqueuer.seed_with_analysis(run_time, source: "hrdps")
|
assert {:ok, 29} = GridTaskEnqueuer.seed_with_analysis(run_time, source: "hrdps")
|
||||||
assert {:ok, 0} = GridTaskEnqueuer.seed_with_analysis(run_time, source: "hrdps")
|
assert {:ok, 0} = GridTaskEnqueuer.seed_with_analysis(run_time, source: "hrdps")
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,7 @@ defmodule Microwaveprop.Propagation.UntestedFunctionsTest do
|
||||||
assert_in_delta diff_s, 3600, 60
|
assert_in_delta diff_s, 3600, 60
|
||||||
|
|
||||||
diff_future_s = DateTime.diff(future, now)
|
diff_future_s = DateTime.diff(future, now)
|
||||||
assert_in_delta diff_future_s, 18 * 3600, 60
|
assert_in_delta diff_future_s, 48 * 3600, 60
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -854,10 +854,10 @@ defmodule Microwaveprop.PropagationTest do
|
||||||
assert Propagation.available_valid_times(10_000) == [valid_time]
|
assert Propagation.available_valid_times(10_000) == [valid_time]
|
||||||
end
|
end
|
||||||
|
|
||||||
test "caps the forward horizon to 18 hours so stale cycles don't clutter the timeline" do
|
test "caps the forward horizon to 48 hours so stale cycles don't clutter the timeline" do
|
||||||
now = DateTime.truncate(DateTime.utc_now(), :second)
|
now = DateTime.truncate(DateTime.utc_now(), :second)
|
||||||
in_range = DateTime.add(now, 3600, :second)
|
in_range = DateTime.add(now, 3600, :second)
|
||||||
past_hrrr_horizon = DateTime.add(now, 25 * 3600, :second)
|
past_hrrr_horizon = DateTime.add(now, 55 * 3600, :second)
|
||||||
|
|
||||||
Enum.each([{in_range, 70}, {past_hrrr_horizon, 50}], fn {t, score} ->
|
Enum.each([{in_range, 70}, {past_hrrr_horizon, 50}], fn {t, score} ->
|
||||||
Propagation.replace_scores(
|
Propagation.replace_scores(
|
||||||
|
|
|
||||||
|
|
@ -4,10 +4,11 @@ defmodule Microwaveprop.Workers.PropagationGridWorkerTest do
|
||||||
|
|
||||||
Elixir no longer runs the chain step; the hourly cron fires
|
Elixir no longer runs the chain step; the hourly cron fires
|
||||||
`perform(%Oban.Job{args: %{}})` with empty args and the worker
|
`perform(%Oban.Job{args: %{}})` with empty args and the worker
|
||||||
inserts 19 grid_tasks rows (1 analysis f00 + 18 forecast f01..f18)
|
inserts grid_tasks rows (1 analysis f00 + forecast f01..f18 hourly,
|
||||||
for the Rust `prop-grid-rs` worker to drain. Rust owns HRRR fetch,
|
f21..f48 3-hourly) for the Rust `prop-grid-rs` worker to drain.
|
||||||
wgrib2 decode, native duct, NEXRAD, commercial, scoring, and both
|
Rust owns HRRR fetch, wgrib2 decode, native duct, NEXRAD,
|
||||||
the ProfilesFile and band score-file writes from here on out.
|
commercial, scoring, and both the ProfilesFile and band score-file
|
||||||
|
writes from here on out.
|
||||||
"""
|
"""
|
||||||
use Microwaveprop.DataCase, async: false
|
use Microwaveprop.DataCase, async: false
|
||||||
use Oban.Testing, repo: Microwaveprop.Repo
|
use Oban.Testing, repo: Microwaveprop.Repo
|
||||||
|
|
@ -67,7 +68,7 @@ defmodule Microwaveprop.Workers.PropagationGridWorkerTest do
|
||||||
end
|
end
|
||||||
|
|
||||||
describe "perform/1 — chain seeding (empty args)" do
|
describe "perform/1 — chain seeding (empty args)" do
|
||||||
test "inserts 1 analysis + 18 forecast grid_tasks rows; no Oban fan-out" do
|
test "inserts 1 analysis + forecast grid_tasks rows; no Oban fan-out" do
|
||||||
Oban.Testing.with_testing_mode(:manual, fn ->
|
Oban.Testing.with_testing_mode(:manual, fn ->
|
||||||
import Ecto.Query
|
import Ecto.Query
|
||||||
|
|
||||||
|
|
@ -116,13 +117,44 @@ defmodule Microwaveprop.Workers.PropagationGridWorkerTest do
|
||||||
order_by: [t.kind, t.forecast_hour]
|
order_by: [t.kind, t.forecast_hour]
|
||||||
)
|
)
|
||||||
|
|
||||||
|
expected_fhs = [
|
||||||
|
1,
|
||||||
|
2,
|
||||||
|
3,
|
||||||
|
4,
|
||||||
|
5,
|
||||||
|
6,
|
||||||
|
7,
|
||||||
|
8,
|
||||||
|
9,
|
||||||
|
10,
|
||||||
|
11,
|
||||||
|
12,
|
||||||
|
13,
|
||||||
|
14,
|
||||||
|
15,
|
||||||
|
16,
|
||||||
|
17,
|
||||||
|
18,
|
||||||
|
21,
|
||||||
|
24,
|
||||||
|
27,
|
||||||
|
30,
|
||||||
|
33,
|
||||||
|
36,
|
||||||
|
39,
|
||||||
|
42,
|
||||||
|
45,
|
||||||
|
48
|
||||||
|
]
|
||||||
|
|
||||||
kinds = rows |> Enum.map(& &1.kind) |> Enum.frequencies()
|
kinds = rows |> Enum.map(& &1.kind) |> Enum.frequencies()
|
||||||
assert kinds == %{"analysis" => 1, "forecast" => 18}
|
assert kinds == %{"analysis" => 1, "forecast" => 28}
|
||||||
|
|
||||||
forecast_fhs =
|
forecast_fhs =
|
||||||
rows |> Enum.filter(&(&1.kind == "forecast")) |> Enum.map(& &1.fh) |> Enum.sort()
|
rows |> Enum.filter(&(&1.kind == "forecast")) |> Enum.map(& &1.fh) |> Enum.sort()
|
||||||
|
|
||||||
assert forecast_fhs == Enum.to_list(1..18)
|
assert forecast_fhs == expected_fhs
|
||||||
end)
|
end)
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
@ -159,7 +191,7 @@ defmodule Microwaveprop.Workers.PropagationGridWorkerTest do
|
||||||
|
|
||||||
assert_receive {:telemetry, [:microwaveprop, :propagation, :grid_worker, :seeded], %{queued_tasks: count}, _}
|
assert_receive {:telemetry, [:microwaveprop, :propagation, :grid_worker, :seeded], %{queued_tasks: count}, _}
|
||||||
|
|
||||||
assert count == 19
|
assert count == 29
|
||||||
after
|
after
|
||||||
:telemetry.detach(handler_id)
|
:telemetry.detach(handler_id)
|
||||||
end
|
end
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue