# Propagation Modeling Improvements Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Rework the propagation scorer to incorporate the physical mechanisms surfaced by the April 2026 meteorologist review: fine vertical resolution, boundary-layer turbulence, ray-traced duct geometry, frontal geometry, observed radar-based stability, and regional/climatological corrections. Retire features that don't survive backtesting against the 58k QSO corpus.

**Architecture:** Add new feature computations on top of the existing HRRR → scoring → propagation_scores pipeline without replacing it. Validate each feature in isolation against historical QSOs. Reweight (and drop) features in a final recalibration pass. Existing scoring must keep working throughout — no feature goes into production until its backtest justifies it.

**Tech Stack:** Elixir (existing), HRRR native hybrid-sigma GRIB2 (new ingestion), NEXRAD Level II via Iowa Environmental Mesonet (new pipeline), Ecto migrations (additive only), Nx for spatial gradients and ray-tracing math.

---

## Context

The meteorologist's review identified:

1. **Vertical resolution too coarse** — 25 hPa ≈ 250 m; ducts are 50-200 m thick, so the current pressure-level product smooths across the layer we care about.
2. **`min_refractivity_gradient` conflates two things** — the *strength* of the gradient and the *smoothness* of the interface. A strong gradient on a turbulent layer does not propagate.
3. **Pressure/wind correlations leak through from unmodeled frontal geometry** — findings 2 and 3 are proxies for "south of an advancing cold front, path parallel to it".
4. **PWAT is a co-variate of frontal position, not a cause** — probably drops out entirely when frontal geometry is modeled directly.
5. **Seasonal/humidity scoring needs regional stratification** — Gulf August is dry, Iowa August is the opposite because of corn evapotranspiration.
6. **NEXRAD clear-air texture is an underused observed BL-stability signal** — real-time, free, independent of HRRR's forecast error.
7. **Elevated ducts above the boundary layer exist** — some X-band contacts don't use the BL-top mechanism at all.
8. **Temperature anomaly vs climatology** matters more than absolute temperature for summer afternoon enhancement.

Phase 0 builds the infrastructure to falsify each of these claims against the QSOs we already have. Every downstream phase is a testable hypothesis.

---

## Phase Ordering and Dependencies

```
Phase 0: Backtest harness ───┬──► all subsequent phases depend on this
                             │
Phase 1: HRRR hybrid-sigma ──┼──┬── Phase 2: BL turbulence features ──┐
                             │  │                                     │
                             │  └── Phase 4: Ray-traced duct geometry ─┤
                             │                                         │
Phase 3: NEXRAD BL stability (parallel, independent) ─────────────────┤
                             │                                         │
Phase 5: Frontal geometry ───┴── (requires spatial gradients)         │
                                                                      │
Phase 6: Temperature anomaly ── (requires climatology backfill) ──────┤
                                                                      │
Phase 7: Regionalized seasons ── (pure config, can start anytime) ────┤
                                                                      │
Phase 8: 5-min METAR ──── (parallel, independent) ────────────────────┤
                                                                      │
Phase 9: Feature selection and recalibration ◄────────────────────────┘
```

- Phase 0 blocks everything.
- Phase 1 blocks Phases 2, 4, and (partially) 5.
- Phase 9 must come last — it's where the scorer actually changes.
- Phases 3, 7, 8 are independent and can run in parallel whenever there's bandwidth.

Each phase has a **gate**: if backtesting against the QSO corpus doesn't show lift, the feature is parked, not merged. The plan is sized so that a negative result on a phase doesn't block the rest of the work.

---

## Phase 0: Backtest Harness

**Why first:** We can't evaluate the meteorologist's claims without the ability to score a proposed feature against historical QSO data. Every subsequent phase's gate depends on this.

**Goal:** `Microwaveprop.Backtest` — takes a feature function `(lat, lon, valid_time) -> float | nil` and produces a report comparing feature distribution at QSO times vs. random baseline, lift by distance bin, lift by band.

**Key constraint:** We don't have non-contact labels. A QSO existing means "propagation was good enough to work at this band between these grids at this time." To create contrast, we use:
- **Random-time baseline** — same (lat, lon) but a random timestamp in the same month. If a feature has predictive power, its distribution at QSO times should differ from its distribution at random times.
- **Distance binning** — longer paths on the same band imply better propagation, so a good feature should correlate with `distance_km`.
- **Band stratification** — a feature that only works for 10 GHz tells a different story than one that works for 47 GHz.

### Task 0.1: Backtest module scaffolding

**Files:**
- Create: `lib/microwaveprop/backtest.ex`
- Create: `test/microwaveprop/backtest_test.exs`

**Step 1: Write failing test**

```elixir
test "evaluate/2 returns counts and distributions for a trivial feature" do
  create_qso(band: "10000", qso_timestamp: ~U[2026-01-15 12:00:00Z], pos1: %{...})

  feature = fn _lat, _lon, _time -> 1.0 end
  report = Backtest.evaluate(feature, sample_size: 10, baseline_size: 10)

  assert report.qso_count == 1
  assert report.baseline_count == 10
  assert %Backtest.Distribution{} = report.qso_distribution
end
```

**Step 2: Implement** `evaluate/2` pulling QSOs with `pos1 != nil`, iterating the feature over them, and producing the same stats for random (lat, lon, random-month-hour) triples.

**Step 3: Commit.** `feat(backtest): scaffold backtest harness module`

### Task 0.2: Distance-binned lift report

**Files:**
- Modify: `lib/microwaveprop/backtest.ex`
- Modify: `test/microwaveprop/backtest_test.exs`

Add `Backtest.lift_by_distance/2`:

- Bin QSOs by `distance_km` (0-100, 100-250, 250-500, 500-1000, 1000+).
- For each bin, compute mean and p50/p90 of the feature value.
- Return a map keyed by distance bin.

A good feature shows monotonically increasing mean with distance bin; a neutral feature shows flat. Write tests with a synthetic feature that has known correlation.

### Task 0.3: Band-stratified report

`Backtest.lift_by_band/2` — same idea, keyed on `band` (decimal). Writes a sortable table.

### Task 0.4: Random-time baseline generator

`Backtest.random_baseline/2` — for a set of (lat, lon) pairs, generate N random timestamps within ±30 days of each QSO's actual timestamp (matched sample). This controls for seasonality.

### Task 0.5: CLI runner

**Files:**
- Create: `lib/mix/tasks/backtest.ex`

`mix backtest --feature Microwaveprop.Backtest.Features.NaiveGradient` — loads a feature module, runs `evaluate/2`, prints a Markdown report to stdout.

### Task 0.6: Baseline feature modules

Create the feature functions for the existing algo factors, so we have known baselines to compare new features against:

- `Backtest.Features.NaiveGradient` — wraps the current `min_refractivity_gradient` scoring.
- `Backtest.Features.TdDepression`
- `Backtest.Features.TimeOfDay`
- `Backtest.Features.Pressure`

Run `mix backtest` on each, save reports to `priv/backtest_reports/baseline_*.md`, commit.

**Gate for Phase 0:** the baseline reports must plausibly match the published algo findings (e.g. refractivity should show *some* lift; time-of-day should show stronger lift at night). If they don't, the backtest harness is wrong and needs fixing before moving on.

**Estimated effort:** ~1 engineer-week.

---

## Phase 1: HRRR Native Hybrid-Sigma Level Ingestion

**Why:** 25 hPa pressure levels smooth across the ducting layer. Phases 2, 4, 5, and 6 need the native vertical resolution.

**Research:** HRRR native output is on a hybrid-sigma grid with 50 levels, densest near the surface (the first few sigma levels are <30 m apart). NOAA publishes this as the `wrfnatf` product alongside the standard `wrfsfcf` / `wrfprsf` files on NOMADS and AWS S3. File size is ~4× the pressure-level product.

**Goal:** Add a parallel ingestion path that pulls and stores hybrid-sigma profiles, without disturbing the existing `hrrr_profiles` table.

### Task 1.1: Spike — download one hybrid-sigma file and inspect

**Files:**
- Create: `scripts/hrrr_hybrid_spike.exs`

- Pull one `hrrr.tXXz.wrfnatf00.grib2` from AWS via the existing `HrrrClient`.
- Run `wgrib2 -V` to list variables and levels.
- Document: which variables are on which levels, what the level spacing looks like in the lowest 2 km, typical file size.
- Write findings to `docs/research/hrrr_native_levels.md`.

**Gate:** if hybrid-sigma files aren't reliably available on AWS S3 for historical dates (they rotate off faster than pressure-level files), we need a different data source. Document this first.

### Task 1.2: New schema `hrrr_native_profiles`

**Files:**
- Create: `priv/repo/migrations/XXXXXXX_add_hrrr_native_profiles.exs`
- Create: `lib/microwaveprop/weather/hrrr_native_profile.ex`

```elixir
create table(:hrrr_native_profiles, primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :valid_time, :utc_datetime
  add :lat, :float
  add :lon, :float
  add :level_count, :integer
  add :heights_m, {:array, :float}       # geopotential height per level
  add :temp_k, {:array, :float}
  add :dewpoint_k, {:array, :float}
  add :pressure_pa, {:array, :float}
  add :u_wind_ms, {:array, :float}
  add :v_wind_ms, {:array, :float}
  add :tke_m2s2, {:array, :float}        # if available in HRRR native
  timestamps()
end

create unique_index(:hrrr_native_profiles, [:lat, :lon, :valid_time])
```

Do NOT modify `hrrr_profiles`. The new table lives alongside it. We flip the scorer only after Phase 9.

### Task 1.3: Native-level GRIB2 extraction

**Files:**
- Modify: `lib/microwaveprop/weather/grib2/wgrib2.ex`
- Create: `lib/microwaveprop/weather/hrrr_native_client.ex`
- Create: `test/microwaveprop/weather/hrrr_native_client_test.exs`

`HrrrNativeClient.fetch_profile(lat, lon, valid_time)`:
- Uses the byte-range index trick on `wrfnatf00.grib2.idx` to pull only the variables and levels we need (keep the transfer small).
- Extracts T, Td, P, U, V, Z, TKE on all hybrid levels for the point.
- Returns a normalized struct with all levels sorted by height.

Test with stubbed Req responses and a recorded GRIB2 fixture.

### Task 1.4: Ingestion worker

**Files:**
- Create: `lib/microwaveprop/workers/hrrr_native_fetch_worker.ex`

On-demand worker (like the existing `HrrrFetchWorker`), enqueued per QSO:

```elixir
use Oban.Worker, queue: :hrrr, max_attempts: 3,
  unique: [period: :infinity, keys: [:lat, :lon, :valid_time]]
```

Fetches, upserts into `hrrr_native_profiles`, marks the QSO's new `hrrr_native_status` column. Add that column in a small migration.

### Task 1.5: Backfill top-500 most-contacted grid cells

Write a mix task `mix hrrr_native_backfill --limit 500` that enqueues the native fetch for the most-used cells so we have enough data to backtest. Run it. Monitor oban_web to see it finish.

### Task 1.6: First backtest — does level-0 refractivity improve?

Create `Backtest.Features.NativeSurfaceRefractivity` using only the lowest native level (vs. the interpolated lowest pressure level the current algo uses). Run backtest. Expected: marginal or no improvement, because the difference is a ~30 m vertical offset.

This is the sanity check that the pipeline works before investing in Phase 2.

**Gate:** pipeline produces native-level profiles for at least 80% of the attempted backfill cells and the sanity-check backtest doesn't regress vs. the baseline report.

**Estimated effort:** ~2 engineer-weeks.

---

## Phase 2: Boundary-Layer Turbulence Features

**Why:** The meteorologist's central claim is that gradient strength and interface smoothness are *separate* physical factors and the algo only measures one of them. Fix it.

**Goal:** Compute Richardson number, θₑ gradient, and wind shear at the inversion top from native-level profiles. Store as new columns. Backtest each independently.

### Task 2.1: Inversion-top detection

**Files:**
- Create: `lib/microwaveprop/propagation/inversion.ex`
- Create: `test/microwaveprop/propagation/inversion_test.exs`

```elixir
def find_inversion_top(profile) do
  # Walk the profile from the surface up.
  # Find the first height where dT/dz > 0 (temperature inversion)
  # or where dN/dz is at its minimum (classic ducting detection).
  # Return {:ok, height_m, level_idx} or :none.
end
```

TDD with three synthetic profiles:
- No inversion (monotonic T cooling with height) → `:none`
- Surface-based inversion (T increases from surface to 500 m) → top at ~500 m
- Elevated inversion at 2 km → top at ~2 km

### Task 2.2: Richardson number computation

**Files:**
- Modify: `lib/microwaveprop/propagation/inversion.ex`

```elixir
def bulk_richardson(profile, top_level_idx) do
  # Ri = (g / theta_ref) * (delta_theta / delta_u^2)
  # where delta_theta and delta_u are across the layer from
  # surface to top_level_idx.
end
```

Literature values:
- Ri < 0.25 → dynamically unstable, turbulent
- 0.25 < Ri < 1.0 → marginal
- Ri > 1.0 → stable/laminar (good for ducting)

TDD with synthetic cases: high-shear + weak-inversion → Ri low; calm + strong-inversion → Ri high.

### Task 2.3: θₑ computation and gradient

**Files:**
- Create: `lib/microwaveprop/weather/theta_e.ex`
- Create: `test/microwaveprop/weather/theta_e_test.exs`

Bolton (1980) formulation for equivalent potential temperature from T, Td, P. Property test: θₑ is conserved along a dry adiabat; use `stream_data`.

Then `Inversion.theta_e_jump(profile, top_level_idx)` returns ΔθₑK across the inversion.

### Task 2.4: Derived fields on `hrrr_native_profiles`

**Files:**
- Create: `priv/repo/migrations/XXXXXXX_add_derived_hrrr_native_fields.exs`
- Modify: `lib/microwaveprop/workers/hrrr_native_fetch_worker.ex`

Add columns (all nullable):
- `inversion_top_m :float`
- `bulk_richardson :float`
- `theta_e_jump_k :float`
- `shear_at_top_ms :float`
- `duct_thickness_m :float` (placeholder for Phase 4)

Compute these in the fetch worker after extraction. Backfill the existing rows with a one-shot mix task `mix hrrr_native_derive`.

### Task 2.5: Backtest the turbulence features

Create three feature modules:
- `Backtest.Features.BulkRichardson`
- `Backtest.Features.ThetaEJump`
- `Backtest.Features.ShearAtTop`

Run `mix backtest` on each. Write the reports to `priv/backtest_reports/phase2_*.md`.

**Gate:** at least one of the three features shows meaningful lift (defined as: p90 of the feature at QSO times > p50 at random baseline times, or monotonic lift across distance bins for at least two bands). If none do, either:
- The backtest sample is too small (backfill more cells), or
- The features are correctly computed but the physical claim doesn't hold for our corpus, or
- Our corpus is dominated by contacts that use a different mechanism (elevated ducts — Phase 4 may rescue this).

Document the finding either way.

**Estimated effort:** ~1.5 engineer-weeks.

---

## Phase 3: NEXRAD Clear-Air Stability (parallel, independent)

**Why:** Real-time, observed, free. If it works, it's the single highest-ROI new data source and it doesn't depend on HRRR being right.

**Goal:** Pull NEXRAD Level II composite reflectivity, compute local spatial variance, expose as a feature.

### Task 3.1: Spike — download and visualize one frame

**Files:**
- Create: `scripts/nexrad_spike.exs`

- Use the Iowa Environmental Mesonet composite reflectivity GeoTIFF endpoint.
- Download one frame for a known dead-band time (e.g. 20Z on a summer afternoon) and one for a known good time (e.g. 10Z at dawn on the same day).
- Decode with `Image` or shell out to `gdal_translate`.
- Visually inspect: does the dead-band frame look "blotchy" as the meteorologist described?

**Gate:** if we can't visually see the difference by eye, the feature probably doesn't have signal and Phase 3 stops here.

### Task 3.2: NEXRAD ingestion worker

**Files:**
- Create: `lib/microwaveprop/weather/nexrad_client.ex`
- Create: `lib/microwaveprop/workers/nexrad_fetch_worker.ex`
- Create: `priv/repo/migrations/XXXXXXX_create_nexrad_stability.exs`

Table:
```elixir
create table(:nexrad_stability, primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :valid_time, :utc_datetime
  add :lat, :float
  add :lon, :float
  add :local_variance, :float
  add :mean_reflectivity, :float
  timestamps()
end
```

Pulls composite reflectivity at 5-6 minute cadence for a CONUS grid matching the existing propagation grid spacing. Store only the *derived* variance — we don't need the raw tiles.

### Task 3.3: Variance computation

**Files:**
- Create: `lib/microwaveprop/weather/radar_texture.ex`
- Create: `test/microwaveprop/weather/radar_texture_test.exs`

For each grid point, compute the variance of reflectivity in a 40×40 km window around it. Exclude pixels with precipitation returns (> ~20 dBZ) — we only care about clear-air texture, not actual weather.

Use Nx for the windowed variance to keep it fast.

### Task 3.4: Backfill

Mix task `mix nexrad_backfill --days 30` — pulls frames for the last 30 days at hourly cadence, computes variance, writes to `nexrad_stability`.

### Task 3.5: Backtest

`Backtest.Features.NexradTexture` — mean local_variance in a ±30 min window at the QSO midpoint. Run `mix backtest`.

**Gate:** NEXRAD texture must show at least as much lift as the best Phase 2 feature to justify carrying a new data pipeline in production. If it's strictly better, it becomes a first-class feature in Phase 9. If it's comparable, ship it alongside as a sanity check.

**Estimated effort:** ~2 engineer-weeks.

---

## Phase 4: Ray-Traced Duct Geometry

**Why:** Turns "is it ducty" into "can *this band* use the duct". Critical for band-specific scoring, and necessary to correctly score elevated ducts.

**Goal:** Replace `min_refractivity_gradient` (scalar) with `{duct_top_m, duct_base_m, duct_thickness_m, trapped_freq_ghz}` — a tuple describing the duct geometry from ray-tracing the native-level profile.

### Task 4.1: Refractivity profile from native levels

**Files:**
- Create: `lib/microwaveprop/propagation/refractivity_profile.ex`
- Create: `test/microwaveprop/propagation/refractivity_profile_test.exs`

```elixir
def from_native_profile(profile) do
  # For each level, compute N = 77.6*P/T + 3.73e5*e/T^2  (ITU-R P.453)
  # Return a list of {height_m, N} tuples sorted by height.
end
```

TDD with the ITU-R sample profiles.

### Task 4.2: Duct detection via M-profile

The *modified refractivity* M = N + 157·h (where h is in km) is the conventional form. A duct exists wherever dM/dh < 0.

**Files:**
- Modify: `lib/microwaveprop/propagation/refractivity_profile.ex`

```elixir
def detect_ducts(refractivity_profile) do
  # Walk the profile computing M at each level.
  # Every contiguous region where dM/dh < 0 is a duct.
  # Return a list of %Duct{base_m, top_m, thickness_m, m_deficit}.
end
```

TDD: synthetic profile with a known 100 m surface-based duct and a known 200 m elevated duct at 1.5 km → detector finds both.

### Task 4.3: Trapped frequency calculation

The minimum frequency a duct of thickness `d` can trap is approximately:

```
f_min_ghz ≈ 3.6 / (d * sqrt(ΔM) / 100) ^ 1.5    (rough engineering approximation)
```

Find the right formula in Bean & Dutton or ITU-R P.834 and encode it. For each duct, compute `f_min_ghz`.

Then a band can use the duct iff `band_ghz >= f_min_ghz`. This gives us per-band scoring *derived from physical geometry*, not hand-tuned tables.

### Task 4.4: Store duct geometry

**Files:**
- Create: `priv/repo/migrations/XXXXXXX_add_duct_geometry.exs`

Add columns to `hrrr_native_profiles`:
- `ducts :map` (stores the list of %Duct{} structs as JSONB)
- `best_duct_band_ghz :float` (lowest `f_min` across all detected ducts — cached for fast scoring)

Backfill via `mix hrrr_native_derive` (reuse task from Phase 2).

### Task 4.5: Backtest duct features per band

Create:
- `Backtest.Features.DuctUsable10GHz` — bool: is there a duct with f_min ≤ 10 GHz
- `Backtest.Features.DuctUsable24GHz`
- `Backtest.Features.DuctUsable47GHz`
- `Backtest.Features.DuctThickness` — continuous: max thickness among detected ducts

Run band-stratified backtests. For each band, check:
- At QSO times on 10 GHz, is `DuctUsable10GHz` true more often than at random times? (Expected: yes.)
- Is the lift larger for 47 GHz than 10 GHz? (Expected: yes, because 47 GHz is harder, so ducting matters more.)

**Gate:** at least one of the duct features must outperform the existing `NaiveGradient` baseline on at least one band. If not, the ray-tracing added no value and we keep the scalar gradient.

**Estimated effort:** ~2 engineer-weeks, most of it in getting the trapped-frequency formula right and validating against known cases from the literature.

---

## Phase 5: Frontal Geometry (biggest conceptual gain)

**Why:** The meteorologist's claim is that pressure, calm winds, and PWAT are all proxies for "proximity to and orientation relative to a cold front". Modeling the front directly should replace three weak features with one strong geometric one.

**Goal:** For every (grid cell, valid_time), compute:
- `distance_to_front_km` — distance to nearest detected cold front
- `front_bearing_deg` — bearing of the front at the nearest point
- `path_front_angle_deg` — angle between the QSO path and the front (0 = parallel, 90 = crosses)

### Task 5.1: Frontal detection spike

**Files:**
- Create: `scripts/front_detection_spike.exs`

Research first. Candidate approaches:
1. **Thermal front parameter** (TFP) — `-∇|∇θ| · ∇θ / |∇θ|`, standard synoptic meteorology tool. Highest negative values mark cold fronts.
2. **Theta-e gradient** magnitude directly.
3. **Maximum of the horizontal wind shift** across a grid.

Implement TFP on one HRRR frame with a known strong cold front (pick a date from the NWS archive). Plot the result as a raster on top of the HRRR temperature field. Visual verification: do the detected fronts line up with the NWS surface analysis for that day?

**Gate:** if the detection doesn't visually match a hand-drawn NWS front, the automated method isn't reliable enough for feature engineering and Phase 5 becomes a research project rather than an implementation task. Document the finding, park the phase.

### Task 5.2: Frontal mask pipeline

**Files:**
- Create: `lib/microwaveprop/weather/frontal_analysis.ex`
- Create: `test/microwaveprop/weather/frontal_analysis_test.exs`

```elixir
def detect_fronts(hrrr_grid) do
  # Input: a 2D grid of HRRR surface fields (T, Td, U, V)
  # Output: a 2D boolean mask of front pixels, plus a bearing per pixel
end
```

Use Nx for the gradient math. Test with a synthetic grid that has a known linear front.

### Task 5.3: Per-cell frontal features

Add to `propagation_scores` or a sidecar table `frontal_features`:
- `distance_to_front_km`
- `front_bearing_deg`

Compute these hourly in the propagation pipeline. Cache alongside scores.

### Task 5.4: Per-path angle feature

When scoring a QSO (or a potential path) the path_front_angle is a function of both endpoints. This doesn't cache well, so expose it as a helper:

```elixir
def path_front_angle(path_lat1, path_lon1, path_lat2, path_lon2, valid_time)
```

It looks up the nearest front at the path midpoint and computes the angle.

### Task 5.5: Backtest frontal features

- `Backtest.Features.DistanceToFront` — distance to nearest front at the contact's grid 1 position.
- `Backtest.Features.ParallelToFront` — continuous: `cos²(path_front_angle)` so perpendicular paths score 0 and parallel paths score 1.

Run backtest stratified by distance. Expected: long-distance contacts cluster at paths with low `path_front_angle` (parallel) and 200-500 km `distance_to_front_km` (not on the front, but near it).

**Gate:** `ParallelToFront` must outperform the existing `Pressure` baseline, confirming the meteorologist's claim that pressure is a weaker proxy for the same thing. If so, pressure is dropped in Phase 9.

**Estimated effort:** ~3 engineer-weeks including the research spike.

---

## Phase 6: Temperature Anomaly vs. Climatology

**Why:** Finding 4 (diurnal) is real but has an important exception: extremely hot summer days produce strong enhancement even in the afternoon. Absolute temperature isn't the right feature; deviation from normal is.

**Goal:** For every (grid cell, month, hour), compute a climatological baseline temperature, and expose the anomaly as a feature.

### Task 6.1: Build climatology from existing `hrrr_profiles`

**Files:**
- Create: `lib/mix/tasks/hrrr_climatology.ex`
- Create: `priv/repo/migrations/XXXXXXX_create_hrrr_climatology.exs`

Table:
```elixir
create table(:hrrr_climatology, primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :lat, :float
  add :lon, :float
  add :month, :integer
  add :hour, :integer
  add :mean_surface_temp_k, :float
  add :stddev_surface_temp_k, :float
  add :sample_count, :integer
end
create unique_index(:hrrr_climatology, [:lat, :lon, :month, :hour])
```

We already have 4.5k `hrrr_profiles`. That's thin for climatology but enough to compute a rough baseline per (lat, lon, month, hour) for the grid cells with the most data. For cells with insufficient samples, fall back to NCEP reanalysis climatology (NARR has 30+ years of data).

### Task 6.2: Temperature anomaly feature

```elixir
def anomaly(lat, lon, valid_time, current_temp_k) do
  # Look up climatology.mean_surface_temp_k for (lat, lon, month, hour)
  # Return current_temp_k - climatology_mean
end
```

### Task 6.3: Backtest

`Backtest.Features.TemperatureAnomaly`. Expected: positive anomalies (hotter than normal) should show lift, especially during afternoon hours in summer — which is exactly when the existing `TimeOfDay` feature has *negative* lift. The two features should be anti-correlated in this regime, meaning anomaly "rescues" hot-day afternoon scoring.

**Gate:** anomaly shows positive lift during summer afternoons where the current `TimeOfDay` suppresses the score.

**Estimated effort:** ~1 engineer-week.

---

## Phase 7: Regionalized Seasonal Scoring

**Why:** The current `@band_configs` seasonal scoring is uniform across CONUS. Gulf August is dry and propagates well; Iowa August is miserable humidity and doesn't. Same month, opposite score.

**Goal:** Partition CONUS into 6-8 climatological regions and maintain per-region seasonal scoring.

### Task 7.1: Regional partition

**Files:**
- Create: `lib/microwaveprop/propagation/region.ex`

Define regions by bounding box (or Köppen climate zone, but bounding boxes are simpler):

```elixir
@regions [
  %{id: :gulf_coast, lat: {25, 32}, lon: {-100, -80}},
  %{id: :southern_plains, lat: {32, 38}, lon: {-105, -93}},
  %{id: :corn_belt, lat: {38, 45}, lon: {-100, -82}},
  %{id: :desert_southwest, lat: {30, 38}, lon: {-120, -105}},
  # ... 6-8 total
]
```

`Region.for_point(lat, lon)` returns the region id.

### Task 7.2: Per-region seasonal tables

**Files:**
- Modify: `lib/microwaveprop/propagation/band_config.ex`

Change `@band_configs` to index by `{band_mhz, region_id}` instead of just `band_mhz`. Start with the existing table replicated across all regions, then tune per region based on backtest.

### Task 7.3: Backtest per-region lift

Run the existing `Backtest.Features.Season` per region, compare against the uniform baseline. Tune the regional seasonal curves until the regional scores fit the regional QSO patterns.

**Estimated effort:** ~1 engineer-week. Mostly data-driven tuning rather than engineering.

---

## Phase 8: 5-Minute METAR Ingestion

**Why:** The current surface observation feed is hourly. Propagation windows can open and close on 20-30 minute timescales. Finer-grained surface data (especially temperature and dewpoint) lets the scorer catch transients.

**Goal:** Ingest NOAA NCEI 5-minute METAR, store alongside the existing `surface_observations` table.

### Task 8.1: NCEI 5-min METAR client

**Files:**
- Create: `lib/microwaveprop/weather/ncei_metar_client.ex`

Use the endpoint from the meteorologist's email (NCEI C00418). Document the format — it's different from the Iowa Mesonet ASOS feed the current pipeline uses.

### Task 8.2: Schema extension

Option A: new table `metar_5min_observations` identical in schema to `surface_observations` but with 5-minute granularity.
Option B: add a `source` column to `surface_observations` and mix them.

A is safer (no risk to existing queries), so do that. Migration adds the new table.

### Task 8.3: Ingestion worker

`Microwaveprop.Workers.NceiMetar5MinFetchWorker`, hourly cron pulling the most recent hour's data.

### Task 8.4: Use in scoring

Add a `recent_surface_obs/2` helper that prefers the 5-min source when available, falls back to `surface_observations` otherwise.

**Estimated effort:** ~1 engineer-week.

---

## Phase 9: Feature Selection and Recalibration

**Why:** Everything above is additive. The *real* improvement is cutting the features that don't survive backtesting and reweighting the ones that do.

**Goal:** Produce a new `BandConfig`-equivalent weight table that uses only the features with demonstrated lift, and validate end-to-end against the QSO corpus.

### Task 9.1: Consolidated backtest report

Run `mix backtest --all` that evaluates every registered feature and writes a consolidated table: feature name, lift by band, lift by distance, sample count, gate pass/fail.

### Task 9.2: Drop dead features

For each existing factor in the scorer (humidity, time_of_day, td_depression, refractivity, sky, season, wind, rain, pressure):
- If the consolidated report shows lift → keep, possibly with new formulation.
- If it shows no lift and there's a new feature measuring the same physics → drop, use the new one.
- If it shows no lift and there's no replacement → drop entirely.

Expected drops (if the meteorologist is right):
- `pressure` — replaced by `ParallelToFront`
- `wind` — same reason
- PWAT if it was ever added — replaced by frontal features

Expected keeps:
- `humidity` — core physics
- `time_of_day` — now augmented with `TemperatureAnomaly`
- `td_depression` — core physics
- `refractivity` — replaced with `DuctUsable` per band
- `sky` — keep but re-validate
- `season` — keep but regionalized (Phase 7)
- `rain` — keep, ITU-R absorption model

Expected adds:
- `bulk_richardson` or `theta_e_jump` (from Phase 2)
- `nexrad_texture` (from Phase 3)
- `duct_usable_band` (from Phase 4)
- `parallel_to_front` + `distance_to_front` (from Phase 5)
- `temperature_anomaly` (from Phase 6)

### Task 9.3: Recalibrate weights via gradient descent

**Files:**
- Create: `lib/mix/tasks/recalibrate_scorer.ex`

With the new feature set, fit weights by minimizing some loss against the QSO corpus. Simplest loss: negative log-likelihood of QSO distances under a logit link, treating QSOs as positives and random-baseline samples as negatives. Can use Nx.

This is a tiny model (< 15 features), so a few thousand iterations on CPU is fine. Cross-validate by month.

### Task 9.4: Side-by-side scoring run

Run the new scorer and the old scorer on the same HRRR frame. Diff the output grids. Look for:
- Any gross regressions (areas where the old scorer said "good" and the new scorer says "dead" with no physical justification)
- Gains in known-good corridors (e.g. Gulf coast → Brazos → DFW on a summer dawn)

### Task 9.5: Shadow deployment

Run the new scorer in parallel with the old one for 2 weeks in production. Both publish scores to PubSub but only the old one drives the map. Collect daily diff statistics. Promote if the new one tracks observed QSOs better.

### Task 9.6: Cutover

Flip the scorer. Keep the old implementation compilable for one release as a fallback.

**Estimated effort:** ~2 engineer-weeks.

---

## Total Effort and Sequencing

| Phase | Description | Weeks | Dependencies |
|---|---|---|---|
| 0 | Backtest harness | 1 | — |
| 1 | HRRR hybrid-sigma ingestion | 2 | 0 |
| 2 | BL turbulence features | 1.5 | 1 |
| 3 | NEXRAD clear-air stability | 2 | 0 |
| 4 | Ray-traced duct geometry | 2 | 1 |
| 5 | Frontal geometry | 3 | 0 |
| 6 | Temperature anomaly | 1 | 1 |
| 7 | Regionalized seasonal scoring | 1 | 0 |
| 8 | 5-minute METAR | 1 | 0 |
| 9 | Feature selection + recalibration | 2 | all |

Total: ~17 engineer-weeks if done serially. With some parallelism across independent phases: ~12 calendar weeks.

**Suggested execution order (maximizing parallelism and learning rate):**

1. **Weeks 1:** Phase 0 (blocking).
2. **Weeks 2-3:** Phase 1 (blocking for most downstream work) + Phase 3 spike in parallel (independent, highest-ROI standalone).
3. **Weeks 4-5:** Phase 3 full + Phase 2 in parallel.
4. **Weeks 6-7:** Phase 4 + Phase 6 in parallel.
5. **Weeks 8-10:** Phase 5 (biggest lift; if the spike doesn't match NWS fronts, park it and move to Phase 7/8 instead).
6. **Weeks 11-12:** Phase 7 + Phase 8 + final Phase 9 recalibration and shadow deployment.

**Kill criteria — when to stop mid-plan:**

- If Phase 0's baseline reports don't match published algo findings → stop, fix the harness.
- If Phase 1's hybrid-sigma data isn't available historically → switch to ERA5 model levels as a plan B for backtesting; stay on pressure levels for live scoring.
- If Phase 2 shows zero lift for all three turbulence features → the "smoothness" mechanism may not be the dominant one in our corpus; park Phase 2 and let Phase 4 (ducts) do the work.
- If Phase 3's NEXRAD spike shows no visual difference between dead-band and good times → park Phase 3.
- If Phase 5's frontal detection doesn't match hand-drawn NWS fronts → park Phase 5 as a research project.

Each phase's backtest report becomes the justification for shipping or parking its work. Nothing gets merged to production on vibes.

---

## Out of Scope (Explicitly)

- **Rewriting the existing scorer in ML** — separate track (there's already an Nx skeleton in `lib/microwaveprop/propagation/model.ex`). This plan is about better *features*; ML can consume them later.
- **Changing the grid resolution** — 0.125° stays.
- **Real-time radar-based scoring without a model fallback** — the existing HRRR-based scorer continues to drive the live map; NEXRAD becomes an additional feature, not a replacement data source.
- **Adding new bands** — 902 MHz and up, same set.
- **Touching terrain analysis** — ITU-R P.526-16 is separately accurate and not part of this plan.
