prop/prediction.md

# Propagation Prediction System Design

## The Opportunity

No tool exists for microwave operators to answer: **"When should I get on the air in the next 24 hours?"**

Current propagation maps (including ours) show what's happening *right now*. But microwave propagation has a strong, predictable diurnal cycle — ducting peaks near sunrise and collapses in the afternoon. An operator planning a Saturday morning contest attempt needs to know whether 06:00 or 07:30 is the better window, and whether conditions are even worth waking up for.

We have all the pieces to build this. The scoring algorithm is time-agnostic — feed it conditions for any future hour and it produces a valid score. HRRR provides 18-48 hour forecasts with all the variables we need. And 30% of the composite score (time-of-day + season) is purely deterministic — we can compute it for any moment in the future with perfect accuracy.

## What We Know About Prediction Accuracy

### Deterministic Factors (30% of composite weight)

**Time of Day (20% weight):** The biggest daily swing factor. For 10 GHz:
- Sunrise window (±1.5h): score 100
- Pre-dawn: 82
- Post-sunrise: 78
- Evening: 72
- Night: 55
- Marginal (3-6h after sunrise): 38
- Afternoon: 18

That's an 82-point swing in a single factor worth 20% of the total — driving a ~16-point composite score change from time alone. This is 100% predictable for any future hour.

**Season (10% weight):** Fixed per month. June scores 90 for 10 GHz; March scores 22. Already known for the entire year.

### Weather-Dependent Factors (70% of composite weight)

These require atmospheric observations or forecasts:
- **Humidity** (20%): Changes slowly — persists well over 6-12 hours
- **Td Depression** (12%): Tied to humidity — similar persistence
- **Refractivity** (10%): Most volatile — ducting layers form and dissolve quickly
- **Sky Cover** (10%): Moderate persistence; NWP forecasts decent to 12h
- **Rain** (8%): Highly variable; HRRR good to ~6h, fair to ~12h
- **Wind** (6%): Good persistence to 6h; HRRR skillful to 12h
- **Pressure** (4%): Excellent persistence — changes slowly over 12-24h

### HRRR Forecast Skill by Lead Time

| Lead Time | Temperature/Dewpoint | Wind | Clouds/Precip | Refractivity |
|-----------|---------------------|------|---------------|--------------|
| f01-f06 | Excellent | Very good | Good | Fair |
| f06-f12 | Good | Good | Fair | Poor |
| f12-f18 | Fair | Fair | Marginal | Poor |
| f18-f48* | Marginal | Marginal | Poor | Very poor |

*f18-f48 only available from 00Z and 12Z model runs.

Refractivity is the hardest factor to forecast. HRRR's 3km grid doesn't fully resolve boundary layer inversions, and ducting events are notoriously chaotic. However, the algorithm is designed to work with imperfect data — the composite score remains informative even with uncertain individual factors.

## Recommended Architecture: Three Phases

### Phase 1 — Instant Diurnal Forecast (No new data fetching)

**What:** When a user clicks a point, immediately compute a 24-hour score timeline using the current weather observation + deterministic time-of-day/season projections.

**How it works:**
1. User clicks point → we already have the current conditions from `point_detail`
2. Hold weather factors constant (humidity, Td, refractivity, sky, wind, rain, pressure)
3. For each of the next 24 hours, recompute the composite score with:
   - Weather factors: persisted from current observation
   - Time of Day: computed for that future hour (deterministic)
   - Season: same month (deterministic)
4. Display as a 24-bar sparkline in the detail panel

**Why this works:** The diurnal cycle IS the dominant signal. An operator seeing "score rises from 60 to 78 at sunrise" gets 80% of the value without any forecast data. The persistence assumption is reasonable for 6-12 hours under stable weather patterns (which are exactly the conditions that produce good propagation).

**What operators see:**
```
24-HOUR OUTLOOK
▁▂▃▅▇█▇▅▃▂▁▁▂▃▅▇█▇▅▃▂▁▁
Now  6h   12h   18h   24h

Best window: 06:15-08:30 CDT
Peak score: 78/100 (GOOD) — +18 from current
Key: Sunrise inversion peak
```

**Implementation cost:** Low. Pure computation — the scorer already handles arbitrary conditions maps. No new data fetching, no schema changes, no new workers. Add a function that loops `composite_score/2` over 24 hours with mutated `utc_hour`/`utc_minute`, render in JS.

**Accuracy:** Good for identifying WHEN the best window is. Less accurate for absolute score at 12+ hours if weather changes.

### Phase 2 — HRRR Point Forecast Enhancement

**What:** When a user clicks a point, also fetch HRRR forecast profiles (f01-f18) for that single lat/lon. Replace persisted weather factors with actual HRRR forecast values.

**How it works:**
1. Modify `hrrr_client.ex` to accept a `forecast_hour` parameter (currently hardcoded to f00)
2. On click, `start_async` fetches f01-f18 HRRR profiles for the clicked lat/lon
3. Each forecast hour provides: temperature, dewpoint, wind, pressure, clouds, precip, profile levels
4. Run `composite_score/2` with actual forecast conditions per hour
5. Replace Phase 1's persistence-based timeline with HRRR-enhanced scores

**What changes for the user:** The timeline becomes more accurate. Instead of "assume current humidity persists," each hour shows what HRRR actually predicts. A cold front moving through at 14Z would show rain scores dropping and pressure rising — Phase 1 wouldn't capture this.

**Implementation cost:** Medium. Requires:
- `hrrr_client.ex`: parameterize forecast_hour in URL builder (~20 lines)
- New function to fetch multi-hour forecasts for a single point
- LiveView async handler for forecast data
- JS timeline update logic
- ~2-3 second latency for 18 forecast-hour fetches (byte-range requests in parallel)

**Accuracy:** Significantly better than Phase 1 for weather-dependent factors, especially f01-f12. Still limited by HRRR's ability to resolve fine-scale refractivity.

### Phase 3 — Full Grid Forecast Map

**What:** Pre-compute propagation scores for the entire CONUS grid at key future time steps. The map overlay shows predicted conditions at selectable future hours.

**How it works:**
1. New `ForecastGridWorker` runs after the hourly PropagationGridWorker
2. Fetches HRRR f03, f06, f12, f18 (4 key time steps, not all 18)
3. Scores all ~20,500 grid points × 8 bands × 4 forecast hours
4. Stores in `propagation_scores` with future valid_times
5. Map UI adds a time slider/stepper to browse forecast hours
6. Animated playback showing propagation evolution

**What changes for the user:** The entire map overlay shifts to show predicted conditions at future hours. Operators can see "at sunrise tomorrow, where will the best propagation be?" and plan which direction to point their antenna.

**Implementation cost:** High. Requires:
- New Oban worker for forecast grid scoring
- Pruning policy update (keep full forecast cycles, not just 2 valid_times)
- Schema consideration: add `run_time`/`forecast_hour` columns to distinguish forecasts from analysis
- Time slider UI component in the map control panel
- Score overlay update mechanism (swap displayed valid_time)
- 4× more HRRR data fetching per hour
- ~660K additional records per forecast cycle

**Accuracy:** Same as Phase 2 per-point, but across the full grid. The map-level view is uniquely valuable — no other tool shows predicted microwave propagation conditions geospatially.

## "Best Time to Operate" — The Killer Feature

This is the output of Phase 1 that delivers immediate value. For a clicked point:

**Header:** "Best window: Tomorrow 06:15-08:30 CDT"
- Computes the highest-score period from the 24-hour timeline
- Shows duration of the window (how long scores stay above threshold)
- Shows expected peak score and tier

**Context:** "Expected score: 78/100 (GOOD) — 18 points above current"
- Delta from current score
- Primary driver (usually sunrise peak or weather change in Phase 2)

**Band comparison table:**
| Band | Best Window | Peak Score | Current |
|------|-----------|------------|---------|
| 10 GHz | 06:15-08:30 | 78 (GOOD) | 60 |
| 24 GHz | 06:15-07:45 | 71 (GOOD) | 58 |
| 47 GHz | 06:30-07:30 | 64 (MARGINAL) | 44 |

Different bands peak at different times and for different durations — showing this across bands helps operators choose which equipment to set up.

**Confidence indicator:**
- Phase 1: "Based on current conditions + sunrise timing" (honest about persistence assumption)
- Phase 2: "Based on HRRR forecast — high confidence to 6h, moderate to 12h"

## Data Flow Summary

```
Phase 1 (instant):
  Click → current conditions → loop 24h with deterministic time changes → timeline

Phase 2 (enhanced):
  Click → fetch HRRR f01-f18 for point → score each hour → timeline
  (replaces Phase 1 timeline with actual forecast data)

Phase 3 (grid):
  Hourly worker → fetch HRRR f03/f06/f12/f18 for full grid → score all → store
  Map UI → time slider → load scores for selected valid_time → render overlay
```

## What Makes This Unique

1. **Microwave-specific:** No general propagation tool models ducting, Fresnel zones, and humidity effects at 10-241 GHz
2. **Diurnal-aware:** The sunrise peak is THE event for microwave operators — we quantify exactly when and how strong
3. **Band-comparative:** Different bands have inverted seasonal and humidity responses — we show which band to use when
4. **Terrain-integrated:** Combined with the viewshed, operators see not just "when" but "where" they can reach
5. **Actionable:** "Wake up at 06:15, point your dish northeast, expect GOOD conditions for 2 hours" — that's what operators need

## Technical Notes

### Scorer Reuse
The `Scorer.composite_score/2` function is completely stateless — it takes a conditions map and a band config, returns a score. For prediction, we just construct conditions maps for future hours and call the same function. No algorithm changes needed.

### HRRR URL Pattern
Current: `hrrr.t{HH}z.wrfsfcf00.grib2`
Forecast: `hrrr.t{HH}z.wrfsfcf{FF}.grib2` where FF = 01-48

The `.idx` index files also exist for forecast hours, so the byte-range fetch strategy works identically.

### Storage Considerations
Phase 3 multiplies storage by ~5× (current 1 valid_time + 4 forecast hours). The covering index on `(band_mhz, valid_time, lat, lon)` already supports efficient queries for any valid_time. Pruning policy needs to shift from "keep 2 most recent valid_times" to "keep the latest complete forecast cycle."

### Pressure Trend Factor
The pressure factor (4% weight) currently compares current vs previous pressure. For predictions:
- Phase 1: Use current absolute pressure (trend = nil → absolute scoring)
- Phase 2: Can compute trend from consecutive HRRR forecast hours (f05→f06 delta)
- Impact is small (4% weight) so this isn't critical.

## Implementation Priority

**Phase 1 first.** It delivers 80% of the value with 20% of the effort. The diurnal cycle is the dominant signal, the computation is instant, and the "best time to operate" output is immediately useful. Phase 2 and 3 add accuracy but the core user question — "when should I get on?" — is already answered by Phase 1.