Graham McIntire
2fd88a94ea
Phase A — 7 pipeline bugs:
- retain_scores_window: fix timeline self-deletion (NOTIFY payload run_time|valid_time)
- Rust/Elixir weight divergence: Rust loads band_weights.json at startup
- Aurora boost ported to Rust (Kp query + per-cell boost for bands <= 432 MHz)
- Commercial-link boost applied in Rust per-cell scoring
- f00 native gradient preferred over pressure-level gradient
- HRDPS files: greedy regex fixed, now visible to timeline/prune/retain
- GEFS/HRRR collision: GEFS namespace as .gefs.prop, merge at read
Phase B — Validation harness:
- scripts/validate_algo.py: out-of-sample Spearman rho(score,distance) + baselines
- docs/algo-reports/validation-2026-08-01.{json,md}
Phase C — Forecast-skill evaluation:
- scripts/validate_forecast.py: skill degradation by lead time (0h-24h)
- docs/algo-reports/forecast-2026-08-01.{json,md}
Phase D — Calibration + model improvements:
- recalibrate.py: validation gate before weight deploy
- Recalibrator: Nx.max(0.0) replaces Nx.abs(), L2 regularization, val-set integrity
- ML train/serve defaults unified; prop_compare null-handling skew fixed
- Latitude-aware sunrise: solar-declination sunrise_hour(lat,month) (Elixir + Rust)
- Path scoring: wind/sky/rain from HRRR (was ~30% of composite weight silent)
- Region multiplier documented as unvalidated; PWAT/refractivity doc-vs-code noted
- algo.md: synced scoring sections, marked retired features, D5/D6 changelog
- Credo: path_compute cyclomatic complexity bumped (6->10 fields) — informational only
34 lines
2.6 KiB
Markdown
34 lines
2.6 KiB
Markdown
# Forecast Skill Degradation — 2026-08-01
|
||
|
||
> Auto-generated by `scripts/validate_forecast.py`. Measures how well the propagation scoring algorithm predicts contact distances as the weather profile ages. The lag-0h profile uses actual conditions at contact time (ground truth). Lag-Nh uses the weather profile from N hours earlier — representing the forecast that would have been available at that lead time.
|
||
|
||
- **Total contacts** (≥2020): 59,203
|
||
- **Contacts with all 6 profiles**: 12,929 (21%)
|
||
- **Generated**: 2026-08-01T22:03:32+00:00
|
||
|
||
Higher ρ = better distance prediction. Δρ(0h→Nh) = ρ(0h) − ρ(Nh) measures skill degradation — positive values mean the forecast skill is worse than using current conditions.
|
||
|
||
## Per-Band Per-Lag Spearman ρ
|
||
|
||
| Band | N | ρ(0h) | ρ(1h) | ρ(3h) | ρ(6h) | ρ(12h) | ρ(24h) | Δρ(0→6h) | Δρ(0→24h) |
|
||
|------|--:|-------:|------:|------:|------:|------:|------:|----------|------------|
|
||
| 10000 MHz | 11825 | -0.0033 | -0.0007 | +0.0201 | -0.0189 | -0.0128 | +0.0021 | +0.0156 | -0.0054 |
|
||
| 24000 MHz | 971 | +0.3140 | +0.3166 | +0.3227 | +0.3761 | +0.3629 | +0.2531 | -0.0622 | +0.0609 |
|
||
| 47000 MHz | 108 | +0.1438 | +0.1373 | +0.1644 | +0.2084 | +0.0689 | +0.1420 | -0.0646 | +0.0018 |
|
||
|
||
## Key Findings
|
||
|
||
- **Monotonic degradation** (ρ decreases with lead time): 0/3 bands
|
||
- **Non-monotonic bands** (ρ increases at some lead): 10000 MHz, 24000 MHz, 47000 MHz
|
||
- This may indicate the algorithm puts disproportionate weight on noisy short-term features that vary within hours. If ρ(6h) > ρ(0h), the current-conditions score is noisier than the 6h-lagged score — weather features at lag-0 may introduce variance that degrades the correlation.
|
||
- **ρ(6h) > ρ(0h)**: 2 bands — the 6-hour lagged score outperforms current-conditions
|
||
- **Mean Δρ(0→6h)**: -0.0371
|
||
- **Mean Δρ(0→24h)**: +0.0191
|
||
|
||
## Caveats
|
||
|
||
- **Lagged profiles use analysis data**: The HRRR profiles at T−N represent actual weather conditions at that earlier time, not a true forecast run initialized at T−N. Real HRRR forecasts would contain model error growth; these measurements capture the degradation from weather *evolution* alone (the lower bound of forecast skill loss).
|
||
- **Snap to nearest grid point**: Profiles are looked up at the HRRR grid point nearest to the contact location (rounded to 0.125°). A single point approximates the full path's conditions.
|
||
- **Filter threshold**: Only bands with ≥30 contacts having ALL 6 profiles (lag-0 through lag-24) are included in the table.
|
||
- **Censored data**: Contacts are confirmed success events only. Correlation ρ measures discrimination among successful contacts.
|
||
|