Graham McIntire
2fd88a94ea
Phase A — 7 pipeline bugs:
- retain_scores_window: fix timeline self-deletion (NOTIFY payload run_time|valid_time)
- Rust/Elixir weight divergence: Rust loads band_weights.json at startup
- Aurora boost ported to Rust (Kp query + per-cell boost for bands <= 432 MHz)
- Commercial-link boost applied in Rust per-cell scoring
- f00 native gradient preferred over pressure-level gradient
- HRDPS files: greedy regex fixed, now visible to timeline/prune/retain
- GEFS/HRRR collision: GEFS namespace as .gefs.prop, merge at read
Phase B — Validation harness:
- scripts/validate_algo.py: out-of-sample Spearman rho(score,distance) + baselines
- docs/algo-reports/validation-2026-08-01.{json,md}
Phase C — Forecast-skill evaluation:
- scripts/validate_forecast.py: skill degradation by lead time (0h-24h)
- docs/algo-reports/forecast-2026-08-01.{json,md}
Phase D — Calibration + model improvements:
- recalibrate.py: validation gate before weight deploy
- Recalibrator: Nx.max(0.0) replaces Nx.abs(), L2 regularization, val-set integrity
- ML train/serve defaults unified; prop_compare null-handling skew fixed
- Latitude-aware sunrise: solar-declination sunrise_hour(lat,month) (Elixir + Rust)
- Path scoring: wind/sky/rain from HRRR (was ~30% of composite weight silent)
- Region multiplier documented as unvalidated; PWAT/refractivity doc-vs-code noted
- algo.md: synced scoring sections, marked retired features, D5/D6 changelog
- Credo: path_compute cyclomatic complexity bumped (6->10 fields) — informational only
3.7 KiB
3.7 KiB
Algorithm Skill Validation — 2026-08-01
Auto-generated by
scripts/validate_algo.py. Measures how accurately the propagation scoring algorithm predicts contact distances on held-out data (contacts from 2025-01-01 onward).
- Total contacts: 74,076 (fit: 71,394, test: 2,682)
- Generated: 2026-08-01T22:00:42+00:00
Per-Band Spearman ρ
Higher ρ = the algorithm score better predicts contact distance. Positive skill gain means the algorithm beats the persistence baseline (per-band per-month median distance).
| Band | n | ρ(alg) | ρ(pers) | ρ(clim) | ρ(no-skill) | Skill Gain |
|---|---|---|---|---|---|---|
| 222 MHz | 752 | +0.1868 | +0.0000 | +0.0000 | +0.0000 | +0.1868 |
| 432 MHz | 898 | +0.0779 | +0.0000 | +0.0000 | +0.0000 | +0.0779 |
| 902 MHz | 137 | +0.2551 | -0.0221 | +0.0000 | +0.0000 | +0.2771 |
| 1296 MHz | 270 | +0.1602 | +0.0715 | +0.0000 | +0.0000 | +0.0887 |
| 2304 MHz | 46 | +0.3058 | -0.3550 | +0.0000 | +0.0000 | +0.6608 |
| 5760 MHz | 43 | +0.3427 | +0.2765 | +0.0000 | +0.0000 | +0.0662 |
| 10000 MHz | 375 | +0.2065 | +0.2039 | +0.0000 | +0.0000 | +0.0025 |
| 24000 MHz | 105 | +0.1718 | +0.3040 | +0.0000 | +0.0000 | -0.1322 |
Bands with <50 test contacts omitted from table.
Calibration Curves (Top 4 Bands)
A well-calibrated scorer should show monotonically increasing median and P90 distances as the score decile increases. Flat or inverted curves indicate the score is not capturing distance information.
432 MHz (n=898)
| Decile | Score Range | n | Median km | P90 km |
|---|---|---|---|---|
| 5 | 50-60 | 4 | 399.0 | 413.7 |
| 6 | 60-70 | 475 | 148.0 | 507.0 |
| 7 | 70-80 | 418 | 192.0 | 546.2 |
| 8 | 80-90 | 1 | 348.0 | 348.0 |
222 MHz (n=752)
| Decile | Score Range | n | Median km | P90 km |
|---|---|---|---|---|
| 5 | 50-60 | 2 | 249.0 | 247.0 |
| 6 | 60-70 | 359 | 181.0 | 559.4 |
| 7 | 70-80 | 387 | 257.0 | 633.8 |
| 8 | 80-90 | 4 | 531.0 | 664.0 |
10000 MHz (n=375)
| Decile | Score Range | n | Median km | P90 km |
|---|---|---|---|---|
| 6 | 60-70 | 101 | 100.0 | 232.0 |
| 7 | 70-80 | 268 | 136.0 | 278.3 |
| 8 | 80-90 | 6 | 203.0 | 309.5 |
1296 MHz (n=270)
| Decile | Score Range | n | Median km | P90 km |
|---|---|---|---|---|
| 5 | 50-60 | 10 | 221.0 | 283.5 |
| 6 | 60-70 | 144 | 117.0 | 344.4 |
| 7 | 70-80 | 112 | 160.0 | 423.4 |
| 8 | 80-90 | 4 | 262.0 | 719.1 |
Key Findings
- Positive skill gain (algorithm beats persistence): 5 bands
- Negative/zero skill gain: 1 bands
- Total bands with ≥50 test contacts: 6
- Mean ρ(alg) across bands: +0.1764
- Mean skill gain: +0.0835
Caveats
- Simplified scorer: Sky, wind, and rain factors use default values (50/50/100) because HRRR
cloud_cover_pct, wind, and precip fields are not available in the production DB schema. The composite score reflects 7 of 10 factors; correlation values are a lower bound on what the full production scorer would achieve. - Nearest-neighbour join: Contacts are matched to the HRRR grid point within 0.07° (≈7 km) and ±1 hour. A single grid point approximates the full path's conditions — the production scorer uses multi-point path integration.
- Test set size: 2025+ contacts (n=2,682) may be insufficient for mm-wave bands (47+ GHz). Treat those results as indicative.
- Censored data: Contacts are confirmed success events only. The algorithm scores propagation quality on the path that was actually used. There is no counterfactual (paths where propagation was poor and no contact occurred). Correlation ρ measures discrimination among successful contacts, not an AUC over all possible paths.