# Algorithm Skill Validation — 2026-08-01

> Auto-generated by `scripts/validate_algo.py`. Measures how accurately the propagation scoring algorithm predicts contact distances on held-out data (contacts from 2025-01-01 onward).

- **Total contacts**: 74,076 (fit: 71,394, test: 2,682)
- **Generated**: 2026-08-01T22:00:42+00:00

## Per-Band Spearman ρ

Higher ρ = the algorithm score better predicts contact distance. Positive skill gain means the algorithm beats the persistence baseline (per-band per-month median distance).

| Band | n | ρ(alg) | ρ(pers) | ρ(clim) | ρ(no-skill) | Skill Gain |
|------|--:|-------:|--------:|--------:|------------:|-----------:|
| 222 MHz | 752 | +0.1868 | +0.0000 | +0.0000 | +0.0000 | +0.1868 |
| 432 MHz | 898 | +0.0779 | +0.0000 | +0.0000 | +0.0000 | +0.0779 |
| 902 MHz | 137 | +0.2551 | -0.0221 | +0.0000 | +0.0000 | +0.2771 |
| 1296 MHz | 270 | +0.1602 | +0.0715 | +0.0000 | +0.0000 | +0.0887 |
| 2304 MHz | 46 | +0.3058 | -0.3550 | +0.0000 | +0.0000 | +0.6608 |
| 5760 MHz | 43 | +0.3427 | +0.2765 | +0.0000 | +0.0000 | +0.0662 |
| 10000 MHz | 375 | +0.2065 | +0.2039 | +0.0000 | +0.0000 | +0.0025 |
| 24000 MHz | 105 | +0.1718 | +0.3040 | +0.0000 | +0.0000 | -0.1322 |

_Bands with <50 test contacts omitted from table._

## Calibration Curves (Top 4 Bands)

A well-calibrated scorer should show monotonically increasing median and P90 distances as the score decile increases. Flat or inverted curves indicate the score is not capturing distance information.

### 432 MHz (n=898)

| Decile | Score Range | n | Median km | P90 km |
|--------|------------|--:|----------:|-------:|
| 5 | 50-60 | 4 | 399.0 | 413.7 |
| 6 | 60-70 | 475 | 148.0 | 507.0 |
| 7 | 70-80 | 418 | 192.0 | 546.2 |
| 8 | 80-90 | 1 | 348.0 | 348.0 |

### 222 MHz (n=752)

| Decile | Score Range | n | Median km | P90 km |
|--------|------------|--:|----------:|-------:|
| 5 | 50-60 | 2 | 249.0 | 247.0 |
| 6 | 60-70 | 359 | 181.0 | 559.4 |
| 7 | 70-80 | 387 | 257.0 | 633.8 |
| 8 | 80-90 | 4 | 531.0 | 664.0 |

### 10000 MHz (n=375)

| Decile | Score Range | n | Median km | P90 km |
|--------|------------|--:|----------:|-------:|
| 6 | 60-70 | 101 | 100.0 | 232.0 |
| 7 | 70-80 | 268 | 136.0 | 278.3 |
| 8 | 80-90 | 6 | 203.0 | 309.5 |

### 1296 MHz (n=270)

| Decile | Score Range | n | Median km | P90 km |
|--------|------------|--:|----------:|-------:|
| 5 | 50-60 | 10 | 221.0 | 283.5 |
| 6 | 60-70 | 144 | 117.0 | 344.4 |
| 7 | 70-80 | 112 | 160.0 | 423.4 |
| 8 | 80-90 | 4 | 262.0 | 719.1 |

## Key Findings

- **Positive skill gain** (algorithm beats persistence): 5 bands
- **Negative/zero skill gain**: 1 bands
- **Total bands with ≥50 test contacts**: 6
- **Mean ρ(alg) across bands**: +0.1764
- **Mean skill gain**: +0.0835

## Caveats

- **Simplified scorer**: Sky, wind, and rain factors use default values (50/50/100) because HRRR `cloud_cover_pct`, wind, and precip fields are not available in the production DB schema. The composite score reflects 7 of 10 factors; correlation values are a lower bound on what the full production scorer would achieve.
- **Nearest-neighbour join**: Contacts are matched to the HRRR grid point within 0.07° (≈7 km) and ±1 hour. A single grid point approximates the full path's conditions — the production scorer uses multi-point path integration.
- **Test set size**: 2025+ contacts (n=2,682) may be insufficient for mm-wave bands (47+ GHz). Treat those results as indicative.
- **Censored data**: Contacts are confirmed success events only. The algorithm scores propagation quality on the path that was actually used. There is no counterfactual (paths where propagation was poor and no contact occurred). Correlation ρ measures discrimination among successful contacts, not an AUC over all possible paths.