feat(propagation): per-band weight calibration from full-corpus correlation
Ran recalibrate_algo.py against the full local prop_dev (81,994 contacts, 18.6M HRRR rows) and derived per-band composite weights for the nine bands with >=200 matched contacts. Moisture (dewpoint/PWAT/surface N) is consistently beneficial through 5.76 GHz, reverses at 24 GHz; rain scales sqrt(rain_k) not linearly so 24 GHz gets 0.215 rain weight instead of the linear-ratio 0.95. 10 GHz stays as the reference band (defaults); 47+ GHz inherits defaults (n<200). - BandConfig.weights/1 returns per-band override or default fallback - @band_configs carries :weights on 222/432/902/1296/2304/3400/5760/24G - Recalibrator.compute_factors/3 + fit(band_mhz:) band-aware fitting - Scorer + ContactLive.Show pass band_config to weights/1 - algo.md Part 2d documents the 2026-04-18 analysis + derivation rule - scripts/derive_band_weights.py turns correlations into weight maps - report preserved at docs/algo-reports/2026-04-18-recalibration.md
This commit is contained in:
parent
41c9a5522f
commit
5fd37a17fd
10 changed files with 1320 additions and 96 deletions
184
algo.md
184
algo.md
|
|
@ -808,6 +808,131 @@ This is the first *measured* signal in the algorithm — every other factor is a
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Part 2d: Full-Corpus Per-Band Recalibration — April 18 2026
|
||||||
|
|
||||||
|
Ran `scripts/recalibrate_algo.py` against the full local `prop_dev` (81,994 contacts 1991–2026, 18.6M `hrrr_profiles` rows 2018–2026, 354 `narr_profiles` rows 1991–2014 for the HRRR-era gap, 11.4K `hrrr_native_profiles`, 7.3K `nexrad_observations`, 26.9K `soundings`). The full report is preserved at `docs/algo-reports/2026-04-18-recalibration.md`.
|
||||||
|
|
||||||
|
### Matched corpus sizes
|
||||||
|
|
||||||
|
Each contact is joined to its nearest HRRR grid point at ±0.07° / ±1h (±0.25° / ±2h for the pre-2014 NARR fallback). `DISTINCT ON (c.id)` keeps a single nearest match per contact. Pre-2014 coverage is intentionally thin: of 207 pre-2014 contacts with `pos1`, 103 matched a NARR profile — the NARR backfill has only fired on the contacts the enqueuer has seen.
|
||||||
|
|
||||||
|
| Band | Contacts in DB | HRRR-matched | NARR-matched |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 222 MHz | 7,595 | 5,392 | 0 |
|
||||||
|
| 432 MHz | 9,177 | 6,583 | 0 |
|
||||||
|
| 902 MHz | 1,794 | 1,317 | 0 |
|
||||||
|
| 1,296 MHz | 2,996 | 2,146 | 0 |
|
||||||
|
| 2,304 MHz | 749 | 564 | 0 |
|
||||||
|
| 3,400 MHz | 351 | 280 | 0 |
|
||||||
|
| 5,760 MHz | 328 | 246 | 0 |
|
||||||
|
| 10 GHz | 54,264 | 6,675 | 103 |
|
||||||
|
| 24 GHz | 3,806 | 613 | 0 |
|
||||||
|
| 47 GHz | 762 | 53 | 0 |
|
||||||
|
|
||||||
|
The per-band HRRR-match rate is 12–75%; the rest are contacts whose timestamps predate the HRRR archive start (2014-10) or whose locations don't have a nearest grid point written. The NARR join only lights up at 10 GHz because pre-2014 QSOs happen to concentrate there in the corpus.
|
||||||
|
|
||||||
|
### Per-band Pearson correlations vs contact distance
|
||||||
|
|
||||||
|
Joining each contact to its nearest HRRR profile (±0.07° / ±1h) and computing Pearson `r` between distance_km and the HRRR field at the origin station:
|
||||||
|
|
||||||
|
| Band (n) | Temp | Dewpoint | Pressure | PWAT | Surface N | dN/dh | HPBL |
|
||||||
|
|---|---|---|---|---|---|---|---|
|
||||||
|
| 222 (5,392) | −0.086 | **+0.063** | −0.038 | **+0.077** | **+0.077** | +0.057 | −0.033 |
|
||||||
|
| 432 (6,583) | −0.089 | **+0.137** | +0.058 | **+0.124** | **+0.136** | +0.041 | −0.066 |
|
||||||
|
| 902 (1,317) | −0.060 | +0.096 | +0.044 | +0.049 | +0.079 | −0.014 | −0.010 |
|
||||||
|
| 1,296 (2,146) | −0.032 | +0.085 | +0.013 | +0.049 | +0.085 | −0.031 | −0.041 |
|
||||||
|
| 2,304 (564) | −0.114 | **+0.130** | −0.006 | +0.086 | **+0.154** | −0.110 | −0.074 |
|
||||||
|
| 3,400 (280) | −0.103 | **+0.150** | −0.025 | **+0.131** | **+0.176** | +0.046 | −0.023 |
|
||||||
|
| 5,760 (246) | −0.082 | +0.087 | +0.040 | **+0.137** | +0.105 | +0.021 | −0.083 |
|
||||||
|
| 10,000 (6,675) | +0.043 | −0.028 | **−0.066** | +0.017 | +0.031 | −0.024 | −0.025 |
|
||||||
|
| 24,000 (613) | −0.007 | **−0.208** | **−0.178** | **−0.172** | −0.103 | **−0.251** | **−0.201** |
|
||||||
|
| 47,000 (53) | +0.222 | **−0.518** | **−0.423** | −0.149 | +0.510 | −0.154 | **−0.465** |
|
||||||
|
|
||||||
|
**Signs tell the story.** From 222 MHz through 5.76 GHz moisture (dewpoint, PWAT, surface refractivity) is consistently *positive* — higher column water → longer contacts, because water-vapor pressure lifts surface-layer refractivity without triggering the absorption penalty that kicks in above ~15 GHz. At 10 GHz the signs flip or collapse into noise. At 24 GHz all seven fields go negative and the magnitudes roughly triple: the same synoptic ridge that bends 222 MHz rays 250 km is also the ridge that loads 24 GHz air with 30+ mm of H₂O absorption. This is the algorithm's humidity-direction reversal made visible in the data.
|
||||||
|
|
||||||
|
`47 GHz (n=53)` is printed for interest only; the correlation magnitudes are eye-watering because the sample is tiny and Aug–Sep contest clusters. **We do not use the 47 GHz row to fit weights.**
|
||||||
|
|
||||||
|
### Per-band weight derivation
|
||||||
|
|
||||||
|
Rule (encoded in `scripts/derive_band_weights.py`):
|
||||||
|
|
||||||
|
1. Start from the global default weight vector (`BandConfig.weights/0`, the 2026-04-11 gradient-descent fit).
|
||||||
|
2. For each of the five correlation-backed factors (humidity↔dewpoint, td_depression↔temp, refractivity↔dN/dh, pressure↔pressure, pwat↔PWAT) compute a multiplier:
|
||||||
|
|
||||||
|
```
|
||||||
|
s_band,factor = (|r_band| + 0.05) / (|r_10GHz| + 0.05)
|
||||||
|
```
|
||||||
|
|
||||||
|
then clamp to [0.5, 2.0]. The 0.05 floor is the noise level at which we can reliably distinguish a correlation from zero given typical n, and the clamp keeps a single noisy correlation from moving a weight more than 2× the prior.
|
||||||
|
|
||||||
|
3. For the five physics-only factors (rain, season, time-of-day, sky, wind) use predetermined per-band multipliers:
|
||||||
|
- **Rain** scales as √(rain_k / 10 GHz rain_k). Linear scaling (rain_k ratio 7× at 24 GHz) would swamp everything; sqrt keeps the weight jump proportionate to the fraction of conditions where rain actually drives the outcome.
|
||||||
|
- **Season** scales up at VHF/UHF (Es-like physics amplifies monthly variation that we don't model directly) and at mm-wave (summer humidity hurts).
|
||||||
|
- **Time-of-day** scales with frequency per Finding 8: 0.6× at 50 MHz, 1.0× at 10 GHz, 5.0× at 122 GHz+.
|
||||||
|
- **Sky, wind** held flat — no per-band evidence.
|
||||||
|
|
||||||
|
4. Multiply default weights by the multipliers, normalize to sum to 1.0, round to 4 decimals.
|
||||||
|
|
||||||
|
Bands with fewer than 200 matched contacts (47+ GHz) and bands with zero contacts (50, 144 MHz) inherit the default vector via `BandConfig.weights/1` fallback.
|
||||||
|
|
||||||
|
### Resulting per-band weight matrix
|
||||||
|
|
||||||
|
| Band | humid | tod | td | refr | sky | season | wind | rain | pwat | press |
|
||||||
|
|---|---|---|---|---|---|---|---|---|---|---|
|
||||||
|
| **default** | 0.1243 | 0.0496 | 0.0978 | 0.1049 | 0.0800 | 0.1112 | 0.0800 | 0.1362 | 0.1128 | 0.1032 |
|
||||||
|
| 222 MHz | 0.1593 | 0.0350 | 0.1250 | 0.1401 | 0.0706 | 0.1276 | 0.0706 | **0.0120** | **0.1916** | 0.0681 |
|
||||||
|
| 432 MHz | **0.2061** | 0.0329 | 0.1200 | 0.1186 | 0.0663 | 0.1106 | 0.0663 | **0.0113** | **0.1870** | 0.0809 |
|
||||||
|
| 902 MHz | **0.2201** | 0.0437 | 0.1102 | 0.0888 | 0.0783 | 0.1197 | 0.0783 | **0.0133** | **0.1635** | 0.0839 |
|
||||||
|
| 1,296 MHz | **0.2131** | 0.0494 | 0.0898 | 0.1392 | 0.0797 | 0.1108 | 0.0797 | **0.0136** | 0.1678 | 0.0568 |
|
||||||
|
| 2,304 MHz | 0.1941 | 0.0387 | 0.1385 | **0.1638** | 0.0625 | 0.0868 | 0.0625 | 0.0336 | 0.1762 | 0.0432 |
|
||||||
|
| 3,400 MHz | 0.1996 | 0.0398 | 0.1251 | 0.1310 | 0.0642 | 0.0893 | 0.0642 | 0.0489 | 0.1811 | 0.0568 |
|
||||||
|
| 5,760 MHz | 0.1829 | 0.0423 | 0.1205 | 0.0881 | 0.0683 | 0.0949 | 0.0683 | 0.0822 | **0.1925** | 0.0602 |
|
||||||
|
| 10 GHz | *defaults (reference band)* | | | | | | | | | |
|
||||||
|
| 24 GHz | 0.1481 | 0.0532 | **0.0360** | 0.1250 | 0.0477 | 0.0729 | 0.0477 | **0.2147** | 0.1344 | 0.1203 |
|
||||||
|
|
||||||
|
Cells in bold are the factors that moved meaningfully from default — roughly, what the data says is *different* at this band. The overall pattern:
|
||||||
|
|
||||||
|
- **VHF/UHF (222–1296 MHz):** PWAT and humidity up, rain and pressure down. These bands need moisture to duct; rain doesn't attenuate them; surface pressure is a weak synoptic proxy that cleaner moisture fields already capture.
|
||||||
|
- **Low microwave (2,304–5,760 MHz):** Moisture still dominant; refractivity gradient picks up weight as HRRR's vertical resolution starts to matter; rain gradually climbs.
|
||||||
|
- **10 GHz:** Reference band, weights unchanged.
|
||||||
|
- **24 GHz:** Rain doubles-plus (0.215), td_depression collapses to 0.036 because the temperature signal alone is essentially zero (humidity via dewpoint already carries the moisture story). Refractivity holds because dN/dh shows the strongest correlation of any band at 24 GHz (−0.25).
|
||||||
|
|
||||||
|
### Soundings refresh (n=26,933 total, 9,574 with refractivity gradient)
|
||||||
|
|
||||||
|
Monthly ducting probability from the expanded sounding corpus. Compared to the Part 2c snapshot (n=6,757), August's sample nearly doubled (2,572 → 5,007) as soundings backfill caught up with the QSO calendar. The headline findings hold: **March is still the worst month** (17.3% ducting), **June–July the peak** (67–70%).
|
||||||
|
|
||||||
|
| Month | Soundings | Ducting % | Avg dN/dh | Avg PWAT (mm) |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| Jan | 95 | 28.4% | −171 | 10.9 |
|
||||||
|
| Feb | 131 | 29.0% | −163 | 7.9 |
|
||||||
|
| Mar | 81 | **17.3%** | −148 | 9.4 |
|
||||||
|
| Apr | 134 | 31.3% | −176 | 13.5 |
|
||||||
|
| May | 311 | 51.8% | −286 | 25.6 |
|
||||||
|
| Jun | 380 | **69.5%** | −356 | 32.0 |
|
||||||
|
| Jul | 302 | 67.5% | −294 | 29.7 |
|
||||||
|
| Aug | 5,007 | 55.2% | −255 | 35.2 |
|
||||||
|
| Sep | 2,335 | 56.2% | −280 | 27.2 |
|
||||||
|
| Oct | 155 | 57.4% | −307 | 19.3 |
|
||||||
|
| Nov | 158 | 50.0% | −257 | 12.0 |
|
||||||
|
| Dec | 140 | 25.7% | −185 | 10.8 |
|
||||||
|
|
||||||
|
### What stayed, what left
|
||||||
|
|
||||||
|
- **Humidity direction flip at ~15 GHz** — preserved. Data shows it clearly (signs flip going from 5.76 GHz to 10 GHz and again from 10 GHz to 24 GHz).
|
||||||
|
- **Pressure is the strongest 10 GHz correlator** — preserved, but magnitude is now −0.066 (vs −0.180 reported in Part 2b); the newer matched corpus has cleaner spatial joins and less contest-seasonality bias.
|
||||||
|
- **HPBL is negative at every band we can fit** — preserved. Already folded into refractivity as a multiplier.
|
||||||
|
- **Per-band recalibration is statistically defensible** — the Part 2c moratorium lifts. 9 bands have n ≥ 200 matched contacts.
|
||||||
|
- **NARR historical calibration — deferred.** 103 pre-2014 matches at 10 GHz isn't enough to validate the weights against the earlier atmosphere; the NARR backfill queue is still draining. Re-run when `narr_profiles` crosses ~10K rows.
|
||||||
|
- **Commercial-link sensor calibration — deferred.** Zero DFW-zone contact overlap with the 20-day commercial_samples window. Re-run after the next contest season.
|
||||||
|
|
||||||
|
### Open items
|
||||||
|
|
||||||
|
1. **50 MHz and 144 MHz are unmodeled (0 contacts in corpus).** Both bands carry default weights but ranges and seasonal tables are pure physics priors. Any VHF calibration needs an import of 6 m and 2 m contacts from external logs before we can say anything.
|
||||||
|
2. **47+ GHz inherit defaults.** 47 GHz has n=53, below the 200-contact floor; 75 GHz has n=106 in the DB but only a handful HRRR-match. Re-fit when contest-season 47/75 GHz logs accumulate.
|
||||||
|
3. **Refractivity gradient signal is load-bearing only at 24 GHz.** At every other band, the gradient correlation rides the noise floor. Moving to native-resolution HRRR (`hrrr_native_profiles.best_duct_band_ghz`) is how this gets better; the data in this table shows 10–15× cleaner gradients when we have a duct-supporting cell.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Part 3: Band Configuration
|
## Part 3: Band Configuration
|
||||||
|
|
||||||
```elixir
|
```elixir
|
||||||
|
|
@ -1368,18 +1493,11 @@ Calibration plan once backfill is complete:
|
||||||
|
|
||||||
## Part 5: Composite Score
|
## Part 5: Composite Score
|
||||||
|
|
||||||
### Weights
|
### Weights — per-band since 2026-04-18
|
||||||
|
|
||||||
Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12, 72% improvement). Five additional upper-air factors (described at the end of Part 4) are queued for the next recalibration — they cannot be fit until the native-profile backfill reaches enough QSO hours to produce a representative training sample. Key changes from the April 2026 manual calibration:
|
The scoring weight vector is now band-specific. `BandConfig.weights(band_config)` returns either the override map stored on the band (for the nine bands with ≥200 matched HRRR samples in the 2026-04-18 full-corpus analysis) or the default vector shown below for everything else.
|
||||||
|
|
||||||
- **Rain: 8% → 13.6%** — Largest increase. Gradient descent found rain is a stronger discriminator than manual analysis suggested.
|
**Default vector** (the April-11 gradient-descent fit, applied at 10 GHz as the reference band and to any band without enough data to fit its own vector):
|
||||||
- **Season: 8% → 11.1%** — Seasonal patterns are more predictive than the correlation analysis indicated (correlations were suppressed by Aug/Sep dataset bias).
|
|
||||||
- **Refractivity: 8% → 10.5%** — Now sourced from native HRRR hybrid-sigma levels (10-50m resolution) when available, resolving thin surface ducts invisible to the 250m pressure-level product. Higher weight justified by higher-quality input data.
|
|
||||||
- **PWAT: 10% → 11.3%** — Confirmed as strong independent predictor.
|
|
||||||
- **Pressure: 15% → 10.3%** — Was overweighted; pressure is a proxy for frontal activity but partially redundant with rain and refractivity.
|
|
||||||
- **Humidity: 18% → 12.4%** — Partially captured by PWAT and Td depression.
|
|
||||||
- **Time of Day: 10% → 5.0%** — Contest timing bias inflated this factor; real diurnal effect is weaker at the dominant 10 GHz band.
|
|
||||||
- **Sky, Wind: 8%, 5% → 8%, 8%** — Hit minimum floor; gradient descent would go lower but physical rationale retains them.
|
|
||||||
|
|
||||||
| Factor | Weight | Source |
|
| Factor | Weight | Source |
|
||||||
|--------|--------|--------|
|
|--------|--------|--------|
|
||||||
|
|
@ -1387,31 +1505,51 @@ Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12,
|
||||||
| Humidity | 12.4% | Absolute humidity from surface T/Td |
|
| Humidity | 12.4% | Absolute humidity from surface T/Td |
|
||||||
| PWAT | 11.3% | HRRR precipitable water (column-integrated) |
|
| PWAT | 11.3% | HRRR precipitable water (column-integrated) |
|
||||||
| Season | 11.1% | Per-band monthly lookup tables |
|
| Season | 11.1% | Per-band monthly lookup tables |
|
||||||
| Refractivity | 10.5% | Native HRRR dM/dh (10-50m), fallback to pressure-level (250m) |
|
| Refractivity | 10.5% | Native HRRR dM/dh (10–50m), fallback to pressure-level (250m) |
|
||||||
| Pressure | 10.3% | Surface pressure (frontal activity proxy) |
|
| Pressure | 10.3% | Surface pressure (frontal activity proxy) |
|
||||||
| Td Depression | 9.8% | Surface T minus Td (stability indicator) |
|
| Td Depression | 9.8% | Surface T minus Td (stability indicator) |
|
||||||
| Sky Cover | 8.0% | HRRR total cloud cover |
|
| Sky Cover | 8.0% | HRRR total cloud cover |
|
||||||
| Wind | 8.0% | HRRR 10m wind speed |
|
| Wind | 8.0% | HRRR 10m wind speed |
|
||||||
| Time of Day | 5.0% | Solar-time adjusted diurnal cycle |
|
| Time of Day | 5.0% | Solar-time adjusted diurnal cycle |
|
||||||
|
|
||||||
|
**Per-band overrides** (see Part 2d for the derivation rule and full matrix):
|
||||||
|
|
||||||
|
| Band | humidity | tod | td | refr | sky | season | wind | rain | pwat | press |
|
||||||
|
|---|---|---|---|---|---|---|---|---|---|---|
|
||||||
|
| 222 MHz | 0.1593 | 0.0350 | 0.1250 | 0.1401 | 0.0706 | 0.1276 | 0.0706 | 0.0120 | 0.1916 | 0.0681 |
|
||||||
|
| 432 MHz | 0.2061 | 0.0329 | 0.1200 | 0.1186 | 0.0663 | 0.1106 | 0.0663 | 0.0113 | 0.1870 | 0.0809 |
|
||||||
|
| 902 MHz | 0.2201 | 0.0437 | 0.1102 | 0.0888 | 0.0783 | 0.1197 | 0.0783 | 0.0133 | 0.1635 | 0.0839 |
|
||||||
|
| 1.296 GHz | 0.2131 | 0.0494 | 0.0898 | 0.1392 | 0.0797 | 0.1108 | 0.0797 | 0.0136 | 0.1678 | 0.0568 |
|
||||||
|
| 2.304 GHz | 0.1941 | 0.0387 | 0.1385 | 0.1638 | 0.0625 | 0.0868 | 0.0625 | 0.0336 | 0.1762 | 0.0432 |
|
||||||
|
| 3.4 GHz | 0.1996 | 0.0398 | 0.1251 | 0.1310 | 0.0642 | 0.0893 | 0.0642 | 0.0489 | 0.1811 | 0.0568 |
|
||||||
|
| 5.76 GHz | 0.1829 | 0.0423 | 0.1205 | 0.0881 | 0.0683 | 0.0949 | 0.0683 | 0.0822 | 0.1925 | 0.0602 |
|
||||||
|
| 10 GHz | *defaults (reference)* | | | | | | | | | |
|
||||||
|
| 24 GHz | 0.1481 | 0.0532 | 0.0360 | 0.1250 | 0.0477 | 0.0729 | 0.0477 | 0.2147 | 0.1344 | 0.1203 |
|
||||||
|
|
||||||
|
Bands inheriting defaults: 50, 144 (0 contacts), 47+ GHz (<200 matched contacts). The `Scorer` composite is now:
|
||||||
|
|
||||||
```elixir
|
```elixir
|
||||||
# Recalibrated 2026-04-11 via gradient descent on 5000 QSOs (loss 0.42 → 0.12)
|
def composite_score(conditions, band_config) do
|
||||||
def composite_score(factors) do
|
factors = compute_factors(conditions, band_config)
|
||||||
|
weights = BandConfig.weights(band_config)
|
||||||
|
|
||||||
round(
|
round(
|
||||||
factors.rain * 0.1362 +
|
factors.rain * weights.rain +
|
||||||
factors.humidity * 0.1243 +
|
factors.humidity * weights.humidity +
|
||||||
factors.pwat * 0.1128 +
|
factors.pwat * weights.pwat +
|
||||||
factors.season * 0.1112 +
|
factors.season * weights.season +
|
||||||
factors.refractivity * 0.1049 +
|
factors.refractivity * weights.refractivity +
|
||||||
factors.pressure * 0.1032 +
|
factors.pressure * weights.pressure +
|
||||||
factors.td_depression * 0.0978 +
|
factors.td_depression * weights.td_depression +
|
||||||
factors.sky * 0.08 +
|
factors.sky * weights.sky +
|
||||||
factors.wind * 0.08 +
|
factors.wind * weights.wind +
|
||||||
factors.time_of_day * 0.0496
|
factors.time_of_day * weights.time_of_day
|
||||||
)
|
)
|
||||||
end
|
end
|
||||||
```
|
```
|
||||||
|
|
||||||
|
All weight vectors sum to 1.0 within round-off (verified by `BandConfigTest`). Re-run `scripts/recalibrate_algo.py` + `scripts/derive_band_weights.py` whenever the contact or HRRR corpus grows materially — the defaults in this section are a snapshot of the 2026-04-18 local `prop_dev` state.
|
||||||
|
|
||||||
### Score Tiers with Per-Band Range Estimates
|
### Score Tiers with Per-Band Range Estimates
|
||||||
|
|
||||||
Range estimates are for CW mode. For SSB/phone, reduce by ~25% at 10 GHz, ~15% at 24 GHz, ~50% at 47 GHz, ~70% at 75+ GHz. For FM, reduce by ~40%.
|
Range estimates are for CW mode. For SSB/phone, reduce by ~25% at 10 GHz, ~15% at 24 GHz, ~50% at 47 GHz, ~70% at 75+ GHz. For FM, reduce by ~40%.
|
||||||
|
|
|
||||||
442
docs/algo-reports/2026-04-18-recalibration.md
Normal file
442
docs/algo-reports/2026-04-18-recalibration.md
Normal file
|
|
@ -0,0 +1,442 @@
|
||||||
|
# Recalibration Analysis Report — 2026-04-18
|
||||||
|
|
||||||
|
> Auto-generated by `scripts/recalibrate_algo.py`. This report is the
|
||||||
|
> input for `algo.md` updates — diff against the prior dated section to
|
||||||
|
> see what's actually moved.
|
||||||
|
|
||||||
|
Connection: `postgresql://postgres@localhost:5432/prop_dev`
|
||||||
|
Statement timeout: 60min
|
||||||
|
|
||||||
|
|
||||||
|
## Row counts and date ranges
|
||||||
|
|
||||||
|
| tbl | n | lo | hi |
|
||||||
|
|:---------------------|-------------:|:-----------|:-----------|
|
||||||
|
| contacts | 81994.000 | 1991-05-04 | 2026-03-07 |
|
||||||
|
| hrrr_native_profiles | 11472.000 | 2019-03-09 | 2025-09-21 |
|
||||||
|
| hrrr_profiles | 18591740.000 | 2018-08-04 | 2026-04-05 |
|
||||||
|
| iemre_observations | 28438.000 | | |
|
||||||
|
| narr_profiles | 354.000 | 1991-05-04 | 2014-09-19 |
|
||||||
|
| nexrad_observations | 7286.000 | 2019-08-17 | 2024-09-22 |
|
||||||
|
| propagation_scores | nan | | |
|
||||||
|
| rtma_observations | 0.000 | | |
|
||||||
|
| soundings | 26933.000 | 1990-06-07 | 2026-04-18 |
|
||||||
|
| surface_observations | 599046.000 | 1991-05-03 | 2026-04-18 |
|
||||||
|
| terrain_profiles | 81994.000 | | |
|
||||||
|
|
||||||
|
|
||||||
|
## Data gaps (these should NOT be zero in a healthy prod)
|
||||||
|
|
||||||
|
- **narr_rows**: 354
|
||||||
|
- **narr_pre2014_rows**: 354
|
||||||
|
- **hrrr_climatology_rows**: 0
|
||||||
|
- **rtma_rows**: 0
|
||||||
|
- **metar5_rows**: 0
|
||||||
|
- **hrrr_complete_contacts**: 67260
|
||||||
|
- **hrrr_pending_contacts**: 14734
|
||||||
|
- **pre2014_contacts**: 207
|
||||||
|
|
||||||
|
|
||||||
|
## Contacts by band (≥50 MHz)
|
||||||
|
|
||||||
|
| band_mhz | contacts | avg_km | max_km |
|
||||||
|
|-----------:|-----------:|---------:|---------:|
|
||||||
|
| 222 | 7595 | 229.000 | 2306.000 |
|
||||||
|
| 432 | 9177 | 200.000 | 1752.000 |
|
||||||
|
| 902 | 1794 | 171.000 | 915.000 |
|
||||||
|
| 1296 | 2996 | 160.000 | 1752.000 |
|
||||||
|
| 2304 | 749 | 153.000 | 655.000 |
|
||||||
|
| 3400 | 351 | 151.000 | 643.000 |
|
||||||
|
| 5760 | 328 | 125.000 | 505.000 |
|
||||||
|
| 10000 | 54264 | 212.000 | 2393.000 |
|
||||||
|
| 24000 | 3806 | 95.000 | 710.000 |
|
||||||
|
| 47000 | 762 | 65.000 | 343.000 |
|
||||||
|
| 75000 | 106 | 60.000 | 289.000 |
|
||||||
|
| 122000 | 35 | 27.000 | 139.000 |
|
||||||
|
| 134000 | 5 | 65.000 | 157.000 |
|
||||||
|
| 142000 | 4 | 47.000 | 79.700 |
|
||||||
|
| 145000 | 2 | 40.000 | 79.600 |
|
||||||
|
| 241000 | 14 | 31.000 | 114.400 |
|
||||||
|
| 288000 | 1 | 1.000 | 1.246 |
|
||||||
|
| 322000 | 1 | 1.000 | 1.400 |
|
||||||
|
| 403000 | 1 | 1.000 | 1.400 |
|
||||||
|
| 411000 | 1 | 0.000 | 0.050 |
|
||||||
|
|
||||||
|
|
||||||
|
## Monthly sounding ducting
|
||||||
|
|
||||||
|
| month | soundings | ducting_pct | avg_min_grad | avg_pwat |
|
||||||
|
|--------:|------------:|--------------:|---------------:|-----------:|
|
||||||
|
| 1 | 95 | 28.400 | -171.400 | 10.900 |
|
||||||
|
| 2 | 131 | 29.000 | -163.100 | 7.900 |
|
||||||
|
| 3 | 81 | 17.300 | -148.100 | 9.400 |
|
||||||
|
| 4 | 134 | 31.300 | -175.600 | 13.500 |
|
||||||
|
| 5 | 311 | 51.800 | -286.400 | 25.600 |
|
||||||
|
| 6 | 380 | 69.500 | -355.900 | 32.000 |
|
||||||
|
| 7 | 302 | 67.500 | -293.800 | 29.700 |
|
||||||
|
| 8 | 5007 | 55.200 | -255.400 | 35.200 |
|
||||||
|
| 9 | 2335 | 56.200 | -279.800 | 27.200 |
|
||||||
|
| 10 | 155 | 57.400 | -307.100 | 19.300 |
|
||||||
|
| 11 | 158 | 50.000 | -256.700 | 12.000 |
|
||||||
|
| 12 | 140 | 25.700 | -185.200 | 10.800 |
|
||||||
|
|
||||||
|
|
||||||
|
## HRRR native-profile duct distribution (best supportable band)
|
||||||
|
|
||||||
|
| band_bin | profiles | avg_inv_top_m | avg_theta_e_jump | avg_richardson |
|
||||||
|
|:-----------|-----------:|----------------:|-------------------:|-----------------:|
|
||||||
|
| <5 GHz | 1432 | 2097.000 | 5.400 | 18.420 |
|
||||||
|
| 15-30 GHz | 10 | 2091.000 | 5.900 | 8.690 |
|
||||||
|
| 30-75 GHz | 4 | 3860.000 | 5.000 | 12.120 |
|
||||||
|
| 5-15 GHz | 98 | 4408.000 | 25.700 | 12.550 |
|
||||||
|
| none | 9928 | 11497.000 | 80.500 | 38.270 |
|
||||||
|
|
||||||
|
|
||||||
|
## Contact ↔ HRRR per-band Pearson correlations (matched n=23885)
|
||||||
|
|
||||||
|
| band_mhz | n | rho_pr | rho_dpc | rho_pwat | rho_sref | rho_grad | rho_tc | rho_hpbl |
|
||||||
|
|-----------:|---------:|---------:|----------:|-----------:|-----------:|-----------:|---------:|-----------:|
|
||||||
|
| 222.000 | 5392.000 | -0.038 | 0.063 | 0.077 | 0.077 | 0.057 | -0.086 | -0.033 |
|
||||||
|
| 432.000 | 6583.000 | 0.058 | 0.137 | 0.124 | 0.136 | 0.041 | -0.089 | -0.066 |
|
||||||
|
| 902.000 | 1317.000 | 0.044 | 0.096 | 0.049 | 0.079 | -0.014 | -0.060 | -0.010 |
|
||||||
|
| 1296.000 | 2146.000 | 0.013 | 0.085 | 0.049 | 0.085 | -0.031 | -0.032 | -0.041 |
|
||||||
|
| 2304.000 | 564.000 | -0.006 | 0.130 | 0.086 | 0.154 | -0.110 | -0.114 | -0.074 |
|
||||||
|
| 3400.000 | 280.000 | -0.025 | 0.150 | 0.131 | 0.176 | 0.046 | -0.103 | -0.023 |
|
||||||
|
| 5760.000 | 246.000 | 0.040 | 0.087 | 0.137 | 0.105 | 0.021 | -0.082 | -0.083 |
|
||||||
|
| 10000.000 | 6675.000 | -0.066 | -0.028 | 0.017 | 0.031 | -0.024 | 0.043 | -0.025 |
|
||||||
|
| 24000.000 | 613.000 | -0.178 | -0.208 | -0.172 | -0.103 | -0.251 | -0.007 | -0.201 |
|
||||||
|
| 47000.000 | 53.000 | -0.423 | -0.518 | -0.149 | 0.510 | -0.154 | 0.222 | -0.465 |
|
||||||
|
|
||||||
|
|
||||||
|
## Per-band HPBL bin distance distribution
|
||||||
|
|
||||||
|
One table per band with ≥50 matched contacts. A band-specific effect here means HPBL belongs in the scorer for that band.
|
||||||
|
|
||||||
|
|
||||||
|
**222 MHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|-----:|---------:|---------:|
|
||||||
|
| <200 | 956 | 273.500 | 210.000 |
|
||||||
|
| 200-500 | 1131 | 215.500 | 164.000 |
|
||||||
|
| 500-1000 | 1185 | 205.400 | 152.000 |
|
||||||
|
| 1000-1500 | 948 | 220.600 | 156.000 |
|
||||||
|
| 1500-2000 | 913 | 223.000 | 171.000 |
|
||||||
|
| >=2000 | 259 | 246.900 | 192.000 |
|
||||||
|
|
||||||
|
|
||||||
|
**432 MHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|-----:|---------:|---------:|
|
||||||
|
| <200 | 1258 | 223.000 | 179.000 |
|
||||||
|
| 200-500 | 1370 | 198.600 | 143.500 |
|
||||||
|
| 500-1000 | 1413 | 183.700 | 131.000 |
|
||||||
|
| 1000-1500 | 1124 | 198.000 | 145.000 |
|
||||||
|
| 1500-2000 | 1129 | 187.100 | 142.000 |
|
||||||
|
| >=2000 | 289 | 169.200 | 126.000 |
|
||||||
|
|
||||||
|
|
||||||
|
**902 MHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <200 | 215 | 205.300 | 165.000 |
|
||||||
|
| 200-500 | 275 | 152.500 | 133.000 |
|
||||||
|
| 500-1000 | 302 | 166.500 | 123.000 |
|
||||||
|
| 1000-1500 | 259 | 175.300 | 137.000 |
|
||||||
|
| 1500-2000 | 222 | 175.900 | 152.000 |
|
||||||
|
| >=2000 | 44 | 158.700 | 131.000 |
|
||||||
|
|
||||||
|
|
||||||
|
**1.296 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <200 | 355 | 179.600 | 151.000 |
|
||||||
|
| 200-500 | 484 | 159.300 | 129.500 |
|
||||||
|
| 500-1000 | 479 | 145.600 | 107.000 |
|
||||||
|
| 1000-1500 | 376 | 154.800 | 131.000 |
|
||||||
|
| 1500-2000 | 380 | 153.600 | 122.000 |
|
||||||
|
| >=2000 | 72 | 143.700 | 108.000 |
|
||||||
|
|
||||||
|
|
||||||
|
**2.304 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <200 | 72 | 188.000 | 160.000 |
|
||||||
|
| 200-500 | 103 | 174.800 | 156.000 |
|
||||||
|
| 500-1000 | 155 | 150.300 | 122.000 |
|
||||||
|
| 1000-1500 | 121 | 151.600 | 136.000 |
|
||||||
|
| 1500-2000 | 97 | 161.000 | 147.000 |
|
||||||
|
| >=2000 | 16 | 144.800 | 140.500 |
|
||||||
|
|
||||||
|
|
||||||
|
**3.4 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <200 | 39 | 157.400 | 131.000 |
|
||||||
|
| 200-500 | 57 | 162.500 | 140.000 |
|
||||||
|
| 500-1000 | 64 | 153.500 | 111.000 |
|
||||||
|
| 1000-1500 | 67 | 150.500 | 136.000 |
|
||||||
|
| 1500-2000 | 39 | 150.300 | 122.000 |
|
||||||
|
| >=2000 | 14 | 157.400 | 140.500 |
|
||||||
|
|
||||||
|
|
||||||
|
**5.76 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <200 | 24 | 174.500 | 138.500 |
|
||||||
|
| 200-500 | 49 | 134.600 | 135.000 |
|
||||||
|
| 500-1000 | 62 | 118.000 | 103.000 |
|
||||||
|
| 1000-1500 | 54 | 125.000 | 133.000 |
|
||||||
|
| 1500-2000 | 45 | 114.600 | 122.000 |
|
||||||
|
| >=2000 | 12 | 142.700 | 140.500 |
|
||||||
|
|
||||||
|
|
||||||
|
**10 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|-----:|---------:|---------:|
|
||||||
|
| <200 | 1573 | 213.900 | 178.800 |
|
||||||
|
| 200-500 | 2146 | 201.300 | 175.000 |
|
||||||
|
| 500-1000 | 1663 | 211.800 | 158.100 |
|
||||||
|
| 1000-1500 | 742 | 214.500 | 212.500 |
|
||||||
|
| 1500-2000 | 410 | 179.400 | 173.000 |
|
||||||
|
| >=2000 | 138 | 201.000 | 173.000 |
|
||||||
|
|
||||||
|
|
||||||
|
**24 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <200 | 115 | 120.600 | 110.000 |
|
||||||
|
| 200-500 | 236 | 95.400 | 99.000 |
|
||||||
|
| 500-1000 | 189 | 79.700 | 80.800 |
|
||||||
|
| 1000-1500 | 44 | 103.800 | 99.000 |
|
||||||
|
| 1500-2000 | 10 | 46.800 | 34.000 |
|
||||||
|
| >=2000 | 19 | 69.200 | 23.200 |
|
||||||
|
|
||||||
|
|
||||||
|
**47 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <200 | 9 | 111.200 | 85.300 |
|
||||||
|
| 200-500 | 19 | 82.200 | 58.500 |
|
||||||
|
| 500-1000 | 16 | 55.600 | 27.600 |
|
||||||
|
| 1000-1500 | 5 | 16.100 | 11.400 |
|
||||||
|
| >=2000 | 4 | 31.800 | 31.800 |
|
||||||
|
|
||||||
|
|
||||||
|
## Per-band pressure bin distance distribution
|
||||||
|
|
||||||
|
Surface pressure tends to proxy synoptic-scale stagnation, so we want to see longer distances under the 1015-1025 mb ridge at every band that ducts.
|
||||||
|
|
||||||
|
|
||||||
|
**222 MHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|-----:|---------:|---------:|
|
||||||
|
| <990 | 1967 | 228.800 | 177.000 |
|
||||||
|
| 990-1000 | 1289 | 248.500 | 191.000 |
|
||||||
|
| 1000-1010 | 1263 | 235.400 | 186.000 |
|
||||||
|
| 1010-1020 | 778 | 183.300 | 141.000 |
|
||||||
|
| >=1020 | 95 | 159.300 | 126.000 |
|
||||||
|
|
||||||
|
|
||||||
|
**432 MHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|-----:|---------:|---------:|
|
||||||
|
| <990 | 2548 | 189.600 | 140.000 |
|
||||||
|
| 990-1000 | 1580 | 223.200 | 173.000 |
|
||||||
|
| 1000-1010 | 1285 | 202.700 | 149.000 |
|
||||||
|
| 1010-1020 | 1050 | 172.700 | 129.000 |
|
||||||
|
| >=1020 | 120 | 144.800 | 84.500 |
|
||||||
|
|
||||||
|
|
||||||
|
**902 MHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <990 | 422 | 157.600 | 131.500 |
|
||||||
|
| 990-1000 | 400 | 206.400 | 157.000 |
|
||||||
|
| 1000-1010 | 241 | 189.200 | 147.000 |
|
||||||
|
| 1010-1020 | 223 | 138.800 | 143.000 |
|
||||||
|
| >=1020 | 31 | 72.000 | 32.000 |
|
||||||
|
|
||||||
|
|
||||||
|
**1.296 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <990 | 696 | 149.600 | 122.000 |
|
||||||
|
| 990-1000 | 579 | 183.700 | 145.000 |
|
||||||
|
| 1000-1010 | 451 | 161.600 | 108.000 |
|
||||||
|
| 1010-1020 | 385 | 130.500 | 114.000 |
|
||||||
|
| >=1020 | 35 | 112.700 | 112.000 |
|
||||||
|
|
||||||
|
|
||||||
|
**2.304 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <990 | 138 | 155.100 | 136.000 |
|
||||||
|
| 990-1000 | 201 | 168.200 | 141.000 |
|
||||||
|
| 1000-1010 | 97 | 178.500 | 117.000 |
|
||||||
|
| 1010-1020 | 116 | 147.200 | 152.000 |
|
||||||
|
| >=1020 | 12 | 126.300 | 137.000 |
|
||||||
|
|
||||||
|
|
||||||
|
**3.4 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <990 | 75 | 157.400 | 138.000 |
|
||||||
|
| 990-1000 | 134 | 156.800 | 135.000 |
|
||||||
|
| 1000-1010 | 34 | 179.700 | 131.000 |
|
||||||
|
| 1010-1020 | 28 | 122.200 | 124.000 |
|
||||||
|
| >=1020 | 9 | 114.600 | 131.000 |
|
||||||
|
|
||||||
|
|
||||||
|
**5.76 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <990 | 55 | 112.700 | 111.000 |
|
||||||
|
| 990-1000 | 120 | 131.600 | 119.500 |
|
||||||
|
| 1000-1010 | 27 | 128.000 | 100.000 |
|
||||||
|
| 1010-1020 | 36 | 146.900 | 152.000 |
|
||||||
|
| >=1020 | 8 | 123.000 | 135.000 |
|
||||||
|
|
||||||
|
|
||||||
|
**10 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|-----:|---------:|---------:|
|
||||||
|
| <990 | 3515 | 222.700 | 196.600 |
|
||||||
|
| 990-1000 | 1620 | 189.700 | 152.700 |
|
||||||
|
| 1000-1010 | 803 | 194.200 | 153.000 |
|
||||||
|
| 1010-1020 | 651 | 182.900 | 133.900 |
|
||||||
|
| >=1020 | 86 | 187.700 | 158.100 |
|
||||||
|
|
||||||
|
|
||||||
|
**24 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <990 | 282 | 97.300 | 98.700 |
|
||||||
|
| 990-1000 | 200 | 97.900 | 99.000 |
|
||||||
|
| 1000-1010 | 57 | 95.200 | 102.300 |
|
||||||
|
| 1010-1020 | 70 | 70.600 | 80.800 |
|
||||||
|
| >=1020 | 4 | 103.600 | 103.600 |
|
||||||
|
|
||||||
|
|
||||||
|
**47 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|----:|---------:|---------:|
|
||||||
|
| <990 | 19 | 80.500 | 43.900 |
|
||||||
|
| 990-1000 | 25 | 62.600 | 31.800 |
|
||||||
|
| 1000-1010 | 2 | 84.600 | 84.600 |
|
||||||
|
| 1010-1020 | 7 | 56.400 | 58.500 |
|
||||||
|
|
||||||
|
|
||||||
|
## Contact ↔ NARR per-band Pearson correlations (pre-2014 only, matched n=201)
|
||||||
|
|
||||||
|
Historical sanity check: if the HRRR-era signs and magnitudes survive on 30+ years of NARR reanalysis, the scoring factors are physical rather than HRRR-artefact. Diverging signs = flag it.
|
||||||
|
|
||||||
|
| band_mhz | n | rho_pr | rho_dpc | rho_pwat | rho_sref | rho_grad | rho_tc | rho_hpbl |
|
||||||
|
|-----------:|--------:|---------:|----------:|-----------:|-----------:|-----------:|---------:|-----------:|
|
||||||
|
| 10000.000 | 103.000 | -0.402 | -0.402 | -0.402 | -0.220 | -0.248 | -0.402 | -0.402 |
|
||||||
|
|
||||||
|
|
||||||
|
## NEXRAD composite reflectivity vs distance (matched n=20900)
|
||||||
|
|
||||||
|
Rain-attenuation sanity check — one table per band with ≥50 matched contacts. At rain-sensitive bands (24+ GHz) higher `max_dbz` should map to shorter contacts; at 10 GHz and below the effect should be barely detectable.
|
||||||
|
|
||||||
|
|
||||||
|
**10 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:-----------------|------:|---------:|---------:|
|
||||||
|
| none (<5) | 14110 | 210.000 | 197.000 |
|
||||||
|
| drizzle (5-20) | 1284 | 200.900 | 199.300 |
|
||||||
|
| light (20-30) | 1736 | 211.800 | 197.100 |
|
||||||
|
| moderate (30-45) | 1609 | 230.900 | 215.200 |
|
||||||
|
| heavy (>=45) | 677 | 191.900 | 146.900 |
|
||||||
|
|
||||||
|
|
||||||
|
**24 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:-----------------|----:|---------:|---------:|
|
||||||
|
| none (<5) | 869 | 90.900 | 83.700 |
|
||||||
|
| drizzle (5-20) | 74 | 108.300 | 123.200 |
|
||||||
|
| light (20-30) | 151 | 88.200 | 70.900 |
|
||||||
|
| moderate (30-45) | 76 | 109.300 | 122.400 |
|
||||||
|
| heavy (>=45) | 49 | 86.600 | 102.300 |
|
||||||
|
|
||||||
|
|
||||||
|
**47 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:-----------------|----:|---------:|---------:|
|
||||||
|
| none (<5) | 186 | 52.400 | 56.000 |
|
||||||
|
| light (20-30) | 30 | 59.300 | 66.300 |
|
||||||
|
| moderate (30-45) | 9 | 75.100 | 81.600 |
|
||||||
|
| heavy (>=45) | 15 | 71.400 | 82.800 |
|
||||||
|
|
||||||
|
|
||||||
|
## hrrr_native_profiles.best_duct_band_ghz vs distance (matched n=57669)
|
||||||
|
|
||||||
|
Validates the 1.15× boost in `Scorer.score_refractivity/4`: contacts where the native duct supports the target band should run longer than those where it does not. One table per band with ≥50 matched contacts.
|
||||||
|
|
||||||
|
|
||||||
|
**10 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|------:|---------:|---------:|
|
||||||
|
| none | 45395 | 212.700 | 195.000 |
|
||||||
|
| <5 GHz | 7151 | 200.500 | 171.800 |
|
||||||
|
| 5-10 GHz | 444 | 200.600 | 174.200 |
|
||||||
|
| 10-24 GHz | 146 | 203.300 | 225.200 |
|
||||||
|
| 24-47 GHz | 5 | 177.200 | 133.300 |
|
||||||
|
| >=47 GHz | 23 | 147.500 | 131.500 |
|
||||||
|
|
||||||
|
|
||||||
|
**24 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:----------|---------:|---------:|---------:|
|
||||||
|
| none | 3188.000 | 95.000 | 92.600 |
|
||||||
|
| <5 GHz | 477.000 | 94.900 | 83.900 |
|
||||||
|
| 5-10 GHz | 36.000 | 97.500 | 113.700 |
|
||||||
|
| 10-24 GHz | 2.000 | 44.700 | 44.700 |
|
||||||
|
| >=47 GHz | 6.000 | 131.500 | 131.500 |
|
||||||
|
|
||||||
|
|
||||||
|
**47 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:---------|--------:|---------:|---------:|
|
||||||
|
| none | 552.000 | 64.300 | 65.400 |
|
||||||
|
| <5 GHz | 131.000 | 56.800 | 61.800 |
|
||||||
|
| 5-10 GHz | 6.000 | 98.100 | 113.700 |
|
||||||
|
|
||||||
|
|
||||||
|
**75 GHz:**
|
||||||
|
|
||||||
|
| bin | n | avg_km | p50_km |
|
||||||
|
|:-------|-------:|---------:|---------:|
|
||||||
|
| none | 76.000 | 54.800 | 36.700 |
|
||||||
|
| <5 GHz | 15.000 | 13.900 | 14.600 |
|
||||||
|
|
||||||
|
|
||||||
|
## Commercial-link rx_power degradation vs contemporaneous DFW contacts (matched n=0)
|
||||||
|
|
||||||
|
_No DFW-zone contacts fall inside the commercial_samples date window yet. Expected to stay empty until the Aug/Sep contest season produces contacts that overlap with live SNMP polling._
|
||||||
|
|
@ -7,8 +7,13 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
||||||
algorithm is tuned or new bands are added, only this module changes.
|
algorithm is tuned or new bands are added, only this module changes.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
# Recalibrated 2026-04-11 via gradient descent on 5000 QSOs (loss 0.42 → 0.12).
|
# Default weight vector. Originally fit by gradient descent on 5,000 QSOs
|
||||||
# Key shifts: rain +70%, season +39%, wind +60%; time_of_day -50%, pressure -31%.
|
# 2026-04-11 (loss 0.42 → 0.12) and retained as the 10 GHz prior. Bands
|
||||||
|
# with ≥200 HRRR-matched contacts in the 2026-04-18 full-corpus
|
||||||
|
# correlation analysis carry per-band `:weights` overrides in
|
||||||
|
# `@band_configs` — see `algo.md` Part 2d. Bands without enough data
|
||||||
|
# (50/144 MHz with 0 contacts, 47+ GHz with <200 matches) inherit this
|
||||||
|
# vector.
|
||||||
@weights %{
|
@weights %{
|
||||||
humidity: 0.1243,
|
humidity: 0.1243,
|
||||||
time_of_day: 0.0496,
|
time_of_day: 0.0496,
|
||||||
|
|
@ -122,6 +127,21 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
||||||
humidity_penalty: 0.0,
|
humidity_penalty: 0.0,
|
||||||
rain_k: 0.0,
|
rain_k: 0.0,
|
||||||
rain_alpha: 1.0,
|
rain_alpha: 1.0,
|
||||||
|
# Per-band weights from 2026-04-18 full-corpus correlation (n=5,392
|
||||||
|
# matched). Moisture (pwat, humidity) dominate; rain and time-of-day
|
||||||
|
# suppressed vs the 10 GHz baseline.
|
||||||
|
weights: %{
|
||||||
|
humidity: 0.1593,
|
||||||
|
time_of_day: 0.0350,
|
||||||
|
td_depression: 0.1250,
|
||||||
|
refractivity: 0.1401,
|
||||||
|
sky: 0.0706,
|
||||||
|
season: 0.1276,
|
||||||
|
wind: 0.0706,
|
||||||
|
rain: 0.0120,
|
||||||
|
pwat: 0.1916,
|
||||||
|
pressure: 0.0681
|
||||||
|
},
|
||||||
seasonal_base: %{
|
seasonal_base: %{
|
||||||
1 => 38,
|
1 => 38,
|
||||||
2 => 40,
|
2 => 40,
|
||||||
|
|
@ -150,6 +170,19 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
||||||
humidity_penalty: 0.0,
|
humidity_penalty: 0.0,
|
||||||
rain_k: 0.0,
|
rain_k: 0.0,
|
||||||
rain_alpha: 1.0,
|
rain_alpha: 1.0,
|
||||||
|
# Per-band weights from 2026-04-18 full-corpus correlation (n=6,583).
|
||||||
|
weights: %{
|
||||||
|
humidity: 0.2061,
|
||||||
|
time_of_day: 0.0329,
|
||||||
|
td_depression: 0.1200,
|
||||||
|
refractivity: 0.1186,
|
||||||
|
sky: 0.0663,
|
||||||
|
season: 0.1106,
|
||||||
|
wind: 0.0663,
|
||||||
|
rain: 0.0113,
|
||||||
|
pwat: 0.1870,
|
||||||
|
pressure: 0.0809
|
||||||
|
},
|
||||||
seasonal_base: %{
|
seasonal_base: %{
|
||||||
1 => 38,
|
1 => 38,
|
||||||
2 => 40,
|
2 => 40,
|
||||||
|
|
@ -178,6 +211,19 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
||||||
humidity_penalty: 0.0,
|
humidity_penalty: 0.0,
|
||||||
rain_k: 0.000,
|
rain_k: 0.000,
|
||||||
rain_alpha: 1.0,
|
rain_alpha: 1.0,
|
||||||
|
# Per-band weights from 2026-04-18 full-corpus correlation (n=1,317).
|
||||||
|
weights: %{
|
||||||
|
humidity: 0.2201,
|
||||||
|
time_of_day: 0.0437,
|
||||||
|
td_depression: 0.1102,
|
||||||
|
refractivity: 0.0888,
|
||||||
|
sky: 0.0783,
|
||||||
|
season: 0.1197,
|
||||||
|
wind: 0.0783,
|
||||||
|
rain: 0.0133,
|
||||||
|
pwat: 0.1635,
|
||||||
|
pressure: 0.0839
|
||||||
|
},
|
||||||
seasonal_base: %{
|
seasonal_base: %{
|
||||||
1 => 38,
|
1 => 38,
|
||||||
2 => 40,
|
2 => 40,
|
||||||
|
|
@ -206,6 +252,19 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
||||||
humidity_penalty: 0.0,
|
humidity_penalty: 0.0,
|
||||||
rain_k: 0.000,
|
rain_k: 0.000,
|
||||||
rain_alpha: 1.0,
|
rain_alpha: 1.0,
|
||||||
|
# Per-band weights from 2026-04-18 full-corpus correlation (n=2,146).
|
||||||
|
weights: %{
|
||||||
|
humidity: 0.2131,
|
||||||
|
time_of_day: 0.0494,
|
||||||
|
td_depression: 0.0898,
|
||||||
|
refractivity: 0.1392,
|
||||||
|
sky: 0.0797,
|
||||||
|
season: 0.1108,
|
||||||
|
wind: 0.0797,
|
||||||
|
rain: 0.0136,
|
||||||
|
pwat: 0.1678,
|
||||||
|
pressure: 0.0568
|
||||||
|
},
|
||||||
seasonal_base: %{
|
seasonal_base: %{
|
||||||
1 => 38,
|
1 => 38,
|
||||||
2 => 40,
|
2 => 40,
|
||||||
|
|
@ -234,6 +293,19 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
||||||
humidity_penalty: 0.0,
|
humidity_penalty: 0.0,
|
||||||
rain_k: 0.001,
|
rain_k: 0.001,
|
||||||
rain_alpha: 1.15,
|
rain_alpha: 1.15,
|
||||||
|
# Per-band weights from 2026-04-18 full-corpus correlation (n=564).
|
||||||
|
weights: %{
|
||||||
|
humidity: 0.1941,
|
||||||
|
time_of_day: 0.0387,
|
||||||
|
td_depression: 0.1385,
|
||||||
|
refractivity: 0.1638,
|
||||||
|
sky: 0.0625,
|
||||||
|
season: 0.0868,
|
||||||
|
wind: 0.0625,
|
||||||
|
rain: 0.0336,
|
||||||
|
pwat: 0.1762,
|
||||||
|
pressure: 0.0432
|
||||||
|
},
|
||||||
seasonal_base: %{
|
seasonal_base: %{
|
||||||
1 => 38,
|
1 => 38,
|
||||||
2 => 40,
|
2 => 40,
|
||||||
|
|
@ -262,6 +334,19 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
||||||
humidity_penalty: 0.0,
|
humidity_penalty: 0.0,
|
||||||
rain_k: 0.002,
|
rain_k: 0.002,
|
||||||
rain_alpha: 1.20,
|
rain_alpha: 1.20,
|
||||||
|
# Per-band weights from 2026-04-18 full-corpus correlation (n=280).
|
||||||
|
weights: %{
|
||||||
|
humidity: 0.1996,
|
||||||
|
time_of_day: 0.0398,
|
||||||
|
td_depression: 0.1251,
|
||||||
|
refractivity: 0.1310,
|
||||||
|
sky: 0.0642,
|
||||||
|
season: 0.0893,
|
||||||
|
wind: 0.0642,
|
||||||
|
rain: 0.0489,
|
||||||
|
pwat: 0.1811,
|
||||||
|
pressure: 0.0568
|
||||||
|
},
|
||||||
seasonal_base: %{
|
seasonal_base: %{
|
||||||
1 => 38,
|
1 => 38,
|
||||||
2 => 40,
|
2 => 40,
|
||||||
|
|
@ -290,6 +375,19 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
||||||
humidity_penalty: 0.0,
|
humidity_penalty: 0.0,
|
||||||
rain_k: 0.005,
|
rain_k: 0.005,
|
||||||
rain_alpha: 1.25,
|
rain_alpha: 1.25,
|
||||||
|
# Per-band weights from 2026-04-18 full-corpus correlation (n=246).
|
||||||
|
weights: %{
|
||||||
|
humidity: 0.1829,
|
||||||
|
time_of_day: 0.0423,
|
||||||
|
td_depression: 0.1205,
|
||||||
|
refractivity: 0.0881,
|
||||||
|
sky: 0.0683,
|
||||||
|
season: 0.0949,
|
||||||
|
wind: 0.0683,
|
||||||
|
rain: 0.0822,
|
||||||
|
pwat: 0.1925,
|
||||||
|
pressure: 0.0602
|
||||||
|
},
|
||||||
seasonal_base: %{
|
seasonal_base: %{
|
||||||
1 => 38,
|
1 => 38,
|
||||||
2 => 40,
|
2 => 40,
|
||||||
|
|
@ -346,6 +444,23 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
||||||
humidity_penalty: 1.6,
|
humidity_penalty: 1.6,
|
||||||
rain_k: 0.070,
|
rain_k: 0.070,
|
||||||
rain_alpha: 1.07,
|
rain_alpha: 1.07,
|
||||||
|
# Per-band weights from 2026-04-18 full-corpus correlation (n=613).
|
||||||
|
# Rain climbs sharply (sqrt-dampened rain_k ratio = √7 ≈ 2.6× vs 10 G),
|
||||||
|
# refractivity and pwat hold up, td_depression falls out because the
|
||||||
|
# temperature signal alone is near-zero — moisture already carries it
|
||||||
|
# via :harmful humidity.
|
||||||
|
weights: %{
|
||||||
|
humidity: 0.1481,
|
||||||
|
time_of_day: 0.0532,
|
||||||
|
td_depression: 0.0360,
|
||||||
|
refractivity: 0.1250,
|
||||||
|
sky: 0.0477,
|
||||||
|
season: 0.0729,
|
||||||
|
wind: 0.0477,
|
||||||
|
rain: 0.2147,
|
||||||
|
pwat: 0.1344,
|
||||||
|
pressure: 0.1203
|
||||||
|
},
|
||||||
seasonal_base: %{
|
seasonal_base: %{
|
||||||
1 => 88,
|
1 => 88,
|
||||||
2 => 84,
|
2 => 84,
|
||||||
|
|
@ -740,10 +855,23 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
||||||
Enum.map(all_bands(), fn band -> {band.label, to_string(band.freq_mhz)} end)
|
Enum.map(all_bands(), fn band -> {band.label, to_string(band.freq_mhz)} end)
|
||||||
end
|
end
|
||||||
|
|
||||||
@doc "Returns the scoring weights map. All 10 values sum to 1.0."
|
@doc "Returns the default scoring weights map. All 10 values sum to 1.0."
|
||||||
@spec weights() :: map()
|
@spec weights() :: map()
|
||||||
def weights, do: @weights
|
def weights, do: @weights
|
||||||
|
|
||||||
|
@doc """
|
||||||
|
Returns the scoring weights for a specific band configuration.
|
||||||
|
|
||||||
|
Bands with enough contacts to support a stable gradient-descent fit
|
||||||
|
carry a `:weights` override map. Bands without it fall back to the
|
||||||
|
global defaults. `nil` is accepted so callers that don't know the band
|
||||||
|
can ask for defaults without an extra guard.
|
||||||
|
"""
|
||||||
|
@spec weights(map() | nil) :: map()
|
||||||
|
def weights(nil), do: @weights
|
||||||
|
def weights(%{weights: override}) when is_map(override), do: override
|
||||||
|
def weights(%{}), do: @weights
|
||||||
|
|
||||||
@doc "Returns the 12-element sunrise hour table (Jan-Dec, local time)."
|
@doc "Returns the 12-element sunrise hour table (Jan-Dec, local time)."
|
||||||
@spec sunrise_table() :: [float()]
|
@spec sunrise_table() :: [float()]
|
||||||
def sunrise_table, do: @sunrise_table
|
def sunrise_table, do: @sunrise_table
|
||||||
|
|
|
||||||
|
|
@ -39,6 +39,9 @@ defmodule Microwaveprop.Propagation.Recalibrator do
|
||||||
* `:sample_size` - max contacts to load (default: 5000)
|
* `:sample_size` - max contacts to load (default: 5000)
|
||||||
* `:learning_rate` - gradient descent step size (default: 0.01)
|
* `:learning_rate` - gradient descent step size (default: 0.01)
|
||||||
* `:epochs` - number of training iterations (default: 2000)
|
* `:epochs` - number of training iterations (default: 2000)
|
||||||
|
* `:band_mhz` - restrict contacts + factor scoring to a specific band
|
||||||
|
(default: unrestricted; factor scoring uses 10 GHz config as it did
|
||||||
|
historically)
|
||||||
|
|
||||||
## Returns
|
## Returns
|
||||||
|
|
||||||
|
|
@ -49,15 +52,17 @@ defmodule Microwaveprop.Propagation.Recalibrator do
|
||||||
sample_size = Keyword.get(opts, :sample_size, 5000)
|
sample_size = Keyword.get(opts, :sample_size, 5000)
|
||||||
learning_rate = Keyword.get(opts, :learning_rate, 0.01)
|
learning_rate = Keyword.get(opts, :learning_rate, 0.01)
|
||||||
epochs = Keyword.get(opts, :epochs, 2000)
|
epochs = Keyword.get(opts, :epochs, 2000)
|
||||||
|
band_mhz = Keyword.get(opts, :band_mhz)
|
||||||
|
|
||||||
Logger.info("Recalibrator: loading contacts and computing factor vectors...")
|
band_label = if band_mhz, do: "#{band_mhz} MHz", else: "all bands"
|
||||||
|
Logger.info("Recalibrator: loading contacts for #{band_label}, computing factor vectors...")
|
||||||
|
|
||||||
# Load contacts with HRRR profiles
|
# Load contacts with HRRR profiles
|
||||||
{positives, contacts} = load_positive_samples(sample_size)
|
{positives, contacts} = load_positive_samples(sample_size, band_mhz)
|
||||||
Logger.info("Recalibrator: #{length(positives)} positive samples from #{length(contacts)} contacts")
|
Logger.info("Recalibrator: #{length(positives)} positive samples from #{length(contacts)} contacts")
|
||||||
|
|
||||||
# Generate matched negatives
|
# Generate matched negatives
|
||||||
negatives = generate_negative_samples(contacts, length(positives))
|
negatives = generate_negative_samples(contacts, length(positives), band_mhz)
|
||||||
Logger.info("Recalibrator: #{length(negatives)} negative samples")
|
Logger.info("Recalibrator: #{length(negatives)} negative samples")
|
||||||
|
|
||||||
if positives == [] or negatives == [] do
|
if positives == [] or negatives == [] do
|
||||||
|
|
@ -82,8 +87,19 @@ defmodule Microwaveprop.Propagation.Recalibrator do
|
||||||
Returns a list of 10 floats, each in [0, 100].
|
Returns a list of 10 floats, each in [0, 100].
|
||||||
"""
|
"""
|
||||||
@spec compute_factors(map(), DateTime.t()) :: [number()]
|
@spec compute_factors(map(), DateTime.t()) :: [number()]
|
||||||
def compute_factors(profile, timestamp) do
|
def compute_factors(profile, timestamp), do: compute_factors(profile, timestamp, 10_000)
|
||||||
band_config = BandConfig.get(10_000)
|
|
||||||
|
@doc """
|
||||||
|
Band-aware variant of `compute_factors/2`.
|
||||||
|
|
||||||
|
Factor scoring reaches into band-specific config (humidity direction,
|
||||||
|
seasonal table, rain coefficients, etc.) so positives and negatives for
|
||||||
|
per-band fits are scored through the exact physics each band uses at
|
||||||
|
runtime. `band_mhz` must be a supported band in `BandConfig`.
|
||||||
|
"""
|
||||||
|
@spec compute_factors(map(), DateTime.t(), pos_integer()) :: [number()]
|
||||||
|
def compute_factors(profile, timestamp, band_mhz) do
|
||||||
|
band_config = BandConfig.get(band_mhz) || BandConfig.get(10_000)
|
||||||
|
|
||||||
temp_f = Scorer.c_to_f(profile.surface_temp_c)
|
temp_f = Scorer.c_to_f(profile.surface_temp_c)
|
||||||
dewpoint_f = Scorer.c_to_f(profile.surface_dewpoint_c)
|
dewpoint_f = Scorer.c_to_f(profile.surface_dewpoint_c)
|
||||||
|
|
@ -140,43 +156,55 @@ defmodule Microwaveprop.Propagation.Recalibrator do
|
||||||
|
|
||||||
# ── Private ──────────────────────────────────────────────────────
|
# ── Private ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
defp load_positive_samples(sample_size) do
|
defp load_positive_samples(sample_size, band_mhz) do
|
||||||
contacts =
|
base_query =
|
||||||
Contact
|
Contact
|
||||||
|> where([c], not is_nil(c.pos1))
|
|> where([c], not is_nil(c.pos1))
|
||||||
|> where([c], c.distance_km < 3000)
|
|> where([c], c.distance_km < 3000)
|
||||||
|> order_by([c], desc: c.qso_timestamp)
|
|> order_by([c], desc: c.qso_timestamp)
|
||||||
|> limit(^sample_size)
|
|> limit(^sample_size)
|
||||||
|> Repo.all()
|
|
||||||
|
query =
|
||||||
|
if band_mhz do
|
||||||
|
from c in base_query, where: c.band == ^Decimal.new(band_mhz)
|
||||||
|
else
|
||||||
|
base_query
|
||||||
|
end
|
||||||
|
|
||||||
|
contacts = Repo.all(query)
|
||||||
|
|
||||||
|
factor_band = band_mhz || 10_000
|
||||||
|
|
||||||
factor_vectors =
|
factor_vectors =
|
||||||
contacts
|
contacts
|
||||||
|> Enum.map(&contact_to_factors/1)
|
|> Enum.map(&contact_to_factors(&1, factor_band))
|
||||||
|> Enum.reject(&is_nil/1)
|
|> Enum.reject(&is_nil/1)
|
||||||
|
|
||||||
{factor_vectors, contacts}
|
{factor_vectors, contacts}
|
||||||
end
|
end
|
||||||
|
|
||||||
defp contact_to_factors(%Contact{pos1: pos, qso_timestamp: ts}) do
|
defp contact_to_factors(%Contact{pos1: pos, qso_timestamp: ts}, band_mhz) do
|
||||||
lat = pos["lat"]
|
lat = pos["lat"]
|
||||||
lon = pos["lon"]
|
lon = pos["lon"]
|
||||||
|
|
||||||
if lat && lon do
|
if lat && lon do
|
||||||
case Weather.find_nearest_hrrr(lat, lon, ts) do
|
case Weather.find_nearest_hrrr(lat, lon, ts) do
|
||||||
nil -> nil
|
nil -> nil
|
||||||
profile -> compute_factors(profile, ts)
|
profile -> compute_factors(profile, ts, band_mhz)
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
defp generate_negative_samples(contacts, n) do
|
defp generate_negative_samples(contacts, n, band_mhz) do
|
||||||
baselines = Backtest.random_baseline(n, sample_size: length(contacts))
|
baselines = Backtest.random_baseline(n, sample_size: length(contacts))
|
||||||
|
|
||||||
|
factor_band = band_mhz || 10_000
|
||||||
|
|
||||||
baselines
|
baselines
|
||||||
|> Enum.map(fn {lat, lon, time} ->
|
|> Enum.map(fn {lat, lon, time} ->
|
||||||
case Weather.find_nearest_hrrr(lat, lon, time) do
|
case Weather.find_nearest_hrrr(lat, lon, time) do
|
||||||
nil -> nil
|
nil -> nil
|
||||||
profile -> compute_factors(profile, time)
|
profile -> compute_factors(profile, time, factor_band)
|
||||||
end
|
end
|
||||||
end)
|
end)
|
||||||
|> Enum.reject(&is_nil/1)
|
|> Enum.reject(&is_nil/1)
|
||||||
|
|
|
||||||
|
|
@ -487,7 +487,7 @@ defmodule Microwaveprop.Propagation.Scorer do
|
||||||
pressure: score_pressure(conditions.pressure_mb, conditions.prev_pressure_mb)
|
pressure: score_pressure(conditions.pressure_mb, conditions.prev_pressure_mb)
|
||||||
}
|
}
|
||||||
|
|
||||||
weights = BandConfig.weights()
|
weights = BandConfig.weights(band_config)
|
||||||
|
|
||||||
weighted_sum =
|
weighted_sum =
|
||||||
Enum.reduce(factors, 0.0, fn {factor, score}, acc ->
|
Enum.reduce(factors, 0.0, fn {factor, score}, acc ->
|
||||||
|
|
|
||||||
|
|
@ -1938,7 +1938,7 @@ defmodule MicrowavepropWeb.ContactLive.Show do
|
||||||
{nil, [], nil}
|
{nil, [], nil}
|
||||||
else
|
else
|
||||||
result = Scorer.composite_score(conditions, band_config)
|
result = Scorer.composite_score(conditions, band_config)
|
||||||
weights = BandConfig.weights()
|
weights = BandConfig.weights(band_config)
|
||||||
|
|
||||||
factor_rows =
|
factor_rows =
|
||||||
[
|
[
|
||||||
|
|
|
||||||
281
scripts/derive_band_weights.py
Normal file
281
scripts/derive_band_weights.py
Normal file
|
|
@ -0,0 +1,281 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Per-band weight derivation from the full-corpus correlation analysis.
|
||||||
|
|
||||||
|
Reads Pearson correlations that `recalibrate_algo.py` wrote into the DB
|
||||||
|
and emits a `@band_weight_overrides` Elixir map ready to drop into
|
||||||
|
`lib/microwaveprop/propagation/band_config.ex`.
|
||||||
|
|
||||||
|
The derivation rule is deliberately simple and legible, because the
|
||||||
|
correlations are noisy (|r| typically 0.05–0.25) and we'd rather preserve
|
||||||
|
the globally-fit default weights as a prior than over-fit per-band.
|
||||||
|
|
||||||
|
Rule:
|
||||||
|
1. Start from the default weight vector (gradient-descent fit, global).
|
||||||
|
2. Compute a per-factor "relative signal strength" ratio
|
||||||
|
s_band,factor = |r_band,factor| / |r_10GHz,factor|
|
||||||
|
with a small ε floor so zero-correlation factors don't collapse.
|
||||||
|
3. Clamp the ratio to [0.4, 2.5] — noisy per-band correlations shouldn't
|
||||||
|
move a factor more than 2.5× or less than 0.4× the prior.
|
||||||
|
4. Factors we don't have direct correlations for (sky, wind, rain,
|
||||||
|
season, time_of_day) carry a physics-informed multiplier per band:
|
||||||
|
* rain scales with `rain_k` (ITU-R P.838 specific attenuation
|
||||||
|
coefficient) normalized to 10 GHz's rain_k.
|
||||||
|
* season scales up at VHF/low-UHF where Es-like physics amplifies
|
||||||
|
monthly variation (we don't model Es directly).
|
||||||
|
* time_of_day scales with frequency (Finding 8 in algo.md).
|
||||||
|
* sky/wind stay flat — no per-band evidence either direction.
|
||||||
|
5. Multiply default weights by the multipliers, then normalize to 1.0.
|
||||||
|
|
||||||
|
The output goes to stdout as Elixir source ready to paste into BandConfig.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
try:
|
||||||
|
import psycopg
|
||||||
|
from psycopg.rows import dict_row
|
||||||
|
except ImportError:
|
||||||
|
sys.exit("psycopg is required: pip install 'psycopg[binary]>=3.1'")
|
||||||
|
|
||||||
|
|
||||||
|
# Default weights (must match `@weights` in band_config.ex). These are the
|
||||||
|
# globally-fit gradient-descent priors — the per-band deltas modulate them.
|
||||||
|
DEFAULT_WEIGHTS = {
|
||||||
|
"rain": 0.1362,
|
||||||
|
"humidity": 0.1243,
|
||||||
|
"pwat": 0.1128,
|
||||||
|
"season": 0.1112,
|
||||||
|
"refractivity": 0.1049,
|
||||||
|
"pressure": 0.1032,
|
||||||
|
"td_depression": 0.0978,
|
||||||
|
"sky": 0.08,
|
||||||
|
"wind": 0.08,
|
||||||
|
"time_of_day": 0.0496,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Correlation-fed factor → source-field mapping. These factors get data-
|
||||||
|
# driven per-band multipliers; the rest get physics-informed multipliers.
|
||||||
|
FACTOR_SOURCE = {
|
||||||
|
"humidity": "dpc", # dewpoint carries the moisture signal
|
||||||
|
"td_depression": "tc", # temperature dominates T-Td variance
|
||||||
|
"refractivity": "grad", # refractivity gradient
|
||||||
|
"pressure": "pr",
|
||||||
|
"pwat": "pwat",
|
||||||
|
}
|
||||||
|
|
||||||
|
# ITU-R P.838-3 rain_k values (per BandConfig). Used for `rain` weight
|
||||||
|
# scaling — rain attenuation grows by ~4 orders of magnitude from VHF
|
||||||
|
# to sub-mm, so `rain` weight should track that.
|
||||||
|
BAND_RAIN_K = {
|
||||||
|
50: 0.0, 144: 0.0, 222: 0.0, 432: 0.0,
|
||||||
|
902: 0.0, 1_296: 0.0,
|
||||||
|
2_304: 0.001, 3_400: 0.002, 5_760: 0.005,
|
||||||
|
10_000: 0.010, 24_000: 0.070, 47_000: 0.187,
|
||||||
|
68_000: 0.310, 75_000: 0.345, 122_000: 0.498,
|
||||||
|
134_000: 0.520, 142_000: 0.530, 145_000: 0.535,
|
||||||
|
241_000: 0.550, 288_000: 0.560, 322_000: 0.570,
|
||||||
|
403_000: 0.580, 411_000: 0.580,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Season weight multiplier by band. VHF has large seasonal swings driven
|
||||||
|
# by tropo-mixing physics (summer peak). Microwave seasonal swings are
|
||||||
|
# contest-schedule artefacts but the physics (summer convective mixing
|
||||||
|
# hurting 24+ GHz) still applies. No per-band evidence in the correlation
|
||||||
|
# report so we use physics priors.
|
||||||
|
BAND_SEASON_MULT = {
|
||||||
|
50: 1.6, 144: 1.4, 222: 1.3, 432: 1.2,
|
||||||
|
902: 1.1, 1_296: 1.0,
|
||||||
|
2_304: 1.0, 3_400: 1.0, 5_760: 1.0,
|
||||||
|
10_000: 1.0, 24_000: 1.1, 47_000: 1.2,
|
||||||
|
68_000: 1.3, 75_000: 1.4, 122_000: 1.5, 134_000: 1.5,
|
||||||
|
142_000: 1.5, 145_000: 1.5, 241_000: 1.6, 288_000: 1.6,
|
||||||
|
322_000: 1.7, 403_000: 1.7, 411_000: 1.7,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Time-of-day multiplier by band (Finding 8 in algo.md — effect scales
|
||||||
|
# with frequency). 10 GHz is the baseline.
|
||||||
|
BAND_TOD_MULT = {
|
||||||
|
50: 0.6, 144: 0.7, 222: 0.8, 432: 0.8,
|
||||||
|
902: 0.9, 1_296: 1.0,
|
||||||
|
2_304: 1.0, 3_400: 1.0, 5_760: 1.0,
|
||||||
|
10_000: 1.0, 24_000: 1.8, 47_000: 2.5,
|
||||||
|
68_000: 3.5, 75_000: 4.0,
|
||||||
|
122_000: 5.0, 134_000: 5.0, 142_000: 5.0, 145_000: 5.0,
|
||||||
|
241_000: 5.0, 288_000: 5.0, 322_000: 5.0, 403_000: 5.0,
|
||||||
|
411_000: 5.0,
|
||||||
|
}
|
||||||
|
|
||||||
|
# EPS sets an absolute floor on |r| so two near-zero correlations don't
|
||||||
|
# produce a giant ratio. Set at the noise floor we can reliably distinguish
|
||||||
|
# from zero given typical n. Tightened along with the clamp so small
|
||||||
|
# corpora don't drag the weights around.
|
||||||
|
EPS = 0.05
|
||||||
|
CLAMP_LO = 0.5
|
||||||
|
CLAMP_HI = 2.0
|
||||||
|
MIN_N_FOR_FIT = 200 # bands with fewer matched contacts use physics only
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class BandCorr:
|
||||||
|
band_mhz: int
|
||||||
|
n: int
|
||||||
|
rho: dict[str, float]
|
||||||
|
|
||||||
|
|
||||||
|
PER_BAND_SQL = """
|
||||||
|
WITH joined AS (
|
||||||
|
SELECT DISTINCT ON (c.id)
|
||||||
|
c.id, c.band::int AS band, c.distance_km::float AS dist,
|
||||||
|
h.surface_temp_c AS tc, h.surface_dewpoint_c AS dpc,
|
||||||
|
h.surface_pressure_mb AS pr, h.pwat_mm AS pwat,
|
||||||
|
h.min_refractivity_gradient AS grad
|
||||||
|
FROM contacts c
|
||||||
|
JOIN hrrr_profiles h
|
||||||
|
ON h.lat BETWEEN (c.pos1->>'lat')::float - 0.07
|
||||||
|
AND (c.pos1->>'lat')::float + 0.07
|
||||||
|
AND h.lon BETWEEN (c.pos1->>'lon')::float - 0.07
|
||||||
|
AND (c.pos1->>'lon')::float + 0.07
|
||||||
|
AND h.valid_time BETWEEN c.qso_timestamp - INTERVAL '1 hour'
|
||||||
|
AND c.qso_timestamp + INTERVAL '1 hour'
|
||||||
|
WHERE c.pos1 IS NOT NULL AND c.distance_km < 3000
|
||||||
|
AND c.flagged_invalid = false
|
||||||
|
ORDER BY c.id, ABS(EXTRACT(EPOCH FROM h.valid_time - c.qso_timestamp))
|
||||||
|
)
|
||||||
|
SELECT band,
|
||||||
|
count(*) AS n,
|
||||||
|
CORR(dist, tc) AS rho_tc,
|
||||||
|
CORR(dist, dpc) AS rho_dpc,
|
||||||
|
CORR(dist, pr) AS rho_pr,
|
||||||
|
CORR(dist, pwat) AS rho_pwat,
|
||||||
|
CORR(dist, grad) AS rho_grad
|
||||||
|
FROM joined
|
||||||
|
WHERE band >= 50
|
||||||
|
GROUP BY band
|
||||||
|
HAVING count(*) >= 50
|
||||||
|
ORDER BY band;
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def load_correlations(dsn: str) -> dict[int, BandCorr]:
|
||||||
|
with psycopg.connect(dsn) as conn:
|
||||||
|
with conn.cursor(row_factory=dict_row) as cur:
|
||||||
|
cur.execute("SET statement_timeout = '30min'")
|
||||||
|
cur.execute(PER_BAND_SQL)
|
||||||
|
rows = cur.fetchall()
|
||||||
|
out = {}
|
||||||
|
for r in rows:
|
||||||
|
out[int(r["band"])] = BandCorr(
|
||||||
|
band_mhz=int(r["band"]),
|
||||||
|
n=int(r["n"]),
|
||||||
|
rho={
|
||||||
|
"tc": r["rho_tc"] or 0.0,
|
||||||
|
"dpc": r["rho_dpc"] or 0.0,
|
||||||
|
"pr": r["rho_pr"] or 0.0,
|
||||||
|
"pwat": r["rho_pwat"] or 0.0,
|
||||||
|
"grad": r["rho_grad"] or 0.0,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def derive_weights(
|
||||||
|
band_mhz: int, corrs: dict[int, BandCorr]
|
||||||
|
) -> dict[str, float] | None:
|
||||||
|
"""Returns a normalized weight map or None if the band should use defaults."""
|
||||||
|
band_corr = corrs.get(band_mhz)
|
||||||
|
ref_corr = corrs.get(10_000)
|
||||||
|
|
||||||
|
if band_corr is None or band_corr.n < MIN_N_FOR_FIT or ref_corr is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
multipliers = {}
|
||||||
|
|
||||||
|
# Data-driven multipliers for correlation-backed factors.
|
||||||
|
for factor, field in FACTOR_SOURCE.items():
|
||||||
|
band_r = abs(band_corr.rho[field]) + EPS
|
||||||
|
ref_r = abs(ref_corr.rho[field]) + EPS
|
||||||
|
ratio = band_r / ref_r
|
||||||
|
multipliers[factor] = max(CLAMP_LO, min(CLAMP_HI, ratio))
|
||||||
|
|
||||||
|
# Physics-driven multipliers for the rest.
|
||||||
|
# Rain: weight tracks rain_k but with sqrt-dampening. A 7× rain_k jump
|
||||||
|
# (10→24 GHz) should not become a 7× weight jump — the score itself
|
||||||
|
# already scales with rain_k via ITU-R P.838 in `score_rain`, so the
|
||||||
|
# weight only needs to capture how often rain drives the outcome.
|
||||||
|
rain_k = BAND_RAIN_K.get(band_mhz, 0.010)
|
||||||
|
rain_k_10g = BAND_RAIN_K[10_000]
|
||||||
|
if rain_k > 0:
|
||||||
|
ratio = rain_k / rain_k_10g
|
||||||
|
rain_mult = max(0.2, min(3.0, ratio**0.5))
|
||||||
|
else:
|
||||||
|
rain_mult = 0.1
|
||||||
|
multipliers["rain"] = rain_mult
|
||||||
|
|
||||||
|
multipliers["season"] = BAND_SEASON_MULT.get(band_mhz, 1.0)
|
||||||
|
multipliers["time_of_day"] = BAND_TOD_MULT.get(band_mhz, 1.0)
|
||||||
|
multipliers["sky"] = 1.0
|
||||||
|
multipliers["wind"] = 1.0
|
||||||
|
|
||||||
|
# Apply and normalize.
|
||||||
|
raw = {k: DEFAULT_WEIGHTS[k] * multipliers[k] for k in DEFAULT_WEIGHTS}
|
||||||
|
total = sum(raw.values())
|
||||||
|
return {k: round(v / total, 4) for k, v in raw.items()}
|
||||||
|
|
||||||
|
|
||||||
|
def format_elixir(band_mhz: int, weights: dict[str, float]) -> str:
|
||||||
|
# Preserve the factor ordering used in band_config.ex comments.
|
||||||
|
order = [
|
||||||
|
"humidity", "time_of_day", "td_depression", "refractivity",
|
||||||
|
"sky", "season", "wind", "rain", "pwat", "pressure",
|
||||||
|
]
|
||||||
|
lines = [f" {k}: {weights[k]:.4f}" for k in order]
|
||||||
|
inner = ",\n".join(lines)
|
||||||
|
return f" weights: %{{\n{inner}\n }}"
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--dsn", default=os.environ.get("PROP_DB_URL"))
|
||||||
|
args = parser.parse_args()
|
||||||
|
if not args.dsn:
|
||||||
|
print("error: PROP_DB_URL not set and --dsn not given", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
print(f"loading correlations from {args.dsn}", file=sys.stderr)
|
||||||
|
corrs = load_correlations(args.dsn)
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"correlations available for {len(corrs)} bands: "
|
||||||
|
f"{sorted(corrs.keys())}",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Every band in BandConfig — we emit an entry for each so the reader
|
||||||
|
# sees at a glance which bands got fit and which inherited defaults.
|
||||||
|
all_bands = sorted(set(BAND_RAIN_K.keys()) | set(corrs.keys()))
|
||||||
|
|
||||||
|
sections = []
|
||||||
|
for band in all_bands:
|
||||||
|
weights = derive_weights(band, corrs)
|
||||||
|
band_corr = corrs.get(band)
|
||||||
|
n = band_corr.n if band_corr else 0
|
||||||
|
if weights is None:
|
||||||
|
sections.append(
|
||||||
|
f"# {band} MHz — n={n}, uses default @weights (insufficient signal)"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
sections.append(
|
||||||
|
f"# {band} MHz — n={n}, derived from corpus correlations\n"
|
||||||
|
f"{format_elixir(band, weights)}"
|
||||||
|
)
|
||||||
|
print("\n\n".join(sections))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
@ -4,14 +4,19 @@ Algo recalibration analysis runner.
|
||||||
|
|
||||||
Connects to the prod (or any) Postgres, joins contacts to the nearest HRRR
|
Connects to the prod (or any) Postgres, joins contacts to the nearest HRRR
|
||||||
grid profile within ±0.07° / ±1h (matching `Weather.find_nearest_hrrr/3`),
|
grid profile within ±0.07° / ±1h (matching `Weather.find_nearest_hrrr/3`),
|
||||||
and produces a Markdown report covering:
|
falls back to `narr_profiles` (NCEP NARR, 32 km / 3-hourly) for pre-2014
|
||||||
|
contacts where HRRR doesn't reach, and produces a Markdown report covering:
|
||||||
|
|
||||||
* row counts and date ranges for the key tables
|
* row counts and date ranges for the key tables (HRRR + NARR + ...)
|
||||||
* per-band contact distribution including bands that BandConfig hasn't seen
|
* per-band contact distribution from 50 MHz up (VHF/UHF/microwave)
|
||||||
* monthly sounding ducting refresh
|
* monthly sounding ducting refresh
|
||||||
* native HRRR profile duct stats
|
* native HRRR profile duct stats
|
||||||
* per-band Pearson correlations of contact distance vs HRRR fields
|
* per-band Pearson correlations of contact distance vs HRRR fields
|
||||||
* binned distance distributions for HPBL and pressure at 10 GHz
|
* per-band Pearson correlations vs NARR fields (historical sanity check)
|
||||||
|
* per-band binned distance distributions for HPBL, pressure, NEXRAD,
|
||||||
|
and native-profile best duct band — every band with ≥50 matched
|
||||||
|
contacts gets its own bin table so we're not hard-coding the model
|
||||||
|
to 10/24 GHz behaviour
|
||||||
* NEXRAD composite reflectivity vs contact distance (rain-scoring check)
|
* NEXRAD composite reflectivity vs contact distance (rain-scoring check)
|
||||||
* hrrr_native_profiles.best_duct_band_ghz as a distance discriminator
|
* hrrr_native_profiles.best_duct_band_ghz as a distance discriminator
|
||||||
* commercial-link rx_power degradation vs contemporaneous DFW contacts
|
* commercial-link rx_power degradation vs contemporaneous DFW contacts
|
||||||
|
|
@ -35,7 +40,6 @@ import argparse
|
||||||
import datetime as dt
|
import datetime as dt
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
import textwrap
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
try:
|
try:
|
||||||
|
|
@ -66,31 +70,22 @@ Statement timeout: {stmt_timeout}
|
||||||
|
|
||||||
# ─── SQL fragments ──────────────────────────────────────────────────────────
|
# ─── SQL fragments ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
ROW_COUNTS_SQL = """
|
# (table, time_column). `time_column = None` means just count rows without
|
||||||
SELECT 'contacts' tbl, count(*)::bigint n,
|
# min/max range. Tables not present in the target DB are silently skipped —
|
||||||
min(qso_timestamp)::date lo, max(qso_timestamp)::date hi FROM contacts
|
# local dev often trails prod (e.g. `propagation_scores` moved to binary files).
|
||||||
UNION ALL SELECT 'surface_observations', count(*),
|
ROW_COUNT_SOURCES = [
|
||||||
min(observed_at)::date, max(observed_at)::date FROM surface_observations
|
("contacts", "qso_timestamp"),
|
||||||
UNION ALL SELECT 'soundings', count(*),
|
("surface_observations", "observed_at"),
|
||||||
min(observed_at)::date, max(observed_at)::date FROM soundings
|
("soundings", "observed_at"),
|
||||||
UNION ALL SELECT 'hrrr_profiles', count(*),
|
("hrrr_profiles", "valid_time"),
|
||||||
min(valid_time)::date, max(valid_time)::date FROM hrrr_profiles
|
("hrrr_native_profiles", "valid_time"),
|
||||||
UNION ALL SELECT 'hrrr_native_profiles', count(*),
|
("narr_profiles", "valid_time"),
|
||||||
min(valid_time)::date, max(valid_time)::date FROM hrrr_native_profiles
|
("iemre_observations", None),
|
||||||
UNION ALL SELECT 'era5_profiles', count(*),
|
("nexrad_observations", "observed_at"),
|
||||||
min(valid_time)::date, max(valid_time)::date FROM era5_profiles
|
("rtma_observations", "valid_time"),
|
||||||
UNION ALL SELECT 'iemre_observations', count(*), NULL::date, NULL::date
|
("terrain_profiles", None),
|
||||||
FROM iemre_observations
|
("propagation_scores", "valid_time"),
|
||||||
UNION ALL SELECT 'nexrad_observations', count(*),
|
]
|
||||||
min(observed_at)::date, max(observed_at)::date FROM nexrad_observations
|
|
||||||
UNION ALL SELECT 'rtma_observations', count(*),
|
|
||||||
min(valid_time)::date, max(valid_time)::date FROM rtma_observations
|
|
||||||
UNION ALL SELECT 'terrain_profiles', count(*), NULL::date, NULL::date
|
|
||||||
FROM terrain_profiles
|
|
||||||
UNION ALL SELECT 'propagation_scores', count(*),
|
|
||||||
min(valid_time)::date, max(valid_time)::date FROM propagation_scores
|
|
||||||
ORDER BY tbl;
|
|
||||||
"""
|
|
||||||
|
|
||||||
CONTACTS_BY_BAND_SQL = """
|
CONTACTS_BY_BAND_SQL = """
|
||||||
SELECT band::int AS band_mhz, count(*) AS contacts,
|
SELECT band::int AS band_mhz, count(*) AS contacts,
|
||||||
|
|
@ -98,7 +93,7 @@ SELECT band::int AS band_mhz, count(*) AS contacts,
|
||||||
MAX(distance_km::numeric) AS max_km
|
MAX(distance_km::numeric) AS max_km
|
||||||
FROM contacts
|
FROM contacts
|
||||||
WHERE pos1 IS NOT NULL AND distance_km IS NOT NULL
|
WHERE pos1 IS NOT NULL AND distance_km IS NOT NULL
|
||||||
AND distance_km < 3000 AND flagged_invalid = false AND band >= 902
|
AND distance_km < 3000 AND flagged_invalid = false AND band >= 50
|
||||||
GROUP BY 1 ORDER BY 1;
|
GROUP BY 1 ORDER BY 1;
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
@ -128,7 +123,7 @@ SELECT
|
||||||
FROM hrrr_native_profiles GROUP BY 1 ORDER BY 1;
|
FROM hrrr_native_profiles GROUP BY 1 ORDER BY 1;
|
||||||
"""
|
"""
|
||||||
|
|
||||||
# Per-band contact ↔ HRRR Pearson correlations.
|
# Per-band contact ↔ HRRR Pearson correlations (2014-10 onward).
|
||||||
PER_BAND_JOIN_SQL = """
|
PER_BAND_JOIN_SQL = """
|
||||||
WITH joined AS (
|
WITH joined AS (
|
||||||
SELECT DISTINCT ON (c.id)
|
SELECT DISTINCT ON (c.id)
|
||||||
|
|
@ -151,6 +146,34 @@ WITH joined AS (
|
||||||
SELECT * FROM joined;
|
SELECT * FROM joined;
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
# Per-band contact ↔ NARR Pearson correlations (pre-2014-10).
|
||||||
|
# NARR is 32 km Lambert conformal / 3-hourly, so both tolerances are wider
|
||||||
|
# than the HRRR join above. Matches `NarrClient.in_coverage?/1` — valid up
|
||||||
|
# through but excluding 2014-10-02 (HRRR takes over after that).
|
||||||
|
PER_BAND_NARR_JOIN_SQL = """
|
||||||
|
WITH joined AS (
|
||||||
|
SELECT DISTINCT ON (c.id)
|
||||||
|
c.id, c.band::int AS band, c.distance_km::float AS dist,
|
||||||
|
n.surface_temp_c AS tc, n.surface_dewpoint_c AS dpc,
|
||||||
|
n.surface_pressure_mb AS pr, n.pwat_mm AS pwat,
|
||||||
|
n.min_refractivity_gradient AS grad,
|
||||||
|
n.surface_refractivity AS sref, n.hpbl_m AS hpbl
|
||||||
|
FROM contacts c
|
||||||
|
JOIN narr_profiles n
|
||||||
|
ON n.lat BETWEEN (c.pos1->>'lat')::float - 0.25
|
||||||
|
AND (c.pos1->>'lat')::float + 0.25
|
||||||
|
AND n.lon BETWEEN (c.pos1->>'lon')::float - 0.25
|
||||||
|
AND (c.pos1->>'lon')::float + 0.25
|
||||||
|
AND n.valid_time BETWEEN c.qso_timestamp - INTERVAL '2 hours'
|
||||||
|
AND c.qso_timestamp + INTERVAL '2 hours'
|
||||||
|
WHERE c.pos1 IS NOT NULL AND c.distance_km < 3000
|
||||||
|
AND c.flagged_invalid = false
|
||||||
|
AND c.qso_timestamp < TIMESTAMP '2014-10-02'
|
||||||
|
ORDER BY c.id, ABS(EXTRACT(EPOCH FROM n.valid_time - c.qso_timestamp))
|
||||||
|
)
|
||||||
|
SELECT * FROM joined;
|
||||||
|
"""
|
||||||
|
|
||||||
# NEXRAD ↔ contact join. Very tight spatial tolerance because nexrad_observations
|
# NEXRAD ↔ contact join. Very tight spatial tolerance because nexrad_observations
|
||||||
# is a per-contact enrichment table, not a grid sample.
|
# is a per-contact enrichment table, not a grid sample.
|
||||||
NEXRAD_JOIN_SQL = """
|
NEXRAD_JOIN_SQL = """
|
||||||
|
|
@ -250,12 +273,17 @@ WHERE baseline_rx IS NOT NULL;
|
||||||
# Empty-table / data-quality smoke checks.
|
# Empty-table / data-quality smoke checks.
|
||||||
DATA_GAP_SQL = """
|
DATA_GAP_SQL = """
|
||||||
SELECT
|
SELECT
|
||||||
(SELECT count(*) FROM era5_profiles) AS era5_rows,
|
(SELECT count(*) FROM narr_profiles) AS narr_rows,
|
||||||
|
(SELECT count(*) FROM narr_profiles WHERE valid_time < '2014-10-02') AS narr_pre2014_rows,
|
||||||
(SELECT count(*) FROM hrrr_climatology) AS hrrr_climatology_rows,
|
(SELECT count(*) FROM hrrr_climatology) AS hrrr_climatology_rows,
|
||||||
(SELECT count(*) FROM rtma_observations) AS rtma_rows,
|
(SELECT count(*) FROM rtma_observations) AS rtma_rows,
|
||||||
(SELECT count(*) FROM metar_5min_observations) AS metar5_rows,
|
(SELECT count(*) FROM metar_5min_observations) AS metar5_rows,
|
||||||
(SELECT count(*) FROM contacts WHERE hrrr_status = 'complete') AS hrrr_complete_contacts,
|
(SELECT count(*) FROM contacts WHERE hrrr_status = 'complete') AS hrrr_complete_contacts,
|
||||||
(SELECT count(*) FROM contacts WHERE hrrr_status != 'complete') AS hrrr_pending_contacts;
|
(SELECT count(*) FROM contacts WHERE hrrr_status != 'complete') AS hrrr_pending_contacts,
|
||||||
|
(SELECT count(*) FROM contacts
|
||||||
|
WHERE qso_timestamp < '2014-10-02'
|
||||||
|
AND pos1 IS NOT NULL AND distance_km < 3000
|
||||||
|
AND flagged_invalid = false) AS pre2014_contacts;
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -275,6 +303,50 @@ def md_table(df: pd.DataFrame) -> str:
|
||||||
return df.to_markdown(index=False, floatfmt=".3f") + "\n"
|
return df.to_markdown(index=False, floatfmt=".3f") + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_row_counts(conn) -> pd.DataFrame:
|
||||||
|
"""Per-table counts + date ranges, tolerant of missing tables."""
|
||||||
|
rows = []
|
||||||
|
for tbl, time_col in ROW_COUNT_SOURCES:
|
||||||
|
with conn.cursor(row_factory=dict_row) as cur:
|
||||||
|
cur.execute("SELECT to_regclass(%s) AS oid", (f"public.{tbl}",))
|
||||||
|
present = cur.fetchone()["oid"] is not None
|
||||||
|
if not present:
|
||||||
|
rows.append({"tbl": tbl, "n": None, "lo": None, "hi": None})
|
||||||
|
continue
|
||||||
|
if time_col:
|
||||||
|
sql = (
|
||||||
|
f"SELECT '{tbl}' tbl, count(*)::bigint n, "
|
||||||
|
f"min({time_col})::date lo, max({time_col})::date hi FROM {tbl}"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
sql = (
|
||||||
|
f"SELECT '{tbl}' tbl, count(*)::bigint n, "
|
||||||
|
f"NULL::date lo, NULL::date hi FROM {tbl}"
|
||||||
|
)
|
||||||
|
with conn.cursor(row_factory=dict_row) as cur:
|
||||||
|
cur.execute(sql)
|
||||||
|
rows.append(cur.fetchone())
|
||||||
|
return pd.DataFrame(rows).sort_values("tbl")
|
||||||
|
|
||||||
|
|
||||||
|
def band_label(band_mhz: int) -> str:
|
||||||
|
"""Human-readable band label, matching `BandConfig` conventions."""
|
||||||
|
if band_mhz < 1_000:
|
||||||
|
return f"{band_mhz} MHz"
|
||||||
|
ghz = band_mhz / 1_000
|
||||||
|
if ghz == int(ghz):
|
||||||
|
return f"{int(ghz)} GHz"
|
||||||
|
return f"{ghz:g} GHz"
|
||||||
|
|
||||||
|
|
||||||
|
def bands_with_samples(df: pd.DataFrame, min_samples: int = 50) -> list[int]:
|
||||||
|
"""Every band in `df` with at least `min_samples` rows, sorted ascending."""
|
||||||
|
if df.empty or "band" not in df.columns:
|
||||||
|
return []
|
||||||
|
counts = df.groupby("band").size()
|
||||||
|
return sorted(int(b) for b, n in counts.items() if n >= min_samples)
|
||||||
|
|
||||||
|
|
||||||
def correlations_per_band(df: pd.DataFrame, min_samples: int = 30) -> pd.DataFrame:
|
def correlations_per_band(df: pd.DataFrame, min_samples: int = 30) -> pd.DataFrame:
|
||||||
fields = ["pr", "dpc", "pwat", "sref", "grad", "tc", "hpbl"]
|
fields = ["pr", "dpc", "pwat", "sref", "grad", "tc", "hpbl"]
|
||||||
rows = []
|
rows = []
|
||||||
|
|
@ -485,7 +557,7 @@ def main() -> int:
|
||||||
|
|
||||||
print("• row counts", file=sys.stderr)
|
print("• row counts", file=sys.stderr)
|
||||||
sections.append("\n## Row counts and date ranges\n")
|
sections.append("\n## Row counts and date ranges\n")
|
||||||
sections.append(md_table(fetch_df(conn, ROW_COUNTS_SQL)))
|
sections.append(md_table(fetch_row_counts(conn)))
|
||||||
|
|
||||||
print("• data gaps", file=sys.stderr)
|
print("• data gaps", file=sys.stderr)
|
||||||
gaps = fetch_df(conn, DATA_GAP_SQL).iloc[0].to_dict()
|
gaps = fetch_df(conn, DATA_GAP_SQL).iloc[0].to_dict()
|
||||||
|
|
@ -495,7 +567,7 @@ def main() -> int:
|
||||||
)
|
)
|
||||||
|
|
||||||
print("• contacts by band", file=sys.stderr)
|
print("• contacts by band", file=sys.stderr)
|
||||||
sections.append("\n## Contacts by band (≥902 MHz)\n")
|
sections.append("\n## Contacts by band (≥50 MHz)\n")
|
||||||
sections.append(md_table(fetch_df(conn, CONTACTS_BY_BAND_SQL)))
|
sections.append(md_table(fetch_df(conn, CONTACTS_BY_BAND_SQL)))
|
||||||
|
|
||||||
print("• sounding monthly", file=sys.stderr)
|
print("• sounding monthly", file=sys.stderr)
|
||||||
|
|
@ -508,9 +580,12 @@ def main() -> int:
|
||||||
)
|
)
|
||||||
sections.append(md_table(fetch_df(conn, NATIVE_DUCT_SQL)))
|
sections.append(md_table(fetch_df(conn, NATIVE_DUCT_SQL)))
|
||||||
|
|
||||||
print("• per-band correlations (this is the slow one)", file=sys.stderr)
|
print("• per-band correlations — HRRR (this is the slow one)", file=sys.stderr)
|
||||||
joined = fetch_df(conn, PER_BAND_JOIN_SQL)
|
joined = fetch_df(conn, PER_BAND_JOIN_SQL)
|
||||||
|
|
||||||
|
print("• per-band correlations — NARR (pre-2014)", file=sys.stderr)
|
||||||
|
joined_narr = fetch_df(conn, PER_BAND_NARR_JOIN_SQL)
|
||||||
|
|
||||||
print("• NEXRAD ↔ contacts", file=sys.stderr)
|
print("• NEXRAD ↔ contacts", file=sys.stderr)
|
||||||
nexrad = fetch_df(conn, NEXRAD_JOIN_SQL)
|
nexrad = fetch_df(conn, NEXRAD_JOIN_SQL)
|
||||||
|
|
||||||
|
|
@ -529,11 +604,47 @@ def main() -> int:
|
||||||
)
|
)
|
||||||
sections.append(md_table(correlations_per_band(joined)))
|
sections.append(md_table(correlations_per_band(joined)))
|
||||||
|
|
||||||
sections.append("\n## 10 GHz: HPBL bin distance distribution\n")
|
sections.append(
|
||||||
sections.append(md_table(hpbl_bins(joined)))
|
"\n## Per-band HPBL bin distance distribution\n\n"
|
||||||
|
"One table per band with ≥50 matched contacts. A band-specific "
|
||||||
|
"effect here means HPBL belongs in the scorer for that band.\n"
|
||||||
|
)
|
||||||
|
for band in bands_with_samples(joined):
|
||||||
|
table = hpbl_bins(joined, band)
|
||||||
|
if table.empty:
|
||||||
|
continue
|
||||||
|
sections.append(f"\n**{band_label(band)}:**\n")
|
||||||
|
sections.append(md_table(table))
|
||||||
|
|
||||||
sections.append("\n## 10 GHz: pressure bin distance distribution\n")
|
sections.append(
|
||||||
sections.append(md_table(pressure_bins(joined)))
|
"\n## Per-band pressure bin distance distribution\n\n"
|
||||||
|
"Surface pressure tends to proxy synoptic-scale stagnation, so we "
|
||||||
|
"want to see longer distances under the 1015-1025 mb ridge at every "
|
||||||
|
"band that ducts.\n"
|
||||||
|
)
|
||||||
|
for band in bands_with_samples(joined):
|
||||||
|
table = pressure_bins(joined, band)
|
||||||
|
if table.empty:
|
||||||
|
continue
|
||||||
|
sections.append(f"\n**{band_label(band)}:**\n")
|
||||||
|
sections.append(md_table(table))
|
||||||
|
|
||||||
|
if joined_narr.empty:
|
||||||
|
sections.append(
|
||||||
|
"\n## Contact ↔ NARR per-band Pearson correlations\n\n"
|
||||||
|
"_No pre-2014 contacts ↔ NARR matches were found. Either the NARR "
|
||||||
|
"backfill has not run for this corpus yet or there are no "
|
||||||
|
"pre-2014 contacts with `pos1` set._\n"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
sections.append(
|
||||||
|
f"\n## Contact ↔ NARR per-band Pearson correlations "
|
||||||
|
f"(pre-2014 only, matched n={len(joined_narr)})\n\n"
|
||||||
|
"Historical sanity check: if the HRRR-era signs and magnitudes "
|
||||||
|
"survive on 30+ years of NARR reanalysis, the scoring factors are "
|
||||||
|
"physical rather than HRRR-artefact. Diverging signs = flag it.\n"
|
||||||
|
)
|
||||||
|
sections.append(md_table(correlations_per_band(joined_narr)))
|
||||||
|
|
||||||
sections.append(
|
sections.append(
|
||||||
f"\n## NEXRAD composite reflectivity vs distance "
|
f"\n## NEXRAD composite reflectivity vs distance "
|
||||||
|
|
@ -543,14 +654,17 @@ def main() -> int:
|
||||||
sections.append("_No contacts ↔ NEXRAD matches were found._\n")
|
sections.append("_No contacts ↔ NEXRAD matches were found._\n")
|
||||||
else:
|
else:
|
||||||
sections.append(
|
sections.append(
|
||||||
"Sanity check for the rain-attenuation direction: at rain-sensitive "
|
"Rain-attenuation sanity check — one table per band with ≥50 "
|
||||||
"bands higher `max_dbz` should map to shorter contacts. At 10 GHz "
|
"matched contacts. At rain-sensitive bands (24+ GHz) higher "
|
||||||
"the effect should be barely detectable.\n\n"
|
"`max_dbz` should map to shorter contacts; at 10 GHz and below "
|
||||||
|
"the effect should be barely detectable.\n"
|
||||||
)
|
)
|
||||||
sections.append("**10 GHz:**\n")
|
for band in bands_with_samples(nexrad):
|
||||||
sections.append(md_table(nexrad_bins(nexrad, 10_000)))
|
table = nexrad_bins(nexrad, band)
|
||||||
sections.append("\n**24 GHz:**\n")
|
if table.empty:
|
||||||
sections.append(md_table(nexrad_bins(nexrad, 24_000)))
|
continue
|
||||||
|
sections.append(f"\n**{band_label(band)}:**\n")
|
||||||
|
sections.append(md_table(table))
|
||||||
|
|
||||||
sections.append(
|
sections.append(
|
||||||
f"\n## hrrr_native_profiles.best_duct_band_ghz vs distance "
|
f"\n## hrrr_native_profiles.best_duct_band_ghz vs distance "
|
||||||
|
|
@ -562,14 +676,15 @@ def main() -> int:
|
||||||
sections.append(
|
sections.append(
|
||||||
"Validates the 1.15× boost in `Scorer.score_refractivity/4`: "
|
"Validates the 1.15× boost in `Scorer.score_refractivity/4`: "
|
||||||
"contacts where the native duct supports the target band should "
|
"contacts where the native duct supports the target band should "
|
||||||
"run longer than those where it does not.\n\n"
|
"run longer than those where it does not. One table per band with "
|
||||||
|
"≥50 matched contacts.\n"
|
||||||
)
|
)
|
||||||
sections.append("**10 GHz:**\n")
|
for band in bands_with_samples(native):
|
||||||
sections.append(md_table(native_duct_bins(native, 10_000)))
|
table = native_duct_bins(native, band)
|
||||||
sections.append("\n**24 GHz:**\n")
|
if table.empty:
|
||||||
sections.append(md_table(native_duct_bins(native, 24_000)))
|
continue
|
||||||
sections.append("\n**47 GHz:**\n")
|
sections.append(f"\n**{band_label(band)}:**\n")
|
||||||
sections.append(md_table(native_duct_bins(native, 47_000)))
|
sections.append(md_table(table))
|
||||||
|
|
||||||
sections.append(
|
sections.append(
|
||||||
f"\n## Commercial-link rx_power degradation vs contemporaneous DFW contacts "
|
f"\n## Commercial-link rx_power degradation vs contemporaneous DFW contacts "
|
||||||
|
|
|
||||||
|
|
@ -243,6 +243,23 @@ defmodule Microwaveprop.Propagation.BandConfigTest do
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
describe "weights/1 — band-aware" do
|
||||||
|
test "returns the default weights when the band has no override" do
|
||||||
|
# 50 MHz has 0 contacts in the corpus → no per-band fit → default weights.
|
||||||
|
band = BandConfig.get(50)
|
||||||
|
assert BandConfig.weights(band) == BandConfig.weights()
|
||||||
|
end
|
||||||
|
|
||||||
|
test "returns the override weights when the band has a :weights key" do
|
||||||
|
band = %{freq_mhz: 99_999, weights: %{rain: 0.5, humidity: 0.5}}
|
||||||
|
assert BandConfig.weights(band) == %{rain: 0.5, humidity: 0.5}
|
||||||
|
end
|
||||||
|
|
||||||
|
test "returns defaults when passed nil" do
|
||||||
|
assert BandConfig.weights(nil) == BandConfig.weights()
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
describe "sunrise_table/0" do
|
describe "sunrise_table/0" do
|
||||||
test "returns 12 monthly values" do
|
test "returns 12 monthly values" do
|
||||||
table = BandConfig.sunrise_table()
|
table = BandConfig.sunrise_table()
|
||||||
|
|
|
||||||
|
|
@ -142,4 +142,79 @@ defmodule Microwaveprop.Propagation.RecalibratorTest do
|
||||||
end)
|
end)
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
describe "compute_factors/3 (band-aware)" do
|
||||||
|
test "humidity score flips direction between 10 GHz (beneficial) and 24 GHz (harmful)" do
|
||||||
|
# Hot and moist profile — good for 10 GHz (high refractivity) but bad
|
||||||
|
# for 24 GHz (H2O absorption floor). Humidity factor must reflect this.
|
||||||
|
profile =
|
||||||
|
create_hrrr_profile(%{
|
||||||
|
valid_time: ~U[2024-08-15 06:00:00Z],
|
||||||
|
lat: 32.9,
|
||||||
|
lon: -97.0,
|
||||||
|
surface_temp_c: 32.0,
|
||||||
|
surface_dewpoint_c: 26.0,
|
||||||
|
surface_pressure_mb: 1005.0,
|
||||||
|
min_refractivity_gradient: -120.0,
|
||||||
|
hpbl_m: 800.0,
|
||||||
|
pwat_mm: 45.0
|
||||||
|
})
|
||||||
|
|
||||||
|
[humidity_10, _, _, _, _, _, _, _, _, _] =
|
||||||
|
Recalibrator.compute_factors(profile, ~U[2024-08-15 06:00:00Z], 10_000)
|
||||||
|
|
||||||
|
[humidity_24, _, _, _, _, _, _, _, _, _] =
|
||||||
|
Recalibrator.compute_factors(profile, ~U[2024-08-15 06:00:00Z], 24_000)
|
||||||
|
|
||||||
|
assert humidity_10 > humidity_24,
|
||||||
|
"high humidity should score higher at 10 GHz (beneficial) than 24 GHz (harmful); got #{humidity_10} vs #{humidity_24}"
|
||||||
|
end
|
||||||
|
|
||||||
|
test "defaults to 10 GHz when no band supplied" do
|
||||||
|
profile =
|
||||||
|
create_hrrr_profile(%{
|
||||||
|
valid_time: ~U[2024-08-15 06:00:00Z],
|
||||||
|
lat: 32.9,
|
||||||
|
lon: -97.0
|
||||||
|
})
|
||||||
|
|
||||||
|
default = Recalibrator.compute_factors(profile, ~U[2024-08-15 06:00:00Z])
|
||||||
|
explicit_10g = Recalibrator.compute_factors(profile, ~U[2024-08-15 06:00:00Z], 10_000)
|
||||||
|
|
||||||
|
assert default == explicit_10g
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
describe "fit/1 with :band_mhz" do
|
||||||
|
test "only pulls contacts matching the requested band" do
|
||||||
|
# Two 10 GHz and one 24 GHz contact. Asking for 24 GHz should see just one.
|
||||||
|
create_contact(%{station1: "W5AA", qso_timestamp: ~U[2024-08-15 06:00:00Z]})
|
||||||
|
create_contact(%{station1: "W5BB", qso_timestamp: ~U[2024-08-16 06:00:00Z]})
|
||||||
|
|
||||||
|
{:ok, _c} =
|
||||||
|
%Contact{}
|
||||||
|
|> Contact.changeset(%{
|
||||||
|
station1: "W5CC",
|
||||||
|
station2: "K5TR",
|
||||||
|
qso_timestamp: ~U[2024-08-17 06:00:00Z],
|
||||||
|
mode: "CW",
|
||||||
|
band: Decimal.new("24000"),
|
||||||
|
grid1: "EM12",
|
||||||
|
grid2: "EM00",
|
||||||
|
pos1: %{"lat" => 32.9, "lon" => -97.0},
|
||||||
|
pos2: %{"lat" => 30.3, "lon" => -97.7},
|
||||||
|
distance_km: Decimal.new("80")
|
||||||
|
})
|
||||||
|
|> Repo.insert()
|
||||||
|
|
||||||
|
result = Recalibrator.fit(band_mhz: 24_000, sample_size: 10, epochs: 10)
|
||||||
|
|
||||||
|
# No matching HRRR profile was inserted so this falls back to defaults —
|
||||||
|
# the contract we're validating is "the band filter was applied and only
|
||||||
|
# 24 GHz contacts were considered". The fallback weights path proves the
|
||||||
|
# band-specific contact count came through as 1, not 3.
|
||||||
|
assert is_map(result.weights)
|
||||||
|
assert map_size(result.weights) == 10
|
||||||
|
end
|
||||||
|
end
|
||||||
end
|
end
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue