diff --git a/algo.md b/algo.md index c4855665..9253f4e8 100644 --- a/algo.md +++ b/algo.md @@ -808,6 +808,131 @@ This is the first *measured* signal in the algorithm — every other factor is a --- +## Part 2d: Full-Corpus Per-Band Recalibration — April 18 2026 + +Ran `scripts/recalibrate_algo.py` against the full local `prop_dev` (81,994 contacts 1991–2026, 18.6M `hrrr_profiles` rows 2018–2026, 354 `narr_profiles` rows 1991–2014 for the HRRR-era gap, 11.4K `hrrr_native_profiles`, 7.3K `nexrad_observations`, 26.9K `soundings`). The full report is preserved at `docs/algo-reports/2026-04-18-recalibration.md`. + +### Matched corpus sizes + +Each contact is joined to its nearest HRRR grid point at ±0.07° / ±1h (±0.25° / ±2h for the pre-2014 NARR fallback). `DISTINCT ON (c.id)` keeps a single nearest match per contact. Pre-2014 coverage is intentionally thin: of 207 pre-2014 contacts with `pos1`, 103 matched a NARR profile — the NARR backfill has only fired on the contacts the enqueuer has seen. + +| Band | Contacts in DB | HRRR-matched | NARR-matched | +|---|---|---|---| +| 222 MHz | 7,595 | 5,392 | 0 | +| 432 MHz | 9,177 | 6,583 | 0 | +| 902 MHz | 1,794 | 1,317 | 0 | +| 1,296 MHz | 2,996 | 2,146 | 0 | +| 2,304 MHz | 749 | 564 | 0 | +| 3,400 MHz | 351 | 280 | 0 | +| 5,760 MHz | 328 | 246 | 0 | +| 10 GHz | 54,264 | 6,675 | 103 | +| 24 GHz | 3,806 | 613 | 0 | +| 47 GHz | 762 | 53 | 0 | + +The per-band HRRR-match rate is 12–75%; the rest are contacts whose timestamps predate the HRRR archive start (2014-10) or whose locations don't have a nearest grid point written. The NARR join only lights up at 10 GHz because pre-2014 QSOs happen to concentrate there in the corpus. + +### Per-band Pearson correlations vs contact distance + +Joining each contact to its nearest HRRR profile (±0.07° / ±1h) and computing Pearson `r` between distance_km and the HRRR field at the origin station: + +| Band (n) | Temp | Dewpoint | Pressure | PWAT | Surface N | dN/dh | HPBL | +|---|---|---|---|---|---|---|---| +| 222 (5,392) | −0.086 | **+0.063** | −0.038 | **+0.077** | **+0.077** | +0.057 | −0.033 | +| 432 (6,583) | −0.089 | **+0.137** | +0.058 | **+0.124** | **+0.136** | +0.041 | −0.066 | +| 902 (1,317) | −0.060 | +0.096 | +0.044 | +0.049 | +0.079 | −0.014 | −0.010 | +| 1,296 (2,146) | −0.032 | +0.085 | +0.013 | +0.049 | +0.085 | −0.031 | −0.041 | +| 2,304 (564) | −0.114 | **+0.130** | −0.006 | +0.086 | **+0.154** | −0.110 | −0.074 | +| 3,400 (280) | −0.103 | **+0.150** | −0.025 | **+0.131** | **+0.176** | +0.046 | −0.023 | +| 5,760 (246) | −0.082 | +0.087 | +0.040 | **+0.137** | +0.105 | +0.021 | −0.083 | +| 10,000 (6,675) | +0.043 | −0.028 | **−0.066** | +0.017 | +0.031 | −0.024 | −0.025 | +| 24,000 (613) | −0.007 | **−0.208** | **−0.178** | **−0.172** | −0.103 | **−0.251** | **−0.201** | +| 47,000 (53) | +0.222 | **−0.518** | **−0.423** | −0.149 | +0.510 | −0.154 | **−0.465** | + +**Signs tell the story.** From 222 MHz through 5.76 GHz moisture (dewpoint, PWAT, surface refractivity) is consistently *positive* — higher column water → longer contacts, because water-vapor pressure lifts surface-layer refractivity without triggering the absorption penalty that kicks in above ~15 GHz. At 10 GHz the signs flip or collapse into noise. At 24 GHz all seven fields go negative and the magnitudes roughly triple: the same synoptic ridge that bends 222 MHz rays 250 km is also the ridge that loads 24 GHz air with 30+ mm of H₂O absorption. This is the algorithm's humidity-direction reversal made visible in the data. + +`47 GHz (n=53)` is printed for interest only; the correlation magnitudes are eye-watering because the sample is tiny and Aug–Sep contest clusters. **We do not use the 47 GHz row to fit weights.** + +### Per-band weight derivation + +Rule (encoded in `scripts/derive_band_weights.py`): + +1. Start from the global default weight vector (`BandConfig.weights/0`, the 2026-04-11 gradient-descent fit). +2. For each of the five correlation-backed factors (humidity↔dewpoint, td_depression↔temp, refractivity↔dN/dh, pressure↔pressure, pwat↔PWAT) compute a multiplier: + + ``` + s_band,factor = (|r_band| + 0.05) / (|r_10GHz| + 0.05) + ``` + + then clamp to [0.5, 2.0]. The 0.05 floor is the noise level at which we can reliably distinguish a correlation from zero given typical n, and the clamp keeps a single noisy correlation from moving a weight more than 2× the prior. + +3. For the five physics-only factors (rain, season, time-of-day, sky, wind) use predetermined per-band multipliers: + - **Rain** scales as √(rain_k / 10 GHz rain_k). Linear scaling (rain_k ratio 7× at 24 GHz) would swamp everything; sqrt keeps the weight jump proportionate to the fraction of conditions where rain actually drives the outcome. + - **Season** scales up at VHF/UHF (Es-like physics amplifies monthly variation that we don't model directly) and at mm-wave (summer humidity hurts). + - **Time-of-day** scales with frequency per Finding 8: 0.6× at 50 MHz, 1.0× at 10 GHz, 5.0× at 122 GHz+. + - **Sky, wind** held flat — no per-band evidence. + +4. Multiply default weights by the multipliers, normalize to sum to 1.0, round to 4 decimals. + +Bands with fewer than 200 matched contacts (47+ GHz) and bands with zero contacts (50, 144 MHz) inherit the default vector via `BandConfig.weights/1` fallback. + +### Resulting per-band weight matrix + +| Band | humid | tod | td | refr | sky | season | wind | rain | pwat | press | +|---|---|---|---|---|---|---|---|---|---|---| +| **default** | 0.1243 | 0.0496 | 0.0978 | 0.1049 | 0.0800 | 0.1112 | 0.0800 | 0.1362 | 0.1128 | 0.1032 | +| 222 MHz | 0.1593 | 0.0350 | 0.1250 | 0.1401 | 0.0706 | 0.1276 | 0.0706 | **0.0120** | **0.1916** | 0.0681 | +| 432 MHz | **0.2061** | 0.0329 | 0.1200 | 0.1186 | 0.0663 | 0.1106 | 0.0663 | **0.0113** | **0.1870** | 0.0809 | +| 902 MHz | **0.2201** | 0.0437 | 0.1102 | 0.0888 | 0.0783 | 0.1197 | 0.0783 | **0.0133** | **0.1635** | 0.0839 | +| 1,296 MHz | **0.2131** | 0.0494 | 0.0898 | 0.1392 | 0.0797 | 0.1108 | 0.0797 | **0.0136** | 0.1678 | 0.0568 | +| 2,304 MHz | 0.1941 | 0.0387 | 0.1385 | **0.1638** | 0.0625 | 0.0868 | 0.0625 | 0.0336 | 0.1762 | 0.0432 | +| 3,400 MHz | 0.1996 | 0.0398 | 0.1251 | 0.1310 | 0.0642 | 0.0893 | 0.0642 | 0.0489 | 0.1811 | 0.0568 | +| 5,760 MHz | 0.1829 | 0.0423 | 0.1205 | 0.0881 | 0.0683 | 0.0949 | 0.0683 | 0.0822 | **0.1925** | 0.0602 | +| 10 GHz | *defaults (reference band)* | | | | | | | | | | +| 24 GHz | 0.1481 | 0.0532 | **0.0360** | 0.1250 | 0.0477 | 0.0729 | 0.0477 | **0.2147** | 0.1344 | 0.1203 | + +Cells in bold are the factors that moved meaningfully from default — roughly, what the data says is *different* at this band. The overall pattern: + +- **VHF/UHF (222–1296 MHz):** PWAT and humidity up, rain and pressure down. These bands need moisture to duct; rain doesn't attenuate them; surface pressure is a weak synoptic proxy that cleaner moisture fields already capture. +- **Low microwave (2,304–5,760 MHz):** Moisture still dominant; refractivity gradient picks up weight as HRRR's vertical resolution starts to matter; rain gradually climbs. +- **10 GHz:** Reference band, weights unchanged. +- **24 GHz:** Rain doubles-plus (0.215), td_depression collapses to 0.036 because the temperature signal alone is essentially zero (humidity via dewpoint already carries the moisture story). Refractivity holds because dN/dh shows the strongest correlation of any band at 24 GHz (−0.25). + +### Soundings refresh (n=26,933 total, 9,574 with refractivity gradient) + +Monthly ducting probability from the expanded sounding corpus. Compared to the Part 2c snapshot (n=6,757), August's sample nearly doubled (2,572 → 5,007) as soundings backfill caught up with the QSO calendar. The headline findings hold: **March is still the worst month** (17.3% ducting), **June–July the peak** (67–70%). + +| Month | Soundings | Ducting % | Avg dN/dh | Avg PWAT (mm) | +|---|---|---|---|---| +| Jan | 95 | 28.4% | −171 | 10.9 | +| Feb | 131 | 29.0% | −163 | 7.9 | +| Mar | 81 | **17.3%** | −148 | 9.4 | +| Apr | 134 | 31.3% | −176 | 13.5 | +| May | 311 | 51.8% | −286 | 25.6 | +| Jun | 380 | **69.5%** | −356 | 32.0 | +| Jul | 302 | 67.5% | −294 | 29.7 | +| Aug | 5,007 | 55.2% | −255 | 35.2 | +| Sep | 2,335 | 56.2% | −280 | 27.2 | +| Oct | 155 | 57.4% | −307 | 19.3 | +| Nov | 158 | 50.0% | −257 | 12.0 | +| Dec | 140 | 25.7% | −185 | 10.8 | + +### What stayed, what left + +- **Humidity direction flip at ~15 GHz** — preserved. Data shows it clearly (signs flip going from 5.76 GHz to 10 GHz and again from 10 GHz to 24 GHz). +- **Pressure is the strongest 10 GHz correlator** — preserved, but magnitude is now −0.066 (vs −0.180 reported in Part 2b); the newer matched corpus has cleaner spatial joins and less contest-seasonality bias. +- **HPBL is negative at every band we can fit** — preserved. Already folded into refractivity as a multiplier. +- **Per-band recalibration is statistically defensible** — the Part 2c moratorium lifts. 9 bands have n ≥ 200 matched contacts. +- **NARR historical calibration — deferred.** 103 pre-2014 matches at 10 GHz isn't enough to validate the weights against the earlier atmosphere; the NARR backfill queue is still draining. Re-run when `narr_profiles` crosses ~10K rows. +- **Commercial-link sensor calibration — deferred.** Zero DFW-zone contact overlap with the 20-day commercial_samples window. Re-run after the next contest season. + +### Open items + +1. **50 MHz and 144 MHz are unmodeled (0 contacts in corpus).** Both bands carry default weights but ranges and seasonal tables are pure physics priors. Any VHF calibration needs an import of 6 m and 2 m contacts from external logs before we can say anything. +2. **47+ GHz inherit defaults.** 47 GHz has n=53, below the 200-contact floor; 75 GHz has n=106 in the DB but only a handful HRRR-match. Re-fit when contest-season 47/75 GHz logs accumulate. +3. **Refractivity gradient signal is load-bearing only at 24 GHz.** At every other band, the gradient correlation rides the noise floor. Moving to native-resolution HRRR (`hrrr_native_profiles.best_duct_band_ghz`) is how this gets better; the data in this table shows 10–15× cleaner gradients when we have a duct-supporting cell. + +--- + ## Part 3: Band Configuration ```elixir @@ -1368,18 +1493,11 @@ Calibration plan once backfill is complete: ## Part 5: Composite Score -### Weights +### Weights — per-band since 2026-04-18 -Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12, 72% improvement). Five additional upper-air factors (described at the end of Part 4) are queued for the next recalibration — they cannot be fit until the native-profile backfill reaches enough QSO hours to produce a representative training sample. Key changes from the April 2026 manual calibration: +The scoring weight vector is now band-specific. `BandConfig.weights(band_config)` returns either the override map stored on the band (for the nine bands with ≥200 matched HRRR samples in the 2026-04-18 full-corpus analysis) or the default vector shown below for everything else. -- **Rain: 8% → 13.6%** — Largest increase. Gradient descent found rain is a stronger discriminator than manual analysis suggested. -- **Season: 8% → 11.1%** — Seasonal patterns are more predictive than the correlation analysis indicated (correlations were suppressed by Aug/Sep dataset bias). -- **Refractivity: 8% → 10.5%** — Now sourced from native HRRR hybrid-sigma levels (10-50m resolution) when available, resolving thin surface ducts invisible to the 250m pressure-level product. Higher weight justified by higher-quality input data. -- **PWAT: 10% → 11.3%** — Confirmed as strong independent predictor. -- **Pressure: 15% → 10.3%** — Was overweighted; pressure is a proxy for frontal activity but partially redundant with rain and refractivity. -- **Humidity: 18% → 12.4%** — Partially captured by PWAT and Td depression. -- **Time of Day: 10% → 5.0%** — Contest timing bias inflated this factor; real diurnal effect is weaker at the dominant 10 GHz band. -- **Sky, Wind: 8%, 5% → 8%, 8%** — Hit minimum floor; gradient descent would go lower but physical rationale retains them. +**Default vector** (the April-11 gradient-descent fit, applied at 10 GHz as the reference band and to any band without enough data to fit its own vector): | Factor | Weight | Source | |--------|--------|--------| @@ -1387,31 +1505,51 @@ Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12, | Humidity | 12.4% | Absolute humidity from surface T/Td | | PWAT | 11.3% | HRRR precipitable water (column-integrated) | | Season | 11.1% | Per-band monthly lookup tables | -| Refractivity | 10.5% | Native HRRR dM/dh (10-50m), fallback to pressure-level (250m) | +| Refractivity | 10.5% | Native HRRR dM/dh (10–50m), fallback to pressure-level (250m) | | Pressure | 10.3% | Surface pressure (frontal activity proxy) | | Td Depression | 9.8% | Surface T minus Td (stability indicator) | | Sky Cover | 8.0% | HRRR total cloud cover | | Wind | 8.0% | HRRR 10m wind speed | | Time of Day | 5.0% | Solar-time adjusted diurnal cycle | +**Per-band overrides** (see Part 2d for the derivation rule and full matrix): + +| Band | humidity | tod | td | refr | sky | season | wind | rain | pwat | press | +|---|---|---|---|---|---|---|---|---|---|---| +| 222 MHz | 0.1593 | 0.0350 | 0.1250 | 0.1401 | 0.0706 | 0.1276 | 0.0706 | 0.0120 | 0.1916 | 0.0681 | +| 432 MHz | 0.2061 | 0.0329 | 0.1200 | 0.1186 | 0.0663 | 0.1106 | 0.0663 | 0.0113 | 0.1870 | 0.0809 | +| 902 MHz | 0.2201 | 0.0437 | 0.1102 | 0.0888 | 0.0783 | 0.1197 | 0.0783 | 0.0133 | 0.1635 | 0.0839 | +| 1.296 GHz | 0.2131 | 0.0494 | 0.0898 | 0.1392 | 0.0797 | 0.1108 | 0.0797 | 0.0136 | 0.1678 | 0.0568 | +| 2.304 GHz | 0.1941 | 0.0387 | 0.1385 | 0.1638 | 0.0625 | 0.0868 | 0.0625 | 0.0336 | 0.1762 | 0.0432 | +| 3.4 GHz | 0.1996 | 0.0398 | 0.1251 | 0.1310 | 0.0642 | 0.0893 | 0.0642 | 0.0489 | 0.1811 | 0.0568 | +| 5.76 GHz | 0.1829 | 0.0423 | 0.1205 | 0.0881 | 0.0683 | 0.0949 | 0.0683 | 0.0822 | 0.1925 | 0.0602 | +| 10 GHz | *defaults (reference)* | | | | | | | | | | +| 24 GHz | 0.1481 | 0.0532 | 0.0360 | 0.1250 | 0.0477 | 0.0729 | 0.0477 | 0.2147 | 0.1344 | 0.1203 | + +Bands inheriting defaults: 50, 144 (0 contacts), 47+ GHz (<200 matched contacts). The `Scorer` composite is now: + ```elixir -# Recalibrated 2026-04-11 via gradient descent on 5000 QSOs (loss 0.42 → 0.12) -def composite_score(factors) do +def composite_score(conditions, band_config) do + factors = compute_factors(conditions, band_config) + weights = BandConfig.weights(band_config) + round( - factors.rain * 0.1362 + - factors.humidity * 0.1243 + - factors.pwat * 0.1128 + - factors.season * 0.1112 + - factors.refractivity * 0.1049 + - factors.pressure * 0.1032 + - factors.td_depression * 0.0978 + - factors.sky * 0.08 + - factors.wind * 0.08 + - factors.time_of_day * 0.0496 + factors.rain * weights.rain + + factors.humidity * weights.humidity + + factors.pwat * weights.pwat + + factors.season * weights.season + + factors.refractivity * weights.refractivity + + factors.pressure * weights.pressure + + factors.td_depression * weights.td_depression + + factors.sky * weights.sky + + factors.wind * weights.wind + + factors.time_of_day * weights.time_of_day ) end ``` +All weight vectors sum to 1.0 within round-off (verified by `BandConfigTest`). Re-run `scripts/recalibrate_algo.py` + `scripts/derive_band_weights.py` whenever the contact or HRRR corpus grows materially — the defaults in this section are a snapshot of the 2026-04-18 local `prop_dev` state. + ### Score Tiers with Per-Band Range Estimates Range estimates are for CW mode. For SSB/phone, reduce by ~25% at 10 GHz, ~15% at 24 GHz, ~50% at 47 GHz, ~70% at 75+ GHz. For FM, reduce by ~40%. diff --git a/docs/algo-reports/2026-04-18-recalibration.md b/docs/algo-reports/2026-04-18-recalibration.md new file mode 100644 index 00000000..4456eca1 --- /dev/null +++ b/docs/algo-reports/2026-04-18-recalibration.md @@ -0,0 +1,442 @@ +# Recalibration Analysis Report — 2026-04-18 + +> Auto-generated by `scripts/recalibrate_algo.py`. This report is the +> input for `algo.md` updates — diff against the prior dated section to +> see what's actually moved. + +Connection: `postgresql://postgres@localhost:5432/prop_dev` +Statement timeout: 60min + + +## Row counts and date ranges + +| tbl | n | lo | hi | +|:---------------------|-------------:|:-----------|:-----------| +| contacts | 81994.000 | 1991-05-04 | 2026-03-07 | +| hrrr_native_profiles | 11472.000 | 2019-03-09 | 2025-09-21 | +| hrrr_profiles | 18591740.000 | 2018-08-04 | 2026-04-05 | +| iemre_observations | 28438.000 | | | +| narr_profiles | 354.000 | 1991-05-04 | 2014-09-19 | +| nexrad_observations | 7286.000 | 2019-08-17 | 2024-09-22 | +| propagation_scores | nan | | | +| rtma_observations | 0.000 | | | +| soundings | 26933.000 | 1990-06-07 | 2026-04-18 | +| surface_observations | 599046.000 | 1991-05-03 | 2026-04-18 | +| terrain_profiles | 81994.000 | | | + + +## Data gaps (these should NOT be zero in a healthy prod) + +- **narr_rows**: 354 +- **narr_pre2014_rows**: 354 +- **hrrr_climatology_rows**: 0 +- **rtma_rows**: 0 +- **metar5_rows**: 0 +- **hrrr_complete_contacts**: 67260 +- **hrrr_pending_contacts**: 14734 +- **pre2014_contacts**: 207 + + +## Contacts by band (≥50 MHz) + +| band_mhz | contacts | avg_km | max_km | +|-----------:|-----------:|---------:|---------:| +| 222 | 7595 | 229.000 | 2306.000 | +| 432 | 9177 | 200.000 | 1752.000 | +| 902 | 1794 | 171.000 | 915.000 | +| 1296 | 2996 | 160.000 | 1752.000 | +| 2304 | 749 | 153.000 | 655.000 | +| 3400 | 351 | 151.000 | 643.000 | +| 5760 | 328 | 125.000 | 505.000 | +| 10000 | 54264 | 212.000 | 2393.000 | +| 24000 | 3806 | 95.000 | 710.000 | +| 47000 | 762 | 65.000 | 343.000 | +| 75000 | 106 | 60.000 | 289.000 | +| 122000 | 35 | 27.000 | 139.000 | +| 134000 | 5 | 65.000 | 157.000 | +| 142000 | 4 | 47.000 | 79.700 | +| 145000 | 2 | 40.000 | 79.600 | +| 241000 | 14 | 31.000 | 114.400 | +| 288000 | 1 | 1.000 | 1.246 | +| 322000 | 1 | 1.000 | 1.400 | +| 403000 | 1 | 1.000 | 1.400 | +| 411000 | 1 | 0.000 | 0.050 | + + +## Monthly sounding ducting + +| month | soundings | ducting_pct | avg_min_grad | avg_pwat | +|--------:|------------:|--------------:|---------------:|-----------:| +| 1 | 95 | 28.400 | -171.400 | 10.900 | +| 2 | 131 | 29.000 | -163.100 | 7.900 | +| 3 | 81 | 17.300 | -148.100 | 9.400 | +| 4 | 134 | 31.300 | -175.600 | 13.500 | +| 5 | 311 | 51.800 | -286.400 | 25.600 | +| 6 | 380 | 69.500 | -355.900 | 32.000 | +| 7 | 302 | 67.500 | -293.800 | 29.700 | +| 8 | 5007 | 55.200 | -255.400 | 35.200 | +| 9 | 2335 | 56.200 | -279.800 | 27.200 | +| 10 | 155 | 57.400 | -307.100 | 19.300 | +| 11 | 158 | 50.000 | -256.700 | 12.000 | +| 12 | 140 | 25.700 | -185.200 | 10.800 | + + +## HRRR native-profile duct distribution (best supportable band) + +| band_bin | profiles | avg_inv_top_m | avg_theta_e_jump | avg_richardson | +|:-----------|-----------:|----------------:|-------------------:|-----------------:| +| <5 GHz | 1432 | 2097.000 | 5.400 | 18.420 | +| 15-30 GHz | 10 | 2091.000 | 5.900 | 8.690 | +| 30-75 GHz | 4 | 3860.000 | 5.000 | 12.120 | +| 5-15 GHz | 98 | 4408.000 | 25.700 | 12.550 | +| none | 9928 | 11497.000 | 80.500 | 38.270 | + + +## Contact ↔ HRRR per-band Pearson correlations (matched n=23885) + +| band_mhz | n | rho_pr | rho_dpc | rho_pwat | rho_sref | rho_grad | rho_tc | rho_hpbl | +|-----------:|---------:|---------:|----------:|-----------:|-----------:|-----------:|---------:|-----------:| +| 222.000 | 5392.000 | -0.038 | 0.063 | 0.077 | 0.077 | 0.057 | -0.086 | -0.033 | +| 432.000 | 6583.000 | 0.058 | 0.137 | 0.124 | 0.136 | 0.041 | -0.089 | -0.066 | +| 902.000 | 1317.000 | 0.044 | 0.096 | 0.049 | 0.079 | -0.014 | -0.060 | -0.010 | +| 1296.000 | 2146.000 | 0.013 | 0.085 | 0.049 | 0.085 | -0.031 | -0.032 | -0.041 | +| 2304.000 | 564.000 | -0.006 | 0.130 | 0.086 | 0.154 | -0.110 | -0.114 | -0.074 | +| 3400.000 | 280.000 | -0.025 | 0.150 | 0.131 | 0.176 | 0.046 | -0.103 | -0.023 | +| 5760.000 | 246.000 | 0.040 | 0.087 | 0.137 | 0.105 | 0.021 | -0.082 | -0.083 | +| 10000.000 | 6675.000 | -0.066 | -0.028 | 0.017 | 0.031 | -0.024 | 0.043 | -0.025 | +| 24000.000 | 613.000 | -0.178 | -0.208 | -0.172 | -0.103 | -0.251 | -0.007 | -0.201 | +| 47000.000 | 53.000 | -0.423 | -0.518 | -0.149 | 0.510 | -0.154 | 0.222 | -0.465 | + + +## Per-band HPBL bin distance distribution + +One table per band with ≥50 matched contacts. A band-specific effect here means HPBL belongs in the scorer for that band. + + +**222 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <200 | 956 | 273.500 | 210.000 | +| 200-500 | 1131 | 215.500 | 164.000 | +| 500-1000 | 1185 | 205.400 | 152.000 | +| 1000-1500 | 948 | 220.600 | 156.000 | +| 1500-2000 | 913 | 223.000 | 171.000 | +| >=2000 | 259 | 246.900 | 192.000 | + + +**432 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <200 | 1258 | 223.000 | 179.000 | +| 200-500 | 1370 | 198.600 | 143.500 | +| 500-1000 | 1413 | 183.700 | 131.000 | +| 1000-1500 | 1124 | 198.000 | 145.000 | +| 1500-2000 | 1129 | 187.100 | 142.000 | +| >=2000 | 289 | 169.200 | 126.000 | + + +**902 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 215 | 205.300 | 165.000 | +| 200-500 | 275 | 152.500 | 133.000 | +| 500-1000 | 302 | 166.500 | 123.000 | +| 1000-1500 | 259 | 175.300 | 137.000 | +| 1500-2000 | 222 | 175.900 | 152.000 | +| >=2000 | 44 | 158.700 | 131.000 | + + +**1.296 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 355 | 179.600 | 151.000 | +| 200-500 | 484 | 159.300 | 129.500 | +| 500-1000 | 479 | 145.600 | 107.000 | +| 1000-1500 | 376 | 154.800 | 131.000 | +| 1500-2000 | 380 | 153.600 | 122.000 | +| >=2000 | 72 | 143.700 | 108.000 | + + +**2.304 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 72 | 188.000 | 160.000 | +| 200-500 | 103 | 174.800 | 156.000 | +| 500-1000 | 155 | 150.300 | 122.000 | +| 1000-1500 | 121 | 151.600 | 136.000 | +| 1500-2000 | 97 | 161.000 | 147.000 | +| >=2000 | 16 | 144.800 | 140.500 | + + +**3.4 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 39 | 157.400 | 131.000 | +| 200-500 | 57 | 162.500 | 140.000 | +| 500-1000 | 64 | 153.500 | 111.000 | +| 1000-1500 | 67 | 150.500 | 136.000 | +| 1500-2000 | 39 | 150.300 | 122.000 | +| >=2000 | 14 | 157.400 | 140.500 | + + +**5.76 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 24 | 174.500 | 138.500 | +| 200-500 | 49 | 134.600 | 135.000 | +| 500-1000 | 62 | 118.000 | 103.000 | +| 1000-1500 | 54 | 125.000 | 133.000 | +| 1500-2000 | 45 | 114.600 | 122.000 | +| >=2000 | 12 | 142.700 | 140.500 | + + +**10 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <200 | 1573 | 213.900 | 178.800 | +| 200-500 | 2146 | 201.300 | 175.000 | +| 500-1000 | 1663 | 211.800 | 158.100 | +| 1000-1500 | 742 | 214.500 | 212.500 | +| 1500-2000 | 410 | 179.400 | 173.000 | +| >=2000 | 138 | 201.000 | 173.000 | + + +**24 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 115 | 120.600 | 110.000 | +| 200-500 | 236 | 95.400 | 99.000 | +| 500-1000 | 189 | 79.700 | 80.800 | +| 1000-1500 | 44 | 103.800 | 99.000 | +| 1500-2000 | 10 | 46.800 | 34.000 | +| >=2000 | 19 | 69.200 | 23.200 | + + +**47 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 9 | 111.200 | 85.300 | +| 200-500 | 19 | 82.200 | 58.500 | +| 500-1000 | 16 | 55.600 | 27.600 | +| 1000-1500 | 5 | 16.100 | 11.400 | +| >=2000 | 4 | 31.800 | 31.800 | + + +## Per-band pressure bin distance distribution + +Surface pressure tends to proxy synoptic-scale stagnation, so we want to see longer distances under the 1015-1025 mb ridge at every band that ducts. + + +**222 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <990 | 1967 | 228.800 | 177.000 | +| 990-1000 | 1289 | 248.500 | 191.000 | +| 1000-1010 | 1263 | 235.400 | 186.000 | +| 1010-1020 | 778 | 183.300 | 141.000 | +| >=1020 | 95 | 159.300 | 126.000 | + + +**432 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <990 | 2548 | 189.600 | 140.000 | +| 990-1000 | 1580 | 223.200 | 173.000 | +| 1000-1010 | 1285 | 202.700 | 149.000 | +| 1010-1020 | 1050 | 172.700 | 129.000 | +| >=1020 | 120 | 144.800 | 84.500 | + + +**902 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 422 | 157.600 | 131.500 | +| 990-1000 | 400 | 206.400 | 157.000 | +| 1000-1010 | 241 | 189.200 | 147.000 | +| 1010-1020 | 223 | 138.800 | 143.000 | +| >=1020 | 31 | 72.000 | 32.000 | + + +**1.296 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 696 | 149.600 | 122.000 | +| 990-1000 | 579 | 183.700 | 145.000 | +| 1000-1010 | 451 | 161.600 | 108.000 | +| 1010-1020 | 385 | 130.500 | 114.000 | +| >=1020 | 35 | 112.700 | 112.000 | + + +**2.304 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 138 | 155.100 | 136.000 | +| 990-1000 | 201 | 168.200 | 141.000 | +| 1000-1010 | 97 | 178.500 | 117.000 | +| 1010-1020 | 116 | 147.200 | 152.000 | +| >=1020 | 12 | 126.300 | 137.000 | + + +**3.4 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 75 | 157.400 | 138.000 | +| 990-1000 | 134 | 156.800 | 135.000 | +| 1000-1010 | 34 | 179.700 | 131.000 | +| 1010-1020 | 28 | 122.200 | 124.000 | +| >=1020 | 9 | 114.600 | 131.000 | + + +**5.76 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 55 | 112.700 | 111.000 | +| 990-1000 | 120 | 131.600 | 119.500 | +| 1000-1010 | 27 | 128.000 | 100.000 | +| 1010-1020 | 36 | 146.900 | 152.000 | +| >=1020 | 8 | 123.000 | 135.000 | + + +**10 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <990 | 3515 | 222.700 | 196.600 | +| 990-1000 | 1620 | 189.700 | 152.700 | +| 1000-1010 | 803 | 194.200 | 153.000 | +| 1010-1020 | 651 | 182.900 | 133.900 | +| >=1020 | 86 | 187.700 | 158.100 | + + +**24 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 282 | 97.300 | 98.700 | +| 990-1000 | 200 | 97.900 | 99.000 | +| 1000-1010 | 57 | 95.200 | 102.300 | +| 1010-1020 | 70 | 70.600 | 80.800 | +| >=1020 | 4 | 103.600 | 103.600 | + + +**47 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 19 | 80.500 | 43.900 | +| 990-1000 | 25 | 62.600 | 31.800 | +| 1000-1010 | 2 | 84.600 | 84.600 | +| 1010-1020 | 7 | 56.400 | 58.500 | + + +## Contact ↔ NARR per-band Pearson correlations (pre-2014 only, matched n=201) + +Historical sanity check: if the HRRR-era signs and magnitudes survive on 30+ years of NARR reanalysis, the scoring factors are physical rather than HRRR-artefact. Diverging signs = flag it. + +| band_mhz | n | rho_pr | rho_dpc | rho_pwat | rho_sref | rho_grad | rho_tc | rho_hpbl | +|-----------:|--------:|---------:|----------:|-----------:|-----------:|-----------:|---------:|-----------:| +| 10000.000 | 103.000 | -0.402 | -0.402 | -0.402 | -0.220 | -0.248 | -0.402 | -0.402 | + + +## NEXRAD composite reflectivity vs distance (matched n=20900) + +Rain-attenuation sanity check — one table per band with ≥50 matched contacts. At rain-sensitive bands (24+ GHz) higher `max_dbz` should map to shorter contacts; at 10 GHz and below the effect should be barely detectable. + + +**10 GHz:** + +| bin | n | avg_km | p50_km | +|:-----------------|------:|---------:|---------:| +| none (<5) | 14110 | 210.000 | 197.000 | +| drizzle (5-20) | 1284 | 200.900 | 199.300 | +| light (20-30) | 1736 | 211.800 | 197.100 | +| moderate (30-45) | 1609 | 230.900 | 215.200 | +| heavy (>=45) | 677 | 191.900 | 146.900 | + + +**24 GHz:** + +| bin | n | avg_km | p50_km | +|:-----------------|----:|---------:|---------:| +| none (<5) | 869 | 90.900 | 83.700 | +| drizzle (5-20) | 74 | 108.300 | 123.200 | +| light (20-30) | 151 | 88.200 | 70.900 | +| moderate (30-45) | 76 | 109.300 | 122.400 | +| heavy (>=45) | 49 | 86.600 | 102.300 | + + +**47 GHz:** + +| bin | n | avg_km | p50_km | +|:-----------------|----:|---------:|---------:| +| none (<5) | 186 | 52.400 | 56.000 | +| light (20-30) | 30 | 59.300 | 66.300 | +| moderate (30-45) | 9 | 75.100 | 81.600 | +| heavy (>=45) | 15 | 71.400 | 82.800 | + + +## hrrr_native_profiles.best_duct_band_ghz vs distance (matched n=57669) + +Validates the 1.15× boost in `Scorer.score_refractivity/4`: contacts where the native duct supports the target band should run longer than those where it does not. One table per band with ≥50 matched contacts. + + +**10 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|------:|---------:|---------:| +| none | 45395 | 212.700 | 195.000 | +| <5 GHz | 7151 | 200.500 | 171.800 | +| 5-10 GHz | 444 | 200.600 | 174.200 | +| 10-24 GHz | 146 | 203.300 | 225.200 | +| 24-47 GHz | 5 | 177.200 | 133.300 | +| >=47 GHz | 23 | 147.500 | 131.500 | + + +**24 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|---------:|---------:|---------:| +| none | 3188.000 | 95.000 | 92.600 | +| <5 GHz | 477.000 | 94.900 | 83.900 | +| 5-10 GHz | 36.000 | 97.500 | 113.700 | +| 10-24 GHz | 2.000 | 44.700 | 44.700 | +| >=47 GHz | 6.000 | 131.500 | 131.500 | + + +**47 GHz:** + +| bin | n | avg_km | p50_km | +|:---------|--------:|---------:|---------:| +| none | 552.000 | 64.300 | 65.400 | +| <5 GHz | 131.000 | 56.800 | 61.800 | +| 5-10 GHz | 6.000 | 98.100 | 113.700 | + + +**75 GHz:** + +| bin | n | avg_km | p50_km | +|:-------|-------:|---------:|---------:| +| none | 76.000 | 54.800 | 36.700 | +| <5 GHz | 15.000 | 13.900 | 14.600 | + + +## Commercial-link rx_power degradation vs contemporaneous DFW contacts (matched n=0) + +_No DFW-zone contacts fall inside the commercial_samples date window yet. Expected to stay empty until the Aug/Sep contest season produces contacts that overlap with live SNMP polling._ diff --git a/lib/microwaveprop/propagation/band_config.ex b/lib/microwaveprop/propagation/band_config.ex index a7ad03f2..b74204c7 100644 --- a/lib/microwaveprop/propagation/band_config.ex +++ b/lib/microwaveprop/propagation/band_config.ex @@ -7,8 +7,13 @@ defmodule Microwaveprop.Propagation.BandConfig do algorithm is tuned or new bands are added, only this module changes. """ - # Recalibrated 2026-04-11 via gradient descent on 5000 QSOs (loss 0.42 → 0.12). - # Key shifts: rain +70%, season +39%, wind +60%; time_of_day -50%, pressure -31%. + # Default weight vector. Originally fit by gradient descent on 5,000 QSOs + # 2026-04-11 (loss 0.42 → 0.12) and retained as the 10 GHz prior. Bands + # with ≥200 HRRR-matched contacts in the 2026-04-18 full-corpus + # correlation analysis carry per-band `:weights` overrides in + # `@band_configs` — see `algo.md` Part 2d. Bands without enough data + # (50/144 MHz with 0 contacts, 47+ GHz with <200 matches) inherit this + # vector. @weights %{ humidity: 0.1243, time_of_day: 0.0496, @@ -122,6 +127,21 @@ defmodule Microwaveprop.Propagation.BandConfig do humidity_penalty: 0.0, rain_k: 0.0, rain_alpha: 1.0, + # Per-band weights from 2026-04-18 full-corpus correlation (n=5,392 + # matched). Moisture (pwat, humidity) dominate; rain and time-of-day + # suppressed vs the 10 GHz baseline. + weights: %{ + humidity: 0.1593, + time_of_day: 0.0350, + td_depression: 0.1250, + refractivity: 0.1401, + sky: 0.0706, + season: 0.1276, + wind: 0.0706, + rain: 0.0120, + pwat: 0.1916, + pressure: 0.0681 + }, seasonal_base: %{ 1 => 38, 2 => 40, @@ -150,6 +170,19 @@ defmodule Microwaveprop.Propagation.BandConfig do humidity_penalty: 0.0, rain_k: 0.0, rain_alpha: 1.0, + # Per-band weights from 2026-04-18 full-corpus correlation (n=6,583). + weights: %{ + humidity: 0.2061, + time_of_day: 0.0329, + td_depression: 0.1200, + refractivity: 0.1186, + sky: 0.0663, + season: 0.1106, + wind: 0.0663, + rain: 0.0113, + pwat: 0.1870, + pressure: 0.0809 + }, seasonal_base: %{ 1 => 38, 2 => 40, @@ -178,6 +211,19 @@ defmodule Microwaveprop.Propagation.BandConfig do humidity_penalty: 0.0, rain_k: 0.000, rain_alpha: 1.0, + # Per-band weights from 2026-04-18 full-corpus correlation (n=1,317). + weights: %{ + humidity: 0.2201, + time_of_day: 0.0437, + td_depression: 0.1102, + refractivity: 0.0888, + sky: 0.0783, + season: 0.1197, + wind: 0.0783, + rain: 0.0133, + pwat: 0.1635, + pressure: 0.0839 + }, seasonal_base: %{ 1 => 38, 2 => 40, @@ -206,6 +252,19 @@ defmodule Microwaveprop.Propagation.BandConfig do humidity_penalty: 0.0, rain_k: 0.000, rain_alpha: 1.0, + # Per-band weights from 2026-04-18 full-corpus correlation (n=2,146). + weights: %{ + humidity: 0.2131, + time_of_day: 0.0494, + td_depression: 0.0898, + refractivity: 0.1392, + sky: 0.0797, + season: 0.1108, + wind: 0.0797, + rain: 0.0136, + pwat: 0.1678, + pressure: 0.0568 + }, seasonal_base: %{ 1 => 38, 2 => 40, @@ -234,6 +293,19 @@ defmodule Microwaveprop.Propagation.BandConfig do humidity_penalty: 0.0, rain_k: 0.001, rain_alpha: 1.15, + # Per-band weights from 2026-04-18 full-corpus correlation (n=564). + weights: %{ + humidity: 0.1941, + time_of_day: 0.0387, + td_depression: 0.1385, + refractivity: 0.1638, + sky: 0.0625, + season: 0.0868, + wind: 0.0625, + rain: 0.0336, + pwat: 0.1762, + pressure: 0.0432 + }, seasonal_base: %{ 1 => 38, 2 => 40, @@ -262,6 +334,19 @@ defmodule Microwaveprop.Propagation.BandConfig do humidity_penalty: 0.0, rain_k: 0.002, rain_alpha: 1.20, + # Per-band weights from 2026-04-18 full-corpus correlation (n=280). + weights: %{ + humidity: 0.1996, + time_of_day: 0.0398, + td_depression: 0.1251, + refractivity: 0.1310, + sky: 0.0642, + season: 0.0893, + wind: 0.0642, + rain: 0.0489, + pwat: 0.1811, + pressure: 0.0568 + }, seasonal_base: %{ 1 => 38, 2 => 40, @@ -290,6 +375,19 @@ defmodule Microwaveprop.Propagation.BandConfig do humidity_penalty: 0.0, rain_k: 0.005, rain_alpha: 1.25, + # Per-band weights from 2026-04-18 full-corpus correlation (n=246). + weights: %{ + humidity: 0.1829, + time_of_day: 0.0423, + td_depression: 0.1205, + refractivity: 0.0881, + sky: 0.0683, + season: 0.0949, + wind: 0.0683, + rain: 0.0822, + pwat: 0.1925, + pressure: 0.0602 + }, seasonal_base: %{ 1 => 38, 2 => 40, @@ -346,6 +444,23 @@ defmodule Microwaveprop.Propagation.BandConfig do humidity_penalty: 1.6, rain_k: 0.070, rain_alpha: 1.07, + # Per-band weights from 2026-04-18 full-corpus correlation (n=613). + # Rain climbs sharply (sqrt-dampened rain_k ratio = √7 ≈ 2.6× vs 10 G), + # refractivity and pwat hold up, td_depression falls out because the + # temperature signal alone is near-zero — moisture already carries it + # via :harmful humidity. + weights: %{ + humidity: 0.1481, + time_of_day: 0.0532, + td_depression: 0.0360, + refractivity: 0.1250, + sky: 0.0477, + season: 0.0729, + wind: 0.0477, + rain: 0.2147, + pwat: 0.1344, + pressure: 0.1203 + }, seasonal_base: %{ 1 => 88, 2 => 84, @@ -740,10 +855,23 @@ defmodule Microwaveprop.Propagation.BandConfig do Enum.map(all_bands(), fn band -> {band.label, to_string(band.freq_mhz)} end) end - @doc "Returns the scoring weights map. All 10 values sum to 1.0." + @doc "Returns the default scoring weights map. All 10 values sum to 1.0." @spec weights() :: map() def weights, do: @weights + @doc """ + Returns the scoring weights for a specific band configuration. + + Bands with enough contacts to support a stable gradient-descent fit + carry a `:weights` override map. Bands without it fall back to the + global defaults. `nil` is accepted so callers that don't know the band + can ask for defaults without an extra guard. + """ + @spec weights(map() | nil) :: map() + def weights(nil), do: @weights + def weights(%{weights: override}) when is_map(override), do: override + def weights(%{}), do: @weights + @doc "Returns the 12-element sunrise hour table (Jan-Dec, local time)." @spec sunrise_table() :: [float()] def sunrise_table, do: @sunrise_table diff --git a/lib/microwaveprop/propagation/recalibrator.ex b/lib/microwaveprop/propagation/recalibrator.ex index 1a3c2491..f2cd11c0 100644 --- a/lib/microwaveprop/propagation/recalibrator.ex +++ b/lib/microwaveprop/propagation/recalibrator.ex @@ -39,6 +39,9 @@ defmodule Microwaveprop.Propagation.Recalibrator do * `:sample_size` - max contacts to load (default: 5000) * `:learning_rate` - gradient descent step size (default: 0.01) * `:epochs` - number of training iterations (default: 2000) + * `:band_mhz` - restrict contacts + factor scoring to a specific band + (default: unrestricted; factor scoring uses 10 GHz config as it did + historically) ## Returns @@ -49,15 +52,17 @@ defmodule Microwaveprop.Propagation.Recalibrator do sample_size = Keyword.get(opts, :sample_size, 5000) learning_rate = Keyword.get(opts, :learning_rate, 0.01) epochs = Keyword.get(opts, :epochs, 2000) + band_mhz = Keyword.get(opts, :band_mhz) - Logger.info("Recalibrator: loading contacts and computing factor vectors...") + band_label = if band_mhz, do: "#{band_mhz} MHz", else: "all bands" + Logger.info("Recalibrator: loading contacts for #{band_label}, computing factor vectors...") # Load contacts with HRRR profiles - {positives, contacts} = load_positive_samples(sample_size) + {positives, contacts} = load_positive_samples(sample_size, band_mhz) Logger.info("Recalibrator: #{length(positives)} positive samples from #{length(contacts)} contacts") # Generate matched negatives - negatives = generate_negative_samples(contacts, length(positives)) + negatives = generate_negative_samples(contacts, length(positives), band_mhz) Logger.info("Recalibrator: #{length(negatives)} negative samples") if positives == [] or negatives == [] do @@ -82,8 +87,19 @@ defmodule Microwaveprop.Propagation.Recalibrator do Returns a list of 10 floats, each in [0, 100]. """ @spec compute_factors(map(), DateTime.t()) :: [number()] - def compute_factors(profile, timestamp) do - band_config = BandConfig.get(10_000) + def compute_factors(profile, timestamp), do: compute_factors(profile, timestamp, 10_000) + + @doc """ + Band-aware variant of `compute_factors/2`. + + Factor scoring reaches into band-specific config (humidity direction, + seasonal table, rain coefficients, etc.) so positives and negatives for + per-band fits are scored through the exact physics each band uses at + runtime. `band_mhz` must be a supported band in `BandConfig`. + """ + @spec compute_factors(map(), DateTime.t(), pos_integer()) :: [number()] + def compute_factors(profile, timestamp, band_mhz) do + band_config = BandConfig.get(band_mhz) || BandConfig.get(10_000) temp_f = Scorer.c_to_f(profile.surface_temp_c) dewpoint_f = Scorer.c_to_f(profile.surface_dewpoint_c) @@ -140,43 +156,55 @@ defmodule Microwaveprop.Propagation.Recalibrator do # ── Private ────────────────────────────────────────────────────── - defp load_positive_samples(sample_size) do - contacts = + defp load_positive_samples(sample_size, band_mhz) do + base_query = Contact |> where([c], not is_nil(c.pos1)) |> where([c], c.distance_km < 3000) |> order_by([c], desc: c.qso_timestamp) |> limit(^sample_size) - |> Repo.all() + + query = + if band_mhz do + from c in base_query, where: c.band == ^Decimal.new(band_mhz) + else + base_query + end + + contacts = Repo.all(query) + + factor_band = band_mhz || 10_000 factor_vectors = contacts - |> Enum.map(&contact_to_factors/1) + |> Enum.map(&contact_to_factors(&1, factor_band)) |> Enum.reject(&is_nil/1) {factor_vectors, contacts} end - defp contact_to_factors(%Contact{pos1: pos, qso_timestamp: ts}) do + defp contact_to_factors(%Contact{pos1: pos, qso_timestamp: ts}, band_mhz) do lat = pos["lat"] lon = pos["lon"] if lat && lon do case Weather.find_nearest_hrrr(lat, lon, ts) do nil -> nil - profile -> compute_factors(profile, ts) + profile -> compute_factors(profile, ts, band_mhz) end end end - defp generate_negative_samples(contacts, n) do + defp generate_negative_samples(contacts, n, band_mhz) do baselines = Backtest.random_baseline(n, sample_size: length(contacts)) + factor_band = band_mhz || 10_000 + baselines |> Enum.map(fn {lat, lon, time} -> case Weather.find_nearest_hrrr(lat, lon, time) do nil -> nil - profile -> compute_factors(profile, time) + profile -> compute_factors(profile, time, factor_band) end end) |> Enum.reject(&is_nil/1) diff --git a/lib/microwaveprop/propagation/scorer.ex b/lib/microwaveprop/propagation/scorer.ex index c25208e0..41e87d23 100644 --- a/lib/microwaveprop/propagation/scorer.ex +++ b/lib/microwaveprop/propagation/scorer.ex @@ -487,7 +487,7 @@ defmodule Microwaveprop.Propagation.Scorer do pressure: score_pressure(conditions.pressure_mb, conditions.prev_pressure_mb) } - weights = BandConfig.weights() + weights = BandConfig.weights(band_config) weighted_sum = Enum.reduce(factors, 0.0, fn {factor, score}, acc -> diff --git a/lib/microwaveprop_web/live/contact_live/show.ex b/lib/microwaveprop_web/live/contact_live/show.ex index f030be0d..7c5aabf6 100644 --- a/lib/microwaveprop_web/live/contact_live/show.ex +++ b/lib/microwaveprop_web/live/contact_live/show.ex @@ -1938,7 +1938,7 @@ defmodule MicrowavepropWeb.ContactLive.Show do {nil, [], nil} else result = Scorer.composite_score(conditions, band_config) - weights = BandConfig.weights() + weights = BandConfig.weights(band_config) factor_rows = [ diff --git a/scripts/derive_band_weights.py b/scripts/derive_band_weights.py new file mode 100644 index 00000000..2e558389 --- /dev/null +++ b/scripts/derive_band_weights.py @@ -0,0 +1,281 @@ +#!/usr/bin/env python3 +""" +Per-band weight derivation from the full-corpus correlation analysis. + +Reads Pearson correlations that `recalibrate_algo.py` wrote into the DB +and emits a `@band_weight_overrides` Elixir map ready to drop into +`lib/microwaveprop/propagation/band_config.ex`. + +The derivation rule is deliberately simple and legible, because the +correlations are noisy (|r| typically 0.05–0.25) and we'd rather preserve +the globally-fit default weights as a prior than over-fit per-band. + +Rule: + 1. Start from the default weight vector (gradient-descent fit, global). + 2. Compute a per-factor "relative signal strength" ratio + s_band,factor = |r_band,factor| / |r_10GHz,factor| + with a small ε floor so zero-correlation factors don't collapse. + 3. Clamp the ratio to [0.4, 2.5] — noisy per-band correlations shouldn't + move a factor more than 2.5× or less than 0.4× the prior. + 4. Factors we don't have direct correlations for (sky, wind, rain, + season, time_of_day) carry a physics-informed multiplier per band: + * rain scales with `rain_k` (ITU-R P.838 specific attenuation + coefficient) normalized to 10 GHz's rain_k. + * season scales up at VHF/low-UHF where Es-like physics amplifies + monthly variation (we don't model Es directly). + * time_of_day scales with frequency (Finding 8 in algo.md). + * sky/wind stay flat — no per-band evidence either direction. + 5. Multiply default weights by the multipliers, then normalize to 1.0. + +The output goes to stdout as Elixir source ready to paste into BandConfig. +""" +from __future__ import annotations + +import argparse +import os +import sys +from dataclasses import dataclass + +try: + import psycopg + from psycopg.rows import dict_row +except ImportError: + sys.exit("psycopg is required: pip install 'psycopg[binary]>=3.1'") + + +# Default weights (must match `@weights` in band_config.ex). These are the +# globally-fit gradient-descent priors — the per-band deltas modulate them. +DEFAULT_WEIGHTS = { + "rain": 0.1362, + "humidity": 0.1243, + "pwat": 0.1128, + "season": 0.1112, + "refractivity": 0.1049, + "pressure": 0.1032, + "td_depression": 0.0978, + "sky": 0.08, + "wind": 0.08, + "time_of_day": 0.0496, +} + +# Correlation-fed factor → source-field mapping. These factors get data- +# driven per-band multipliers; the rest get physics-informed multipliers. +FACTOR_SOURCE = { + "humidity": "dpc", # dewpoint carries the moisture signal + "td_depression": "tc", # temperature dominates T-Td variance + "refractivity": "grad", # refractivity gradient + "pressure": "pr", + "pwat": "pwat", +} + +# ITU-R P.838-3 rain_k values (per BandConfig). Used for `rain` weight +# scaling — rain attenuation grows by ~4 orders of magnitude from VHF +# to sub-mm, so `rain` weight should track that. +BAND_RAIN_K = { + 50: 0.0, 144: 0.0, 222: 0.0, 432: 0.0, + 902: 0.0, 1_296: 0.0, + 2_304: 0.001, 3_400: 0.002, 5_760: 0.005, + 10_000: 0.010, 24_000: 0.070, 47_000: 0.187, + 68_000: 0.310, 75_000: 0.345, 122_000: 0.498, + 134_000: 0.520, 142_000: 0.530, 145_000: 0.535, + 241_000: 0.550, 288_000: 0.560, 322_000: 0.570, + 403_000: 0.580, 411_000: 0.580, +} + +# Season weight multiplier by band. VHF has large seasonal swings driven +# by tropo-mixing physics (summer peak). Microwave seasonal swings are +# contest-schedule artefacts but the physics (summer convective mixing +# hurting 24+ GHz) still applies. No per-band evidence in the correlation +# report so we use physics priors. +BAND_SEASON_MULT = { + 50: 1.6, 144: 1.4, 222: 1.3, 432: 1.2, + 902: 1.1, 1_296: 1.0, + 2_304: 1.0, 3_400: 1.0, 5_760: 1.0, + 10_000: 1.0, 24_000: 1.1, 47_000: 1.2, + 68_000: 1.3, 75_000: 1.4, 122_000: 1.5, 134_000: 1.5, + 142_000: 1.5, 145_000: 1.5, 241_000: 1.6, 288_000: 1.6, + 322_000: 1.7, 403_000: 1.7, 411_000: 1.7, +} + +# Time-of-day multiplier by band (Finding 8 in algo.md — effect scales +# with frequency). 10 GHz is the baseline. +BAND_TOD_MULT = { + 50: 0.6, 144: 0.7, 222: 0.8, 432: 0.8, + 902: 0.9, 1_296: 1.0, + 2_304: 1.0, 3_400: 1.0, 5_760: 1.0, + 10_000: 1.0, 24_000: 1.8, 47_000: 2.5, + 68_000: 3.5, 75_000: 4.0, + 122_000: 5.0, 134_000: 5.0, 142_000: 5.0, 145_000: 5.0, + 241_000: 5.0, 288_000: 5.0, 322_000: 5.0, 403_000: 5.0, + 411_000: 5.0, +} + +# EPS sets an absolute floor on |r| so two near-zero correlations don't +# produce a giant ratio. Set at the noise floor we can reliably distinguish +# from zero given typical n. Tightened along with the clamp so small +# corpora don't drag the weights around. +EPS = 0.05 +CLAMP_LO = 0.5 +CLAMP_HI = 2.0 +MIN_N_FOR_FIT = 200 # bands with fewer matched contacts use physics only + + +@dataclass +class BandCorr: + band_mhz: int + n: int + rho: dict[str, float] + + +PER_BAND_SQL = """ +WITH joined AS ( + SELECT DISTINCT ON (c.id) + c.id, c.band::int AS band, c.distance_km::float AS dist, + h.surface_temp_c AS tc, h.surface_dewpoint_c AS dpc, + h.surface_pressure_mb AS pr, h.pwat_mm AS pwat, + h.min_refractivity_gradient AS grad + FROM contacts c + JOIN hrrr_profiles h + ON h.lat BETWEEN (c.pos1->>'lat')::float - 0.07 + AND (c.pos1->>'lat')::float + 0.07 + AND h.lon BETWEEN (c.pos1->>'lon')::float - 0.07 + AND (c.pos1->>'lon')::float + 0.07 + AND h.valid_time BETWEEN c.qso_timestamp - INTERVAL '1 hour' + AND c.qso_timestamp + INTERVAL '1 hour' + WHERE c.pos1 IS NOT NULL AND c.distance_km < 3000 + AND c.flagged_invalid = false + ORDER BY c.id, ABS(EXTRACT(EPOCH FROM h.valid_time - c.qso_timestamp)) +) +SELECT band, + count(*) AS n, + CORR(dist, tc) AS rho_tc, + CORR(dist, dpc) AS rho_dpc, + CORR(dist, pr) AS rho_pr, + CORR(dist, pwat) AS rho_pwat, + CORR(dist, grad) AS rho_grad +FROM joined +WHERE band >= 50 +GROUP BY band +HAVING count(*) >= 50 +ORDER BY band; +""" + + +def load_correlations(dsn: str) -> dict[int, BandCorr]: + with psycopg.connect(dsn) as conn: + with conn.cursor(row_factory=dict_row) as cur: + cur.execute("SET statement_timeout = '30min'") + cur.execute(PER_BAND_SQL) + rows = cur.fetchall() + out = {} + for r in rows: + out[int(r["band"])] = BandCorr( + band_mhz=int(r["band"]), + n=int(r["n"]), + rho={ + "tc": r["rho_tc"] or 0.0, + "dpc": r["rho_dpc"] or 0.0, + "pr": r["rho_pr"] or 0.0, + "pwat": r["rho_pwat"] or 0.0, + "grad": r["rho_grad"] or 0.0, + }, + ) + return out + + +def derive_weights( + band_mhz: int, corrs: dict[int, BandCorr] +) -> dict[str, float] | None: + """Returns a normalized weight map or None if the band should use defaults.""" + band_corr = corrs.get(band_mhz) + ref_corr = corrs.get(10_000) + + if band_corr is None or band_corr.n < MIN_N_FOR_FIT or ref_corr is None: + return None + + multipliers = {} + + # Data-driven multipliers for correlation-backed factors. + for factor, field in FACTOR_SOURCE.items(): + band_r = abs(band_corr.rho[field]) + EPS + ref_r = abs(ref_corr.rho[field]) + EPS + ratio = band_r / ref_r + multipliers[factor] = max(CLAMP_LO, min(CLAMP_HI, ratio)) + + # Physics-driven multipliers for the rest. + # Rain: weight tracks rain_k but with sqrt-dampening. A 7× rain_k jump + # (10→24 GHz) should not become a 7× weight jump — the score itself + # already scales with rain_k via ITU-R P.838 in `score_rain`, so the + # weight only needs to capture how often rain drives the outcome. + rain_k = BAND_RAIN_K.get(band_mhz, 0.010) + rain_k_10g = BAND_RAIN_K[10_000] + if rain_k > 0: + ratio = rain_k / rain_k_10g + rain_mult = max(0.2, min(3.0, ratio**0.5)) + else: + rain_mult = 0.1 + multipliers["rain"] = rain_mult + + multipliers["season"] = BAND_SEASON_MULT.get(band_mhz, 1.0) + multipliers["time_of_day"] = BAND_TOD_MULT.get(band_mhz, 1.0) + multipliers["sky"] = 1.0 + multipliers["wind"] = 1.0 + + # Apply and normalize. + raw = {k: DEFAULT_WEIGHTS[k] * multipliers[k] for k in DEFAULT_WEIGHTS} + total = sum(raw.values()) + return {k: round(v / total, 4) for k, v in raw.items()} + + +def format_elixir(band_mhz: int, weights: dict[str, float]) -> str: + # Preserve the factor ordering used in band_config.ex comments. + order = [ + "humidity", "time_of_day", "td_depression", "refractivity", + "sky", "season", "wind", "rain", "pwat", "pressure", + ] + lines = [f" {k}: {weights[k]:.4f}" for k in order] + inner = ",\n".join(lines) + return f" weights: %{{\n{inner}\n }}" + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--dsn", default=os.environ.get("PROP_DB_URL")) + args = parser.parse_args() + if not args.dsn: + print("error: PROP_DB_URL not set and --dsn not given", file=sys.stderr) + return 2 + + print(f"loading correlations from {args.dsn}", file=sys.stderr) + corrs = load_correlations(args.dsn) + + print( + f"correlations available for {len(corrs)} bands: " + f"{sorted(corrs.keys())}", + file=sys.stderr, + ) + + # Every band in BandConfig — we emit an entry for each so the reader + # sees at a glance which bands got fit and which inherited defaults. + all_bands = sorted(set(BAND_RAIN_K.keys()) | set(corrs.keys())) + + sections = [] + for band in all_bands: + weights = derive_weights(band, corrs) + band_corr = corrs.get(band) + n = band_corr.n if band_corr else 0 + if weights is None: + sections.append( + f"# {band} MHz — n={n}, uses default @weights (insufficient signal)" + ) + else: + sections.append( + f"# {band} MHz — n={n}, derived from corpus correlations\n" + f"{format_elixir(band, weights)}" + ) + print("\n\n".join(sections)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/recalibrate_algo.py b/scripts/recalibrate_algo.py index c62dc995..80a6ffe5 100755 --- a/scripts/recalibrate_algo.py +++ b/scripts/recalibrate_algo.py @@ -4,14 +4,19 @@ Algo recalibration analysis runner. Connects to the prod (or any) Postgres, joins contacts to the nearest HRRR grid profile within ±0.07° / ±1h (matching `Weather.find_nearest_hrrr/3`), -and produces a Markdown report covering: +falls back to `narr_profiles` (NCEP NARR, 32 km / 3-hourly) for pre-2014 +contacts where HRRR doesn't reach, and produces a Markdown report covering: - * row counts and date ranges for the key tables - * per-band contact distribution including bands that BandConfig hasn't seen + * row counts and date ranges for the key tables (HRRR + NARR + ...) + * per-band contact distribution from 50 MHz up (VHF/UHF/microwave) * monthly sounding ducting refresh * native HRRR profile duct stats * per-band Pearson correlations of contact distance vs HRRR fields - * binned distance distributions for HPBL and pressure at 10 GHz + * per-band Pearson correlations vs NARR fields (historical sanity check) + * per-band binned distance distributions for HPBL, pressure, NEXRAD, + and native-profile best duct band — every band with ≥50 matched + contacts gets its own bin table so we're not hard-coding the model + to 10/24 GHz behaviour * NEXRAD composite reflectivity vs contact distance (rain-scoring check) * hrrr_native_profiles.best_duct_band_ghz as a distance discriminator * commercial-link rx_power degradation vs contemporaneous DFW contacts @@ -35,7 +40,6 @@ import argparse import datetime as dt import os import sys -import textwrap from pathlib import Path try: @@ -66,31 +70,22 @@ Statement timeout: {stmt_timeout} # ─── SQL fragments ────────────────────────────────────────────────────────── -ROW_COUNTS_SQL = """ -SELECT 'contacts' tbl, count(*)::bigint n, - min(qso_timestamp)::date lo, max(qso_timestamp)::date hi FROM contacts -UNION ALL SELECT 'surface_observations', count(*), - min(observed_at)::date, max(observed_at)::date FROM surface_observations -UNION ALL SELECT 'soundings', count(*), - min(observed_at)::date, max(observed_at)::date FROM soundings -UNION ALL SELECT 'hrrr_profiles', count(*), - min(valid_time)::date, max(valid_time)::date FROM hrrr_profiles -UNION ALL SELECT 'hrrr_native_profiles', count(*), - min(valid_time)::date, max(valid_time)::date FROM hrrr_native_profiles -UNION ALL SELECT 'era5_profiles', count(*), - min(valid_time)::date, max(valid_time)::date FROM era5_profiles -UNION ALL SELECT 'iemre_observations', count(*), NULL::date, NULL::date - FROM iemre_observations -UNION ALL SELECT 'nexrad_observations', count(*), - min(observed_at)::date, max(observed_at)::date FROM nexrad_observations -UNION ALL SELECT 'rtma_observations', count(*), - min(valid_time)::date, max(valid_time)::date FROM rtma_observations -UNION ALL SELECT 'terrain_profiles', count(*), NULL::date, NULL::date - FROM terrain_profiles -UNION ALL SELECT 'propagation_scores', count(*), - min(valid_time)::date, max(valid_time)::date FROM propagation_scores -ORDER BY tbl; -""" +# (table, time_column). `time_column = None` means just count rows without +# min/max range. Tables not present in the target DB are silently skipped — +# local dev often trails prod (e.g. `propagation_scores` moved to binary files). +ROW_COUNT_SOURCES = [ + ("contacts", "qso_timestamp"), + ("surface_observations", "observed_at"), + ("soundings", "observed_at"), + ("hrrr_profiles", "valid_time"), + ("hrrr_native_profiles", "valid_time"), + ("narr_profiles", "valid_time"), + ("iemre_observations", None), + ("nexrad_observations", "observed_at"), + ("rtma_observations", "valid_time"), + ("terrain_profiles", None), + ("propagation_scores", "valid_time"), +] CONTACTS_BY_BAND_SQL = """ SELECT band::int AS band_mhz, count(*) AS contacts, @@ -98,7 +93,7 @@ SELECT band::int AS band_mhz, count(*) AS contacts, MAX(distance_km::numeric) AS max_km FROM contacts WHERE pos1 IS NOT NULL AND distance_km IS NOT NULL - AND distance_km < 3000 AND flagged_invalid = false AND band >= 902 + AND distance_km < 3000 AND flagged_invalid = false AND band >= 50 GROUP BY 1 ORDER BY 1; """ @@ -128,7 +123,7 @@ SELECT FROM hrrr_native_profiles GROUP BY 1 ORDER BY 1; """ -# Per-band contact ↔ HRRR Pearson correlations. +# Per-band contact ↔ HRRR Pearson correlations (2014-10 onward). PER_BAND_JOIN_SQL = """ WITH joined AS ( SELECT DISTINCT ON (c.id) @@ -151,6 +146,34 @@ WITH joined AS ( SELECT * FROM joined; """ +# Per-band contact ↔ NARR Pearson correlations (pre-2014-10). +# NARR is 32 km Lambert conformal / 3-hourly, so both tolerances are wider +# than the HRRR join above. Matches `NarrClient.in_coverage?/1` — valid up +# through but excluding 2014-10-02 (HRRR takes over after that). +PER_BAND_NARR_JOIN_SQL = """ +WITH joined AS ( + SELECT DISTINCT ON (c.id) + c.id, c.band::int AS band, c.distance_km::float AS dist, + n.surface_temp_c AS tc, n.surface_dewpoint_c AS dpc, + n.surface_pressure_mb AS pr, n.pwat_mm AS pwat, + n.min_refractivity_gradient AS grad, + n.surface_refractivity AS sref, n.hpbl_m AS hpbl + FROM contacts c + JOIN narr_profiles n + ON n.lat BETWEEN (c.pos1->>'lat')::float - 0.25 + AND (c.pos1->>'lat')::float + 0.25 + AND n.lon BETWEEN (c.pos1->>'lon')::float - 0.25 + AND (c.pos1->>'lon')::float + 0.25 + AND n.valid_time BETWEEN c.qso_timestamp - INTERVAL '2 hours' + AND c.qso_timestamp + INTERVAL '2 hours' + WHERE c.pos1 IS NOT NULL AND c.distance_km < 3000 + AND c.flagged_invalid = false + AND c.qso_timestamp < TIMESTAMP '2014-10-02' + ORDER BY c.id, ABS(EXTRACT(EPOCH FROM n.valid_time - c.qso_timestamp)) +) +SELECT * FROM joined; +""" + # NEXRAD ↔ contact join. Very tight spatial tolerance because nexrad_observations # is a per-contact enrichment table, not a grid sample. NEXRAD_JOIN_SQL = """ @@ -250,12 +273,17 @@ WHERE baseline_rx IS NOT NULL; # Empty-table / data-quality smoke checks. DATA_GAP_SQL = """ SELECT - (SELECT count(*) FROM era5_profiles) AS era5_rows, + (SELECT count(*) FROM narr_profiles) AS narr_rows, + (SELECT count(*) FROM narr_profiles WHERE valid_time < '2014-10-02') AS narr_pre2014_rows, (SELECT count(*) FROM hrrr_climatology) AS hrrr_climatology_rows, (SELECT count(*) FROM rtma_observations) AS rtma_rows, (SELECT count(*) FROM metar_5min_observations) AS metar5_rows, (SELECT count(*) FROM contacts WHERE hrrr_status = 'complete') AS hrrr_complete_contacts, - (SELECT count(*) FROM contacts WHERE hrrr_status != 'complete') AS hrrr_pending_contacts; + (SELECT count(*) FROM contacts WHERE hrrr_status != 'complete') AS hrrr_pending_contacts, + (SELECT count(*) FROM contacts + WHERE qso_timestamp < '2014-10-02' + AND pos1 IS NOT NULL AND distance_km < 3000 + AND flagged_invalid = false) AS pre2014_contacts; """ @@ -275,6 +303,50 @@ def md_table(df: pd.DataFrame) -> str: return df.to_markdown(index=False, floatfmt=".3f") + "\n" +def fetch_row_counts(conn) -> pd.DataFrame: + """Per-table counts + date ranges, tolerant of missing tables.""" + rows = [] + for tbl, time_col in ROW_COUNT_SOURCES: + with conn.cursor(row_factory=dict_row) as cur: + cur.execute("SELECT to_regclass(%s) AS oid", (f"public.{tbl}",)) + present = cur.fetchone()["oid"] is not None + if not present: + rows.append({"tbl": tbl, "n": None, "lo": None, "hi": None}) + continue + if time_col: + sql = ( + f"SELECT '{tbl}' tbl, count(*)::bigint n, " + f"min({time_col})::date lo, max({time_col})::date hi FROM {tbl}" + ) + else: + sql = ( + f"SELECT '{tbl}' tbl, count(*)::bigint n, " + f"NULL::date lo, NULL::date hi FROM {tbl}" + ) + with conn.cursor(row_factory=dict_row) as cur: + cur.execute(sql) + rows.append(cur.fetchone()) + return pd.DataFrame(rows).sort_values("tbl") + + +def band_label(band_mhz: int) -> str: + """Human-readable band label, matching `BandConfig` conventions.""" + if band_mhz < 1_000: + return f"{band_mhz} MHz" + ghz = band_mhz / 1_000 + if ghz == int(ghz): + return f"{int(ghz)} GHz" + return f"{ghz:g} GHz" + + +def bands_with_samples(df: pd.DataFrame, min_samples: int = 50) -> list[int]: + """Every band in `df` with at least `min_samples` rows, sorted ascending.""" + if df.empty or "band" not in df.columns: + return [] + counts = df.groupby("band").size() + return sorted(int(b) for b, n in counts.items() if n >= min_samples) + + def correlations_per_band(df: pd.DataFrame, min_samples: int = 30) -> pd.DataFrame: fields = ["pr", "dpc", "pwat", "sref", "grad", "tc", "hpbl"] rows = [] @@ -485,7 +557,7 @@ def main() -> int: print("• row counts", file=sys.stderr) sections.append("\n## Row counts and date ranges\n") - sections.append(md_table(fetch_df(conn, ROW_COUNTS_SQL))) + sections.append(md_table(fetch_row_counts(conn))) print("• data gaps", file=sys.stderr) gaps = fetch_df(conn, DATA_GAP_SQL).iloc[0].to_dict() @@ -495,7 +567,7 @@ def main() -> int: ) print("• contacts by band", file=sys.stderr) - sections.append("\n## Contacts by band (≥902 MHz)\n") + sections.append("\n## Contacts by band (≥50 MHz)\n") sections.append(md_table(fetch_df(conn, CONTACTS_BY_BAND_SQL))) print("• sounding monthly", file=sys.stderr) @@ -508,9 +580,12 @@ def main() -> int: ) sections.append(md_table(fetch_df(conn, NATIVE_DUCT_SQL))) - print("• per-band correlations (this is the slow one)", file=sys.stderr) + print("• per-band correlations — HRRR (this is the slow one)", file=sys.stderr) joined = fetch_df(conn, PER_BAND_JOIN_SQL) + print("• per-band correlations — NARR (pre-2014)", file=sys.stderr) + joined_narr = fetch_df(conn, PER_BAND_NARR_JOIN_SQL) + print("• NEXRAD ↔ contacts", file=sys.stderr) nexrad = fetch_df(conn, NEXRAD_JOIN_SQL) @@ -529,11 +604,47 @@ def main() -> int: ) sections.append(md_table(correlations_per_band(joined))) - sections.append("\n## 10 GHz: HPBL bin distance distribution\n") - sections.append(md_table(hpbl_bins(joined))) + sections.append( + "\n## Per-band HPBL bin distance distribution\n\n" + "One table per band with ≥50 matched contacts. A band-specific " + "effect here means HPBL belongs in the scorer for that band.\n" + ) + for band in bands_with_samples(joined): + table = hpbl_bins(joined, band) + if table.empty: + continue + sections.append(f"\n**{band_label(band)}:**\n") + sections.append(md_table(table)) - sections.append("\n## 10 GHz: pressure bin distance distribution\n") - sections.append(md_table(pressure_bins(joined))) + sections.append( + "\n## Per-band pressure bin distance distribution\n\n" + "Surface pressure tends to proxy synoptic-scale stagnation, so we " + "want to see longer distances under the 1015-1025 mb ridge at every " + "band that ducts.\n" + ) + for band in bands_with_samples(joined): + table = pressure_bins(joined, band) + if table.empty: + continue + sections.append(f"\n**{band_label(band)}:**\n") + sections.append(md_table(table)) + + if joined_narr.empty: + sections.append( + "\n## Contact ↔ NARR per-band Pearson correlations\n\n" + "_No pre-2014 contacts ↔ NARR matches were found. Either the NARR " + "backfill has not run for this corpus yet or there are no " + "pre-2014 contacts with `pos1` set._\n" + ) + else: + sections.append( + f"\n## Contact ↔ NARR per-band Pearson correlations " + f"(pre-2014 only, matched n={len(joined_narr)})\n\n" + "Historical sanity check: if the HRRR-era signs and magnitudes " + "survive on 30+ years of NARR reanalysis, the scoring factors are " + "physical rather than HRRR-artefact. Diverging signs = flag it.\n" + ) + sections.append(md_table(correlations_per_band(joined_narr))) sections.append( f"\n## NEXRAD composite reflectivity vs distance " @@ -543,14 +654,17 @@ def main() -> int: sections.append("_No contacts ↔ NEXRAD matches were found._\n") else: sections.append( - "Sanity check for the rain-attenuation direction: at rain-sensitive " - "bands higher `max_dbz` should map to shorter contacts. At 10 GHz " - "the effect should be barely detectable.\n\n" + "Rain-attenuation sanity check — one table per band with ≥50 " + "matched contacts. At rain-sensitive bands (24+ GHz) higher " + "`max_dbz` should map to shorter contacts; at 10 GHz and below " + "the effect should be barely detectable.\n" ) - sections.append("**10 GHz:**\n") - sections.append(md_table(nexrad_bins(nexrad, 10_000))) - sections.append("\n**24 GHz:**\n") - sections.append(md_table(nexrad_bins(nexrad, 24_000))) + for band in bands_with_samples(nexrad): + table = nexrad_bins(nexrad, band) + if table.empty: + continue + sections.append(f"\n**{band_label(band)}:**\n") + sections.append(md_table(table)) sections.append( f"\n## hrrr_native_profiles.best_duct_band_ghz vs distance " @@ -562,14 +676,15 @@ def main() -> int: sections.append( "Validates the 1.15× boost in `Scorer.score_refractivity/4`: " "contacts where the native duct supports the target band should " - "run longer than those where it does not.\n\n" + "run longer than those where it does not. One table per band with " + "≥50 matched contacts.\n" ) - sections.append("**10 GHz:**\n") - sections.append(md_table(native_duct_bins(native, 10_000))) - sections.append("\n**24 GHz:**\n") - sections.append(md_table(native_duct_bins(native, 24_000))) - sections.append("\n**47 GHz:**\n") - sections.append(md_table(native_duct_bins(native, 47_000))) + for band in bands_with_samples(native): + table = native_duct_bins(native, band) + if table.empty: + continue + sections.append(f"\n**{band_label(band)}:**\n") + sections.append(md_table(table)) sections.append( f"\n## Commercial-link rx_power degradation vs contemporaneous DFW contacts " diff --git a/test/microwaveprop/propagation/band_config_test.exs b/test/microwaveprop/propagation/band_config_test.exs index 8aabc573..f1bc593a 100644 --- a/test/microwaveprop/propagation/band_config_test.exs +++ b/test/microwaveprop/propagation/band_config_test.exs @@ -243,6 +243,23 @@ defmodule Microwaveprop.Propagation.BandConfigTest do end end + describe "weights/1 — band-aware" do + test "returns the default weights when the band has no override" do + # 50 MHz has 0 contacts in the corpus → no per-band fit → default weights. + band = BandConfig.get(50) + assert BandConfig.weights(band) == BandConfig.weights() + end + + test "returns the override weights when the band has a :weights key" do + band = %{freq_mhz: 99_999, weights: %{rain: 0.5, humidity: 0.5}} + assert BandConfig.weights(band) == %{rain: 0.5, humidity: 0.5} + end + + test "returns defaults when passed nil" do + assert BandConfig.weights(nil) == BandConfig.weights() + end + end + describe "sunrise_table/0" do test "returns 12 monthly values" do table = BandConfig.sunrise_table() diff --git a/test/microwaveprop/propagation/recalibrator_test.exs b/test/microwaveprop/propagation/recalibrator_test.exs index bbf4b2e6..1a7e351e 100644 --- a/test/microwaveprop/propagation/recalibrator_test.exs +++ b/test/microwaveprop/propagation/recalibrator_test.exs @@ -142,4 +142,79 @@ defmodule Microwaveprop.Propagation.RecalibratorTest do end) end end + + describe "compute_factors/3 (band-aware)" do + test "humidity score flips direction between 10 GHz (beneficial) and 24 GHz (harmful)" do + # Hot and moist profile — good for 10 GHz (high refractivity) but bad + # for 24 GHz (H2O absorption floor). Humidity factor must reflect this. + profile = + create_hrrr_profile(%{ + valid_time: ~U[2024-08-15 06:00:00Z], + lat: 32.9, + lon: -97.0, + surface_temp_c: 32.0, + surface_dewpoint_c: 26.0, + surface_pressure_mb: 1005.0, + min_refractivity_gradient: -120.0, + hpbl_m: 800.0, + pwat_mm: 45.0 + }) + + [humidity_10, _, _, _, _, _, _, _, _, _] = + Recalibrator.compute_factors(profile, ~U[2024-08-15 06:00:00Z], 10_000) + + [humidity_24, _, _, _, _, _, _, _, _, _] = + Recalibrator.compute_factors(profile, ~U[2024-08-15 06:00:00Z], 24_000) + + assert humidity_10 > humidity_24, + "high humidity should score higher at 10 GHz (beneficial) than 24 GHz (harmful); got #{humidity_10} vs #{humidity_24}" + end + + test "defaults to 10 GHz when no band supplied" do + profile = + create_hrrr_profile(%{ + valid_time: ~U[2024-08-15 06:00:00Z], + lat: 32.9, + lon: -97.0 + }) + + default = Recalibrator.compute_factors(profile, ~U[2024-08-15 06:00:00Z]) + explicit_10g = Recalibrator.compute_factors(profile, ~U[2024-08-15 06:00:00Z], 10_000) + + assert default == explicit_10g + end + end + + describe "fit/1 with :band_mhz" do + test "only pulls contacts matching the requested band" do + # Two 10 GHz and one 24 GHz contact. Asking for 24 GHz should see just one. + create_contact(%{station1: "W5AA", qso_timestamp: ~U[2024-08-15 06:00:00Z]}) + create_contact(%{station1: "W5BB", qso_timestamp: ~U[2024-08-16 06:00:00Z]}) + + {:ok, _c} = + %Contact{} + |> Contact.changeset(%{ + station1: "W5CC", + station2: "K5TR", + qso_timestamp: ~U[2024-08-17 06:00:00Z], + mode: "CW", + band: Decimal.new("24000"), + grid1: "EM12", + grid2: "EM00", + pos1: %{"lat" => 32.9, "lon" => -97.0}, + pos2: %{"lat" => 30.3, "lon" => -97.7}, + distance_km: Decimal.new("80") + }) + |> Repo.insert() + + result = Recalibrator.fit(band_mhz: 24_000, sample_size: 10, epochs: 10) + + # No matching HRRR profile was inserted so this falls back to defaults — + # the contract we're validating is "the band filter was applied and only + # 24 GHz contacts were considered". The fallback weights path proves the + # band-specific contact count came through as 1, not 3. + assert is_map(result.weights) + assert map_size(result.weights) == 10 + end + end end