First full recalibration since 2026-05-24. Corpus is 1,726,757 PSKR
samples (50/144/432 MHz) plus 74,072 contacts joined to hrrr_profiles
(microwave), yielding overrides for 12 bands.
The 6m fit moved most, and coherently: humidity 0.125 -> 0.204 and pwat
0.120 -> 0.185, paid for by season 0.217 -> 0.147. 6m contributes 1.63M
of the 1.73M PSKR samples, so ten weeks of new spot density pulled the
band off a seasonal prior and onto direct moisture signal. Next largest
is 144 MHz refractivity 0.111 -> 0.053.
75 GHz (n=93) and 122 GHz (n=63) appear in the evidence table but get
no overrides -- they fall under MIN_N_FOR_FIT, which is correct; 75 GHz
reports rho_pr of -0.774 on 93 samples, which is noise, not physics.
Caveat: the refractivity multipliers are numerically unstable. They are
computed as |band_rho_grad| / |ref_rho_grad| against the 10 GHz
reference, whose rho_grad is +0.030 -- an order of magnitude smaller
than the other reference correlations. Two runs minutes apart moved
47 GHz refractivity 0.0688 -> 0.0522 (-24% relative) purely on new rows
landing between queries. The stable factors barely moved. These weights
are live once deployed, so this commit is isolated from the script fixes
to keep it independently revertable.
Replace the two-script Python pipeline (analysis report + Elixir-source
emitter) with a single `scripts/recalibrate.py` that fits per-band
weights from PSKR spot density (VHF/UHF) and contacts↔HRRR correlations
(microwave), writing `priv/algo/band_weights.json` as a machine-readable
artifact. A new Elixir BandWeights module loads this JSON once via
`:persistent_term` cache; BandConfig.weights/1 consults it before falling
back to in-source overrides or global defaults. The script never touches
Elixir source — recalibration is now `python3 scripts/recalibrate.py`
followed by an app restart.
Both retired in light of the 2026-04-25 revision report
(`docs/algo-reports/2026-04-25-algo-revisions.md`):
* HPBL boundary-layer multiplier in `Scorer.score_refractivity/{3,4,5}`
is gone. On the n=47,418 10 GHz HRRR-matched corpus rho_hpbl is
+0.004; binned distance is flat to within ±5 % across 200–2,000 m.
The previously reported 2.3× ratio between shallow and deep HPBL
bins was a small-corpus artefact that disappeared once the matched
corpus passed ~5,000 contacts.
* Native-profile 1.15× duct boost is gone. At 10 GHz on n=52,341
matched contacts, cells where `best_duct_band_ghz` ≥ band ran
*shorter* than no-duct cells (198 km vs 211 km, n=173 vs 44,658).
The Bulk Richardson gate that existed only to suppress that boost
in turbulent conditions is also retired; arity preserved on every
`score_refractivity` overload so callers compile unchanged.
Same retirements applied in the Rust port (`rust/prop_grid_rs/src/scorer.rs`)
to keep the Elixir/Rust parity tests green.
algo.md updated end-to-end:
* Finding 4 (HPBL) rewritten to "no usable signal" with the
n=47,418 bin table.
* Finding 5 (gradient) tightened: load-bearing only at 24 GHz,
set band-specific gradient weight to 0 outside [10, 47] GHz on
the next derive_band_weights run.
* Finding 8 (time-of-day) augmented with the robust hour-bucketed
amplitude index (40 / 82 / 101 % at 10 / 24 / 47 GHz) and a
selection-bias caveat for VHF/UHF (where 129–148 % amplitude
reflects evening contest scheduling, not propagation).
* Finding 9 (sounding) refreshed against the n=27,058 corpus.
* Part 2c "Native-profile duct boost" rewritten as
"retired 2026-04-25" with the falsifying join.
* Part 2c "Commercial-link inverse sensor" promoted from deferred
with the 22-day, 38k-sample SNMP corpus, the +0.61 / −0.61 PWAT
and pressure correlations against the 37-hour HRRR overlap, and
the 04–05 / 10–12 local two-peak morning fade window.
* Part 2b "Pressure" annotated with the U-shape at 10 GHz and the
next-iteration target.
* Part 2d "What stayed, what left" + "Open items" updated to
reflect the four code/doc moves.
119 Elixir scorer tests green, 134 Rust scorer tests green, 2,888
total Elixir tests + 221 properties green via `mix precommit`.
Ran recalibrate_algo.py against the full local prop_dev (81,994 contacts,
18.6M HRRR rows) and derived per-band composite weights for the nine
bands with >=200 matched contacts. Moisture (dewpoint/PWAT/surface N)
is consistently beneficial through 5.76 GHz, reverses at 24 GHz; rain
scales sqrt(rain_k) not linearly so 24 GHz gets 0.215 rain weight
instead of the linear-ratio 0.95. 10 GHz stays as the reference band
(defaults); 47+ GHz inherits defaults (n<200).
- BandConfig.weights/1 returns per-band override or default fallback
- @band_configs carries :weights on 222/432/902/1296/2304/3400/5760/24G
- Recalibrator.compute_factors/3 + fit(band_mhz:) band-aware fitting
- Scorer + ContactLive.Show pass band_config to weights/1
- algo.md Part 2d documents the 2026-04-18 analysis + derivation rule
- scripts/derive_band_weights.py turns correlations into weight maps
- report preserved at docs/algo-reports/2026-04-18-recalibration.md
Three new analysis passes in scripts/recalibrate_algo.py that match the
scoring factors landed in commit 86364f4:
* NEXRAD composite reflectivity vs distance per band — checks that the
Marshall-Palmer rain rate in Scorer.dbz_to_rain_rate_mmhr/1 moves the
needle in the expected direction at 24+ GHz.
* hrrr_native_profiles.best_duct_band_ghz vs distance per band —
validates the 1.15× boost in Scorer.score_refractivity/4. Current
corpus shows essentially zero contacts with a native duct supporting
≥5 GHz at the target band, consistent with the 0.12% rate in Part 2c;
the boost is physics-correct but almost never fires today.
* Commercial-link rx_power degradation vs contemporaneous DFW contacts —
pearson + binned distance. Empty today because commercial_samples only
covers Mar 30–Apr 13 and contests don't overlap. Scaffolding is in
place so the moment contest-season overlap exists, the signal shows.
Report rebuilt at docs/algo-reports/2026-04-13-recalibration.md.
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k
surface obs, new hrrr_native_profiles table) drive these changes:
* Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity
score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus
removed" conclusion was an artifact of the prior matching strategy;
with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km
avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins.
* Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in
those bands stop being silently dropped. Coefficients extrapolate
ITU-R P.676/P.838 trends from the existing 134/241 GHz entries.
* Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker
max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump
the full failure body when CDS omits the message field — the prior
"ERA5 job failed: nil" was masking real error reasons in oban_jobs.
* Add scripts/recalibrate_algo.py so this analysis can be re-run any
time new data lands without manual SQL. Reads PROP_PROD_DB_URL from
.envrc, drops a Markdown report into docs/algo-reports/.
* Append a dated Part 2c to algo.md documenting the corpus expansion,
the honest accounting of historical HRRR coverage (only ~1,020 of 58k
contacts are precision-matchable), and the empty era5/rtma/climatology
tables. Update the gaseous-absorption and rain-attenuation tables to
include the new bands.
Test suite: 1,335 tests, 0 failures.
