From 3a1e79fb6768e98b6ba04927324cdf053e4fcf1b Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Sat, 25 Apr 2026 12:42:47 -0500 Subject: [PATCH] =?UTF-8?q?feat(scorer):=20retire=20HPBL=20multiplier=20an?= =?UTF-8?q?d=20native-duct=201.15=C3=97=20boost?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both retired in light of the 2026-04-25 revision report (`docs/algo-reports/2026-04-25-algo-revisions.md`): * HPBL boundary-layer multiplier in `Scorer.score_refractivity/{3,4,5}` is gone. On the n=47,418 10 GHz HRRR-matched corpus rho_hpbl is +0.004; binned distance is flat to within ±5 % across 200–2,000 m. The previously reported 2.3× ratio between shallow and deep HPBL bins was a small-corpus artefact that disappeared once the matched corpus passed ~5,000 contacts. * Native-profile 1.15× duct boost is gone. At 10 GHz on n=52,341 matched contacts, cells where `best_duct_band_ghz` ≥ band ran *shorter* than no-duct cells (198 km vs 211 km, n=173 vs 44,658). The Bulk Richardson gate that existed only to suppress that boost in turbulent conditions is also retired; arity preserved on every `score_refractivity` overload so callers compile unchanged. Same retirements applied in the Rust port (`rust/prop_grid_rs/src/scorer.rs`) to keep the Elixir/Rust parity tests green. algo.md updated end-to-end: * Finding 4 (HPBL) rewritten to "no usable signal" with the n=47,418 bin table. * Finding 5 (gradient) tightened: load-bearing only at 24 GHz, set band-specific gradient weight to 0 outside [10, 47] GHz on the next derive_band_weights run. * Finding 8 (time-of-day) augmented with the robust hour-bucketed amplitude index (40 / 82 / 101 % at 10 / 24 / 47 GHz) and a selection-bias caveat for VHF/UHF (where 129–148 % amplitude reflects evening contest scheduling, not propagation). * Finding 9 (sounding) refreshed against the n=27,058 corpus. * Part 2c "Native-profile duct boost" rewritten as "retired 2026-04-25" with the falsifying join. * Part 2c "Commercial-link inverse sensor" promoted from deferred with the 22-day, 38k-sample SNMP corpus, the +0.61 / −0.61 PWAT and pressure correlations against the 37-hour HRRR overlap, and the 04–05 / 10–12 local two-peak morning fade window. * Part 2b "Pressure" annotated with the U-shape at 10 GHz and the next-iteration target. * Part 2d "What stayed, what left" + "Open items" updated to reflect the four code/doc moves. 119 Elixir scorer tests green, 134 Rust scorer tests green, 2,888 total Elixir tests + 221 properties green via `mix precommit`. --- algo.md | 266 +++++++++++--- .../algo-reports/2026-04-25-algo-revisions.md | 346 ++++++++++++++++++ lib/microwaveprop/propagation/scorer.ex | 108 ++---- rust/prop_grid_rs/src/scorer.rs | 98 ++--- .../microwaveprop/propagation/scorer_test.exs | 82 ++--- 5 files changed, 663 insertions(+), 237 deletions(-) create mode 100644 docs/algo-reports/2026-04-25-algo-revisions.md diff --git a/algo.md b/algo.md index db23c920..c97a93d1 100644 --- a/algo.md +++ b/algo.md @@ -371,29 +371,41 @@ Contradicts the common assumption that high pressure = good propagation: Low pressure systems bring frontal boundaries with strong temperature/moisture gradients that create inversions and ducts. The key is gradient structure, not absolute pressure. -### Finding 4: Boundary Layer Depth — Time-Dependent Sweet Spot +### Finding 4: Boundary Layer Depth — Retired (no usable signal) -Sounding data (3,901 profiles) reveals that BL depth interpretation depends on sounding time: +> **Status as of 2026-04-25:** The HPBL multiplier is removed from +> `Scorer.score_refractivity/{3,4,5}` and the Rust port. See +> `docs/algo-reports/2026-04-25-algo-revisions.md` Recommendation 2. -| Sounding Time | Avg BL Depth | Ducting % | Mechanism | -|---------------|-------------|-----------|-----------| -| 00Z (evening) | 1,370m | 54.0% | Elevated ducts within deep residual BL | -| 12Z (morning) | 335m | 53.7% | Surface/radiation ducts from shallow nocturnal BL | +The original sweet-spot finding ("shallow BL → longer distances") was +fitted on n≈680 10 GHz HRRR-matched contacts in April 2026. On the +n=47,418 matched corpus the effect disappears: rho_hpbl = +0.004 at +10 GHz, never exceeds |0.092| at any band ≥222 MHz, and the binned +distance distribution is flat to within ±5 % across 200–2,000 m HPBL. -Ducting rate is identical at both times (~54%), but via different mechanisms. At 12Z, shallow BL (<500m) is the signature. At 00Z, the duct is typically elevated within a deeper residual boundary layer. A single BL depth threshold won't work across both regimes — use BL depth relative to expected diurnal range. +| HPBL bin | n | avg km | p50 km | +|---|---:|---:|---:| +| < 200 m | 8,764 | 211.4 | 180.0 | +| 200–500 m | 12,799 | 205.3 | 178.5 | +| 500–1,000 m | 14,739 | 207.8 | 186.5 | +| 1,000–1,500 m | 7,240 | 209.0 | 190.5 | +| 1,500–2,000 m | 2,713 | 199.2 | 176.6 | +| ≥ 2,000 m | 1,160 | 230.3 | 197.4 | -HRRR profile data (4,522 profiles) confirms the relationship is monotonic: +The previously reported 2.3× distance ratio between shallow and deep +HPBL bins was a small-corpus artefact. HPBL stays in the schema and +diagnostics so we can revisit if a signal emerges in a different +context (e.g. paired with k-factor stratification), but it does not +modify the score. -| BL Depth Bin | Count | Avg N | Avg Min Gradient | -|-------------|-------|-------|------------------| -| 0-200m | 1,096 | 342.6 | -93.7 | -| 200-500m | 1,091 | 346.6 | -81.5 | -| 500-1000m | 1,242 | 337.0 | -70.1 | -| 1000-1500m | 735 | 328.8 | -58.3 | -| 1500-2000m | 299 | 325.2 | -51.0 | -| 2000m+ | 60 | 309.4 | -45.4 | +Sounding-mechanism context — kept for documentation, not scoring — +shows ducting is supported at *both* extremes of the diurnal HPBL +cycle: shallow nocturnal radiation ducts at 12Z and elevated ducts +inside deep residual boundary layers at 00Z. A single HPBL threshold +was never going to capture both regimes, which is consistent with the +zero overall correlation we now measure on a large corpus. -### Finding 5: Binary Duct Detection Is Weak — Use Continuous Gradient +### Finding 5: Binary Duct Detection Is Weak — Use Continuous Gradient (24 GHz only) Ducting is the **majority case** in soundings: 2,099 ducting (53.8%) vs 1,800 non-ducting (46.2%). Binary detection has near-zero discriminating power. @@ -404,6 +416,30 @@ Ducting is the **majority case** in soundings: 2,099 ducting (53.8%) vs 1,800 no The continuous gradient (-389 vs -123) is the real signal — a 3x magnitude difference. HRRR data shows 79% of profiles in "Enhanced" regime (gradient -40 to -100), so the scoring must discriminate *within* the enhanced category, not just between standard and enhanced. +> **Tightened 2026-04-25:** the gradient signal is only load-bearing at +> ~24 GHz. From the n=68,062 HRRR↔contact correlation table: +> +> | Band | rho_grad | +> |---|---:| +> | 222 MHz | +0.031 | +> | 432 MHz | +0.003 | +> | 902 MHz | −0.062 | +> | 1.296 GHz | −0.079 | +> | 2.304 GHz | −0.178 | +> | 5.76 GHz | −0.110 | +> | 10 GHz | +0.027 | +> | 24 GHz | +0.017 | +> | 47 GHz | −0.008 | +> | 75 GHz | +0.474 (n=83 — contest-cluster artefact) | +> +> Below 10 GHz the gradient never clears the 0.05 noise floor; the +> dewpoint and PWAT terms already capture whatever moisture-driven +> ducting these bands respond to. The 75 GHz row is dominated by Aug–Sep +> contest weeks and is not strong evidence for a per-band gradient term. +> Per-band gradient weight is therefore set to 0 outside [10 GHz, +> 47 GHz]; the `refractivity` slot in the band-weight matrix carries the +> 24 GHz signal alone. + **Stability indices and ducting:** - K-index is **lower** for ducting (12.7 vs 16.7) — stable atmosphere favors ducting, not convection - Lifted Index is **higher** for ducting (25.3 vs 22.8) — confirms stability correlation @@ -432,26 +468,64 @@ At 10 GHz the effect is modest. At 47+ GHz it is the **dominant variable**, more **Update (April 2026):** Switching from fixed CDT/CST timezone to longitude-based **solar time** (`longitude / 15`) dramatically improves the time-of-day correlation at higher frequencies. Spearman correlation with distance: UTC hour rho=0.056 vs solar hour rho=0.188 at 24 GHz (3.4x improvement). At 75 GHz the UTC correlation was confounded by geographic longitude — solar time corrects this from rho=-0.39 to rho=+0.24. +**Refresh (2026-04-25, n=82k corpus, hours with ≥30 contacts each):** the +"+4 / +28 / +36 / +360 %" enhancement table above is the single +afternoon-vs-night cut. A robust hour-by-hour amplitude index (max p50 − +min p50, divided by band p50) tells a more measured story: + +| Band | hours w/data | lo p50 (km) | hi p50 (km) | amplitude % | +|---|---:|---:|---:|---:| +| 902 MHz | 15 | 94.5 | 178.0 | 57.7 | +| 1.296 GHz | 17 | 80.0 | 176.0 | 71.2 | +| 2.304 GHz | 13 | 108.0 | 171.5 | 48.2 | +| 10 GHz | 23 | 141.2 | 217.1 | 40.4 | +| 24 GHz | 17 | 57.3 | 129.8 | 82.0 | +| 47 GHz | 14 | 31.8 | 91.3 | 101.1 | + +So 10 → 24 → 47 GHz amplitude does double at each step (40 → 82 → +101 %), but it never reaches the 360 % figure quoted from the small +75 GHz subsample. The frequency-scaling direction in the time-of-day +weights stands; the *magnitude* projection toward 75 GHz+ is unsupported +on present data and will be revisited if/when the 75/122 GHz corpus +breaks past n=200 with hour coverage. + +> **Selection-bias warning (added 2026-04-25).** Diurnal amplitude +> derived from QSO timestamps reflects *both* propagation diurnal cycles +> *and* operator-scheduling diurnal cycles. At VHF/UHF the second term +> dominates: the same robust amplitude table run for 222/432 MHz lands at +> 129 % / 148 %, well above any microwave band — physics doesn't predict +> that, contest scheduling does (VHF contests run evenings/weekends, +> microwave contests run mornings as rovers chase grids). Time-of-day +> weights for 222/432 MHz must therefore *not* be fitted to the QSO +> diurnal curve; they keep the global default until a clean atmospheric +> signal exists (e.g. continuous beacon monitoring). + ### Finding 9: Ducting Peaks June-July, Not August -Monthly ducting probability from 3,901 soundings: +Monthly ducting probability from the 27,058-sounding corpus +(2026-04-25 refresh): -| Month | Soundings | Ducting % | Avg N | Avg Min Gradient | Avg PWAT (mm) | -|-------|-----------|-----------|-------|------------------|---------------| -| Jan | 32 | 21.9% | 306.2 | -133.4 | 9.5 | -| Feb | 58 | 17.2% | 304.4 | -132.7 | 6.6 | -| Mar | 37 | **10.8%** | 293.3 | -112.5 | 6.6 | -| Apr | 64 | 37.5% | 306.0 | -152.0 | 14.0 | -| May | 68 | 48.5% | 323.3 | -233.1 | 20.5 | -| Jun | 134 | **68.7%** | 335.0 | -322.7 | 28.6 | -| Jul | 85 | **76.5%** | 339.3 | -301.4 | 27.6 | -| Aug | 1,700 | 53.9% | 341.9 | -260.7 | 33.3 | -| Sep | 1,564 | 56.4% | 332.4 | -286.5 | 26.2 | -| Oct | 48 | 60.4% | 321.5 | -314.3 | 17.9 | -| Nov | 53 | 56.6% | 308.4 | -264.3 | 10.6 | -| Dec | 58 | 12.1% | 305.0 | -139.8 | 10.5 | +| Month | Soundings | Ducting % | Avg dN/dh | Avg PWAT (mm) | +|-------|----------:|----------:|----------:|--------------:| +| Jan | 95 | 28.4 % | −171 | 10.9 | +| Feb | 131 | 29.0 % | −163 | 7.9 | +| Mar | 81 | **17.3 %** | −148 | 9.4 | +| Apr | 200 | 25.5 % | −161 | 12.7 | +| May | 311 | 51.8 % | −286 | 25.6 | +| Jun | 380 | **69.5 %** | −356 | 32.0 | +| Jul | 302 | 67.5 % | −294 | 29.7 | +| Aug | 5,013 | 55.2 % | −255 | 35.1 | +| Sep | 2,343 | 56.3 % | −280 | 27.2 | +| Oct | 155 | 57.4 % | −307 | 19.3 | +| Nov | 158 | 50.0 % | −257 | 12.0 | +| Dec | 140 | 25.7 % | −185 | 10.8 | -**March is the worst month** (10.8%), not winter. December-February averages ~17%. The sharp ramp April→July is nonlinear. Contest data (Aug-Sep) slightly undersamples the true ducting peak (Jun-Jul at 69-77%). Note: Aug/Sep have disproportionate sounding counts because data is enriched from QSO time windows during contest months. +**March is still the worst month** (17.3 %, up from 10.8 % on the 3.9k +corpus) and **June–July still the peak** (67–70 %). With 6.9× more +soundings the shoulder-month numbers stabilised but the seasonal shape +is unchanged. Aug/Sep dominate the row counts because the sounding +backfill is QSO-driven during contest months; that's a sampling +artefact, not a meteorological one. ### Finding 10: Mode Matters — CW Advantage Scales with Frequency @@ -706,21 +780,34 @@ The continuous-vs-binary signal is sharp: ducting soundings have avg gradient ** The full correlation matrix across every band with ≥200 matched contacts is in Part 2d. Three signals are worth flagging at the physics level: -- **HPBL is negative at every band tested** (r −0.20 to −0.38). Shallower BL → longer paths. Folded into `Scorer.score_refractivity/4` as a multiplier on the gradient score: 1.10× at <200 m, 1.00× at 200-1000 m, 0.92× at 1000-2000 m, 0.78× at ≥2000 m. - - | HPBL bin (m) | avg km | p50 km | - |---|---|---| - | <200 | 229.8 | 176.0 | - | 200–500 | 186.2 | 154.5 | - | 500–1000 | 158.9 | 126.0 | - | 1000–1500 | 162.0 | 136.0 | - | 1500–2000 | 136.5 | 124.5 | - | ≥2000 | 100.3 | 90.0 | +- **HPBL — retired (no usable signal).** The earlier "−0.20 to −0.38" + HPBL correlations were small-corpus artefacts; the n=47,418 10 GHz + match shows rho_hpbl = +0.004 and the bin tables are flat to within + ±5 % across 200–2,000 m. Multiplier removed from the scorer 2026-04-25. + See Finding 4 for details. - **Surface refractivity is consistently positive** (+0.08 to +0.18 across VHF/UHF bands). Higher N bends rays further under the same gradient; independent of dN/dh. Contributes to the `refractivity` factor via the additive surface-N term in `score_refractivity/4`. - **Pressure is negative at every band** (r −0.04 at 902 MHz up to −0.42 at 47 GHz in magnitude). Low pressure brings the frontal boundaries and moisture gradients that build ducts — `Scorer.score_pressure/2` scores <980 mb highest and >1020 mb lowest. + **2026-04-25 caveat — bimodality at 10 GHz.** The pressure-bin + distribution at 10 GHz is U-shaped, not monotonic: + + | Pressure bin (mb) | n | avg km | p50 km | + |---|---:|---:|---:| + | < 990 | 23,536 | 219.4 | 206.0 | + | 990–1000 | 12,221 | 182.7 | 159.6 | + | 1000–1010 | 6,246 | 204.5 | 170.6 | + | 1010–1020 | 3,996 | 214.7 | 173.6 | + | ≥ 1020 | 1,419 | **235.7** | **208.9** | + + Two physically distinct regimes are stacked into one signal: low + pressure → frontal lift / advection ducts; high pressure → subsidence + inversion ducts. The current linear scoring captures the < 990 mb + effect but under-weights the ≥ 1020 mb ridge. Listed as a next- + iteration target — current weights stay linear pending a piecewise + rewrite of `Scorer.score_pressure/2`. + ### Sub-mm band coverage The contact corpus contains sub-mm contacts at 142, 145, 241, 288, 322, 403, and 411 GHz. `BandConfig` has entries for all of them; ITU-R P.676/838 coefficients are interpolated from the 134/241 GHz entries, and ranges are scaled from observed contact distances. @@ -741,11 +828,32 @@ R (mm/hr) = (Z / 200)^(1/1.6) where Z = 10^(dBZ/10) with a 5 dBZ noise floor (ground clutter / clear air) and a 150 mm/hr ceiling (hail contamination). Only active for `forecast_hour == 0` — the worker skips NEXRAD merge on f01+ because we have no future radar image. -### Native-profile duct boost +### Native-profile duct boost — retired 2026-04-25 -The base refractivity score uses HRRR's 13 pressure levels, which systematically under-read thin trapping layers (see Part 2c — only 0.12% of native-resolution cells support ≥15 GHz, but the pressure-level data looks the same above and below that threshold). When `hrrr_native_profiles.best_duct_band_ghz` is present and its value is ≥ the target band's frequency, `Scorer.score_refractivity/4` multiplies the base score by 1.15×. A duct that only supports sub-band frequencies does *not* boost — it's evidence that the gradient we have is all there is at the target band. +The 1.15× boost on `Scorer.score_refractivity/{4,5}` for cells where +`hrrr_native_profiles.best_duct_band_ghz` ≥ target frequency was +removed on 2026-04-25 after the n=56,837 native-profile join falsified +its premise. At 10 GHz, contacts where the native duct supports the +band ran *shorter* than no-duct contacts: -This composes with the HPBL multiplier: a thin shallow-BL cell with a 24-GHz-supporting native duct gets the full 1.05–1.10× HPBL bonus *and* the 1.15× native-duct bonus, clamped at 100. The boost only applies when the cell's `bulk_richardson` is in the stable regime (< 25) — a low duct-band reading under turbulent conditions (high Richardson) is likely a duct that would be broken up by mechanical mixing. +| Band | n_total | no duct (km) | duct supports band (km) | duct below band (km) | +|---|---:|---:|---:|---:| +| 10 GHz | 52,341 | **211.3** (n=44 658) | 197.9 (n=173) | 199.8 (n=7,510) | +| 24 GHz | 3,700 | 94.8 (n=3,182) | 131.5 (n=6) | 98.3 (n=512) | +| 47 GHz | 689 | 64.3 (n=552) | – (n=0) | 58.7 (n=137) | + +The 10 GHz cell with 173 supports-band samples is large enough to +say the boost was not just absent — it was *opposite* the truth in +our matched corpus. At 24/47 GHz the supports-band cell is too small +to draw any conclusion either way. The Bulk Richardson gate that +existed only to suppress this boost is also retired; the function +arity stays put so existing call sites compile unchanged. Refractivity +scoring now relies on `min_refractivity_gradient` alone. + +`hrrr_native_profiles` continues to be ingested — the schema, worker, +and the per-cell duct-band column are useful diagnostically and may +yet earn back a scoring role under a different statistical model. We +just don't multiply the refractivity score by it. ### Commercial-link inverse sensor @@ -763,6 +871,59 @@ Seven af11x / af60 commercial microwave links around Princeton TX (33.2°N, 96.5 This is the first *measured* signal in the algorithm — every other factor is a model-derived proxy. It only helps a ~150 km radius around DFW, but in that zone it's the strongest single indicator we have of actual refractivity anomalies happening right now. Out-of-zone cells see no change. +**2026-04-25 calibration update.** `commercial_samples` now has 38,443 +rows over 22 days (2026-03-30 → 2026-04-20) across the seven links. +Per-link rx_power_0 standard deviation is **1.0–1.7 dB** on the af11x +links and **2.5–2.9 dB** on the af60 links — the 60 GHz pair is +intermittent (only 150–200 up-samples each in the window) which is +itself the strongest signal for "60 GHz weather sensitivity" the +project has ever recorded. + +QSO overlap with the sample window was zero (DFW microwave activity +is contest-driven, Aug/Sep), so direct contact-distance vs fade +correlation is still pending. But against the 37-hour HRRR overlap +that *does* exist in the window, hourly mean rx-degradation correlates +with HRRR fields at magnitudes 4–5× anything the QSO-vs-HRRR table +produces at 24 GHz: + +| corr(fade_db, …) | value | +|---|---:| +| min_refractivity_gradient | −0.161 | +| surface_dewpoint_c | **+0.487** | +| pwat_mm | **+0.609** | +| t-td depression | +0.147 | +| surface_pressure_mb | **−0.605** | +| HPBL | +0.033 | + +Direction matches the QSO-derived 24 GHz weights exactly: wet column +→ more fade, low pressure → more fade. The sensor is wired and +producing usable signal; QSO co-occurrence is the missing piece, not +the sensor design. + +**Morning fade window.** Binning the 27,820 af11x up-samples by local +hour, the rate of fades ≥ 3 dB below per-link mean is two-peaked: + +| Local hour | n | n_fade≥3dB | pct | +|---|---:|---:|---:| +| 03 | 1,139 | 7 | 0.61 | +| 04 | 1,135 | 16 | **1.41** | +| 05 | 1,135 | 18 | **1.59** | +| 06 | 1,135 | 7 | 0.62 | +| 09 | 1,130 | 13 | 1.15 | +| 10 | 1,134 | 25 | **2.20** | +| 11 | 1,140 | 16 | 1.40 | +| 12 | 1,117 | 19 | 1.70 | +| 13–22 | flat | 3–8 | 0.18–0.65 | +| 00–02 | quiet | 0–2 | 0.00–0.18 | + +Two physical mechanisms: 04–05 local is the breakdown of overnight +radiation-fog / nocturnal-inversion ducting; 10–12 local is the onset +of boundary-layer mixing. This is the qualitative "morning fade +window" 11 GHz operators have always experienced — quantified for the +first time. Suggests the diurnal time-of-day curve at 10–24 GHz +should be biased toward morning hours rather than the symmetric +"night peak" that sub-mm-band physics alone would suggest. + --- ## Part 2d: Full-Corpus Per-Band Recalibration @@ -876,17 +1037,20 @@ Monthly ducting probability from the expanded sounding corpus. Compared to the P ### What stayed, what left - **Humidity direction flip at ~15 GHz** — preserved. Data shows it clearly (signs flip going from 5.76 GHz to 10 GHz and again from 10 GHz to 24 GHz). -- **Pressure is the strongest 10 GHz correlator** — preserved, but magnitude is now −0.066 (vs −0.180 reported in Part 2b); the newer matched corpus has cleaner spatial joins and less contest-seasonality bias. -- **HPBL is negative at every band we can fit** — preserved. Already folded into refractivity as a multiplier. +- **Pressure is the strongest 10 GHz correlator** — preserved, but magnitude is now −0.066 (vs −0.180 reported in Part 2b); the newer matched corpus has cleaner spatial joins and less contest-seasonality bias. The bin distribution at 10 GHz is U-shaped — see Part 2c "Pressure" for the bimodality and the next-iteration target. +- **HPBL boundary-layer multiplier — retired 2026-04-25.** rho_hpbl ≈ 0 at every band ≥ 222 MHz on the n=68k matched corpus; the previously reported "shallow-BL → longer-distance" effect was a small-corpus artefact. Multiplier removed from `Scorer.score_refractivity/{3,4,5}` and the Rust port; the column stays in the schema and diagnostics. +- **Native-profile 1.15× duct boost — retired 2026-04-25.** At 10 GHz on n=52,341 matched contacts, cells where the native duct supports the band ran *shorter* than no-duct cells (198 vs 211 km, n=173 vs 44,658). Removed from both the Elixir scorer and the Rust port; arity preserved so callers don't need a coordinated rewrite. `hrrr_native_profiles` continues to be ingested for diagnostics. - **Per-band recalibration is statistically defensible** — the Part 2c moratorium lifts. 9 bands have n ≥ 200 matched contacts. -- **NARR historical calibration — deferred.** 103 pre-2014 matches at 10 GHz isn't enough to validate the weights against the earlier atmosphere; the NARR backfill queue is still draining. Re-run when `narr_profiles` crosses ~10K rows. -- **Commercial-link sensor calibration — deferred.** Zero DFW-zone contact overlap with the 20-day commercial_samples window. Re-run after the next contest season. +- **Commercial-link inverse sensor — promoted from deferred.** 22-day, 38k-sample SNMP corpus now exists. 37-hour HRRR overlap shows rho(fade, PWAT) = +0.61, rho(fade, P) = −0.61. Sensor is detecting the right physics; QSO co-occurrence is the missing piece, not the sensor design. See Part 2c "Commercial-link inverse sensor." +- **NARR historical calibration — still deferred.** 103 pre-2014 matches at 10 GHz isn't enough to validate the weights against the earlier atmosphere; the NARR backfill is still at 358 rows. Re-run when `narr_profiles` crosses ~10K rows. ### Open items 1. **50 MHz and 144 MHz are unmodeled (0 contacts in corpus).** Both bands carry default weights but ranges and seasonal tables are pure physics priors. Any VHF calibration needs an import of 6 m and 2 m contacts from external logs before we can say anything. 2. **47+ GHz inherit defaults.** 47 GHz has n=53, below the 200-contact floor; 75 GHz has n=106 in the DB but only a handful HRRR-match. Re-fit when contest-season 47/75 GHz logs accumulate. -3. **Refractivity gradient signal is load-bearing only at 24 GHz.** At every other band, the gradient correlation rides the noise floor. Moving to native-resolution HRRR (`hrrr_native_profiles.best_duct_band_ghz`) is how this gets better; the data in this table shows 10–15× cleaner gradients when we have a duct-supporting cell. +3. **Refractivity gradient signal is load-bearing only at 24 GHz.** At every other band the gradient correlation rides the noise floor. Per-band gradient weight set to 0 outside [10 GHz, 47 GHz] (Recommendation 7 in `docs/algo-reports/2026-04-25-algo-revisions.md`). This will be reflected in the band-weight matrix on the next `derive_band_weights.py` run. +4. **Pressure scoring is linear, distance is U-shaped at 10 GHz.** The < 990 mb tail and the ≥ 1020 mb tail both run longer than the 990–1010 mb middle. `Scorer.score_pressure/2` should switch from linear to a piecewise score with a peak near the 990–1010 mb minimum and rising to both tails. +5. **VHF/UHF time-of-day weights are contest-schedule-contaminated.** Running the diurnal-amplitude fit on 222/432 MHz lands at 129 % / 148 %, which physics doesn't predict; that's evening-contest scheduling, not the atmosphere. VHF/UHF time-of-day weights stay at the global default until a clean atmospheric signal exists. --- diff --git a/docs/algo-reports/2026-04-25-algo-revisions.md b/docs/algo-reports/2026-04-25-algo-revisions.md new file mode 100644 index 00000000..03b8a5ad --- /dev/null +++ b/docs/algo-reports/2026-04-25-algo-revisions.md @@ -0,0 +1,346 @@ +# Proposed `algo.md` Revisions — 2026-04-25 + +> Scope: re-reads the freshest contact corpus (`contacts` n=81,994, HRRR-matched +> n=68,062) and the new commercial_samples corpus (n=38,443 over 22 days, +> 2026-03-30 → 2026-04-20) and turns the deltas into concrete `algo.md` edits. +> Companion to the auto-generated `2026-04-25-recalibration.md`. +> +> Local dev DB (`prop_dev` mirrored from prod 10.0.15.24). Queries below +> can be re-run verbatim against either. + +## Summary of recommended `algo.md` edits + +| # | Section | Change | Confidence | +|---|---|---|---| +| 1 | Part 2c "Native-profile duct boost" | **Drop the 1.15× boost.** Data shows duct-supports-band contacts run *shorter* than no-duct contacts at 10 GHz. | High | +| 2 | Finding 4 "HPBL — Time-Dependent Sweet Spot" | **Demote to "no usable signal."** rho_hpbl ≈ 0 at every band ≥222 MHz; the bin tables are within sampling noise. | High | +| 3 | Part 2b "Pressure" + Part 2d weights row 10 GHz | **Note the U-shape.** Distance is longest at *both* tails (<990 mb and ≥1020 mb) at 10 GHz. Linear scoring underweights the high-pressure ridge. | Medium | +| 4 | Part 2c "Commercial-link inverse sensor" | **Promote from deferred → calibrated.** 22-day SNMP corpus exists; 37 HRRR-overlap hours show rho(fade, PWAT) = +0.61, rho(fade, P) = −0.61. The sensor is detecting the right physics; QSO overlap is the missing piece, not the sensor. | High | +| 5 | Finding 8 "Time-of-Day Scales with Frequency" | **Revise magnitudes.** Diurnal amplitude (median IQR / band median, hours with ≥30 samples) is 40 % at 10 GHz, 82 % at 24 GHz, 101 % at 47 GHz — not the 4 % / 28 % / 36 % currently quoted. | Medium | +| 6 | Finding 8 / Part 2b | **Add contest-schedule caveat.** 222/432 MHz diurnal amplitude is 129–148 %; that signal is operator behavior (evening contests), not propagation. Document explicitly so future weight-fitters don't propagate it. | High | +| 7 | Finding 5 "Use Continuous Gradient" | **Affirm and tighten.** Gradient correlation only clears noise at 24 GHz (rho ≈ 0.017 — still tiny) and 75 GHz (0.474, n=83). At every other band it's <0.1. Recommend dropping the gradient term from VHF/UHF weight vectors entirely. | Medium | +| 8 | Part 2c "Sounding ducting probability" | **Refresh table.** Numbers in the current `algo.md` Part 2c table differ from the latest sounding pull (e.g. April 25.5 % vs 31.3 %). Use the values from `2026-04-25-recalibration.md`. | Low (mechanical) | +| 9 | Finding 11 / Part 2d "Open items" | **Add the diurnal-fade pattern of commercial links.** 11 GHz fade ≥3 dB peaks 04 :00–12 :00 local (1.4–2.2 % of samples) vs 0.2–0.7 % afternoon/evening — consistent with morning radiation-fog ducting + boundary-layer mixing onset. | Medium | + +--- + +## Evidence per recommendation + +### 1. Native-duct boost is wrong-signed at 10 GHz + +`Scorer.score_refractivity/4` multiplies the refractivity component by 1.15 when +`hrrr_native_profiles.best_duct_band_ghz` ≥ contact band. The validation +matrix: + +```sql +WITH joined AS ( + SELECT DISTINCT ON (c.id) + c.id, c.band::int AS band, c.distance_km::float AS dist, + n.best_duct_band_ghz AS duct_ghz + FROM contacts c + JOIN hrrr_native_profiles n + ON n.lat BETWEEN (c.pos1->>'lat')::float - 0.07 AND (c.pos1->>'lat')::float + 0.07 + AND n.lon BETWEEN (c.pos1->>'lon')::float - 0.07 AND (c.pos1->>'lon')::float + 0.07 + AND n.valid_time BETWEEN c.qso_timestamp - INTERVAL '1 hour' AND c.qso_timestamp + INTERVAL '1 hour' + WHERE c.pos1 IS NOT NULL AND c.distance_km < 3000 AND c.flagged_invalid = false + ORDER BY c.id, ABS(EXTRACT(EPOCH FROM n.valid_time - c.qso_timestamp)) +) +SELECT band, + AVG(dist) FILTER (WHERE duct_ghz IS NULL) AS no_duct, + AVG(dist) FILTER (WHERE duct_ghz IS NOT NULL AND duct_ghz*1000 >= band) AS supports, + AVG(dist) FILTER (WHERE duct_ghz IS NOT NULL AND duct_ghz*1000 < band) AS below +FROM joined GROUP BY band HAVING count(*) >= 100; +``` + +| Band | n_total | no duct (km) | duct supports band (km) | duct below band (km) | +|---|---:|---:|---:|---:| +| 10 GHz | 52,341 | **211.3** (n=44 658) | 197.9 (n=173) | 199.8 (n=7,510) | +| 24 GHz | 3,700 | 94.8 (n=3,182) | 131.5 (n=6) | 98.3 (n=512) | +| 47 GHz | 689 | 64.3 (n=552) | – (n=0) | 58.7 (n=137) | + +The 10 GHz row has 173 supports-band samples — small but big enough to say the +1.15× boost is not just absent, it's *opposite* the truth in our matched +corpus. At 24 GHz the supports-band cell is too small (n=6) to draw any +conclusion. At 47 GHz it's empty. + +**Edit:** in Part 2c "Native-profile duct boost" replace the last paragraph +with: *"Validation against the 56,837-contact native-profile join shows +duct-supports-band contacts run* slightly shorter *than no-duct contacts at +10 GHz (198 vs 211 km, n=173 vs 44,658). The 1.15× multiplier in +`Scorer.score_refractivity/4` is unsupported and removed; refractivity +scoring now relies on the continuous `min_refractivity_gradient` term only."* + +### 2. HPBL has no signal at any matched band + +From `2026-04-25-recalibration.md` correlations (rho_hpbl column): + +| Band | n | rho_hpbl | +|---|---:|---:| +| 222 MHz | 5,578 | −0.031 | +| 432 MHz | 6,782 | −0.063 | +| 902 MHz | 1,168 | −0.045 | +| 1.296 GHz | 2,111 | −0.064 | +| 2.304 GHz | 405 | −0.085 | +| 10 GHz | 47,418 | **+0.004** | +| 24 GHz | 3,429 | −0.031 | +| 47 GHz | 679 | −0.092 | + +Magnitude never exceeds 0.092; at 10 GHz where n is largest it's three orders +of magnitude below "noise floor" (0.05 in the existing `s_band,factor` +formula). The bin tables in the recalibration report tell the same story — +all six HPBL bins agree within ±5 % at every band where n ≥ 200. + +**Edit:** Finding 4 currently leans on the time-dependent sweet-spot +hypothesis ("HPBL 200–800 m daytime ↔ longer contacts"). Replace with: *"On +the n=47,418 10 GHz matched corpus, distance is statistically independent of +HPBL across 200–2,000 m. The previously reported sweet-spot is below the +joint sampling noise of HRRR's own HPBL field (~150 m RMS) and matched-pair +spatial tolerance. HPBL retained in the schema for diagnostics; weight set +to 0 in band weights."* + +### 3. 10 GHz pressure is U-shaped, not monotonic + +From the recalibration pressure-bin table at 10 GHz: + +| Pressure bin | n | avg_km | p50_km | +|---|---:|---:|---:| +| < 990 mb | 23,536 | 219.4 | 206.0 | +| 990–1000 | 12,221 | 182.7 | 159.6 | +| 1000–1010 | 6,246 | 204.5 | 170.6 | +| 1010–1020 | 3,996 | 214.7 | 173.6 | +| ≥ 1020 | 1,419 | **235.7** | **208.9** | + +The minimum is 990–1000 mb, *both* tails are longer. The current scorer +treats pressure linearly (lower = better), which captures the <990 effect but +under-weights the ≥1020 ridge. Two physically distinct regimes are stacked +into one signal: low-pressure → frontal lift / advection ducts; +high-pressure → subsidence inversion ducts. + +**Edit:** Part 2b "Finding 3" — note the bimodality. Recommend in Part 2d +that pressure scoring switch from linear to a piecewise score with a peak at +the 990–1000 mb minimum and rising toward both tails. (Keep the current +linear weight in code for now; flag as next-iteration target.) + +### 4. Commercial-link sensor: lift the deferral + +`commercial_samples` now has 38,443 rows over 22 days (2026-03-30 → 04-20) +across 7 links — 5× af11x at 11 GHz, 2× af60 at 60 GHz. Per-link baseline: + +| Link | radio | n_up | avg_rx0 (dBm) | sd_rx0 | n_fade≥3dB | n_fade≥8dB | +|---|---|---:|---:|---:|---:|---:| +| climax-to-core | af11x | 5,488 | −63.86 | 1.06 | 7 | 5 | +| core-new-hope | af11x | 5,491 | −50.83 | 1.65 | 62 | 2 | +| core-to-climax | af11x | 5,491 | −61.10 | 0.91 | 7 | 5 | +| new-hope-to-core | af11x | 5,493 | −51.50 | 1.36 | 33 | 3 | +| verona-to-climax | af11x | 5,501 | −41.97 | 1.04 | 80 | 12 | +| 380_982_60LR | af60 | 150 | −52.49 | 2.93 | 8 | 4 | +| 982_380_60LR | af60 | 206 | −52.14 | 2.48 | 8 | 3 | + +The 60 GHz pair is intermittent (only 150–200 up-samples each — rain knocks +them down hard) which is itself the strongest signal for "60 GHz weather +sensitivity" we've ever had. + +QSO overlap: zero contacts in the DFW box during the 22-day window. **But** +the sensor is correlating with HRRR cleanly. Hourly mean degradation +vs nearest-grid HRRR profile (n=37 hours of overlap — limited because the +local `hrrr_profiles` mirror only has 3 days in the window): + +| corr(fade_db, …) | value | +|---|---:| +| min_refractivity_gradient | −0.161 | +| surface_dewpoint_c | **+0.487** | +| pwat_mm | **+0.609** | +| t-td depression | +0.147 | +| surface_pressure_mb | **−0.605** | +| HPBL | +0.033 | + +Magnitudes are 4–5× anything the contact-vs-HRRR table produces at 24 GHz +because the sensor is sampled every 30 s instead of once per QSO. + +**Edit:** in Part 2c "Commercial-link inverse sensor" replace "deferred until +contest season" with: *"The 7-link cluster at 33.2N/−96.5W now has 22 days +of 30-second polling. Per-link rx_power_0 SD is 1.0–1.7 dB on af11x and +2.5–2.9 dB on af60. Hourly mean rx-degradation correlates with HRRR PWAT at ++0.61 and surface pressure at −0.61 — direction matches the QSO-derived +24 GHz weights. The sensor is wired and producing usable signal; QSO overlap +is gated on contest-season activity in the DFW zone."* + +### 5. Diurnal amplitude scales with frequency, but the magnitudes in algo.md are off + +Restricting to local hours with ≥30 contacts per band (so 0300-local doesn't +get a single 79 km outlier) produces a defensible amplitude index: + +```sql +WITH local_q AS ( + SELECT band::int AS band, distance_km::float AS dist, + EXTRACT(HOUR FROM qso_timestamp AT TIME ZONE 'America/Chicago')::int AS local_hr + FROM contacts + WHERE pos1 IS NOT NULL AND distance_km < 3000 AND flagged_invalid = false + AND band >= 222 AND qso_timestamp >= '2014-10-02' +), +hourly AS ( + SELECT band, local_hr, count(*) AS n, + PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY dist) AS med + FROM local_q GROUP BY band, local_hr HAVING count(*) >= 30 +) +SELECT band, count(*) hours, + MIN(med) lo, MAX(med) hi, + 100.0 * (MAX(med) - MIN(med)) / AVG(med) AS pct_amp +FROM hourly GROUP BY band; +``` + +| Band | hours | lo p50 (km) | hi p50 (km) | amplitude % | +|---|---:|---:|---:|---:| +| 222 MHz | 19 | 112.0 | 379.5 | **129.2** | +| 432 MHz | 19 | 95.0 | 355.0 | **148.4** | +| 902 MHz | 15 | 94.5 | 178.0 | 57.7 | +| 1.296 GHz | 17 | 80.0 | 176.0 | 71.2 | +| 2.304 GHz | 13 | 108.0 | 171.5 | 48.2 | +| 10 GHz | 23 | 141.2 | 217.1 | 40.4 | +| 24 GHz | 17 | 57.3 | 129.8 | 82.0 | +| 47 GHz | 14 | 31.8 | 91.3 | 101.1 | + +**Microwave bands behave as advertised** (10 → 24 → 47 GHz amplitude doubles +roughly each step). **VHF/UHF bands do not** — and this is the big gotcha: + +### 6. VHF/UHF diurnal "signal" is contest-schedule artifact + +The 222 MHz and 432 MHz amplitudes (129 %, 148 %) blow past every microwave +band. Physics doesn't predict this — at VHF the optical-path-loss and +diffraction terms are not strongly time-of-day-dependent. What is happening +is operator behavior: VHF contests run primarily evenings and weekends, +microwave contests run primarily mornings (rovers chase grids with +predictable schedules). The QSO database conflates "what propagation looks +like at 21 :00 local" with "what 21 :00-local QSOs look like, which is +overwhelmingly evening contest exchanges." + +**Edit:** Add a new caveat box to Finding 8: + +> **Selection bias warning.** Diurnal amplitude derived from QSO timestamps +> reflects *both* propagation diurnal cycles *and* operator-scheduling +> diurnal cycles. At VHF/UHF the second term dominates: the 130–150 % +> amplitude at 222/432 MHz is contest scheduling, not the atmosphere. Time- +> of-day weights for these bands should be **smaller** than what the QSO +> diurnal fit suggests, not larger. Until a clean atmospheric-only signal +> exists (e.g. continuous beacon monitoring, see project plan +> `beacon_monitoring`), keep VHF/UHF time-of-day weights at the global +> default, not at a per-band-fit value. + +### 7. Refractivity gradient is below noise at most bands + +Reproduced from `2026-04-25-recalibration.md`: + +| Band | rho_grad | +|---|---:| +| 222 MHz | +0.031 | +| 432 MHz | +0.003 | +| 902 MHz | −0.062 | +| 1.296 GHz | −0.079 | +| 2.304 GHz | −0.178 | +| 3.4 GHz | 0.000 | +| 5.76 GHz | −0.110 | +| 10 GHz | +0.027 | +| 24 GHz | +0.017 | +| 47 GHz | −0.008 | +| 75 GHz | +0.474 (n=83) | + +The 75 GHz row is n=83, dominated by Aug/Sep contest weeks where the gradient +just happens to track the same boundary-layer evolution that drives 75 GHz +contacts. It is not strong evidence for a per-band gradient term. + +**Edit:** Finding 5 already says "use continuous gradient." Tighten to: *"… +use continuous gradient* at 24 GHz only*; below 10 GHz the gradient signal +is at or below the 0.05 noise floor and adds nothing the dewpoint and PWAT +terms don't already capture. Set band-specific gradient weight to 0 outside +[10 GHz, 47 GHz]."* + +### 8. Mechanical updates + +The Part 2c sounding-ducting table was last refreshed at n=6,757. The +2026-04-25 recalibration ran on n=27,058. April ducting % is 25.5 %, not +31.3 % as currently shown. April PWAT is 12.7 mm, not 13.5 mm. Replace +verbatim from the recalibration report. + +### 9. Diurnal fade pattern of 11 GHz commercial links + +Bonus observation from the same 22-day SNMP corpus (af11x links, 27,820 +up-samples binned by local hour, fade ≥3 dB below per-link mean): + +| Local hour | n | n_fade≥3dB | pct | +|---|---:|---:|---:| +| 00 | 1,100 | 0 | 0.00 | +| 01 | 1,135 | 2 | 0.18 | +| 02 | 1,140 | 0 | 0.00 | +| 03 | 1,139 | 7 | 0.61 | +| 04 | 1,135 | 16 | **1.41** | +| 05 | 1,135 | 18 | **1.59** | +| 06 | 1,135 | 7 | 0.62 | +| 07 | 1,140 | 4 | 0.35 | +| 08 | 1,140 | 10 | 0.88 | +| 09 | 1,130 | 13 | 1.15 | +| 10 | 1,134 | 25 | **2.20** | +| 11 | 1,140 | 16 | 1.40 | +| 12 | 1,117 | 19 | 1.70 | +| 13–22 | flat | 3–8 | 0.18–0.65 | + +Two-peak structure: 04–05 local (radiation-fog / nocturnal-inversion ducting +breaks down) and 10–12 local (boundary-layer mixing onset). This is the +"morning fade window" that 11 GHz operators have always experienced +qualitatively; we now have a quantitative number for it. + +**Edit:** Add a short subsection under Part 2c "Commercial-link inverse +sensor" titled *Morning fade window* documenting the 4–12 local two-peak +pattern, with the suggestion that the diurnal scoring curve at 10–24 GHz be +*biased toward morning hours* rather than the symmetric "night peak" that +sub-mm-band physics suggests. + +--- + +## What did NOT change + +| Finding | Status | +|---|---| +| F1: humidity reverses at ~15 GHz | preserved — rho signs flip 5.76 GHz → 10 GHz → 24 GHz | +| F3: low pressure correlates with longer distances | preserved (rho 10 GHz −0.075, 24 GHz −0.20, 47 GHz −0.20) | +| F9: ducting peaks Jun–Jul | preserved (sounding ducting % Jun 69.5, Jul 67.5, Aug 55.2) | +| F10: CW advantage scales with freq | not re-tested in this run; corpus and methodology unchanged | +| F11: regional performance unmodeled | unchanged | +| 2014-10-02 NARR/HRRR cutover | unchanged; NARR pre-2014 corpus still 358 rows, deferred | +| 50/144 MHz unmodeled | unchanged; need external imports | + +## Carry-overs into ingestion / pipeline + +These are not algo.md edits but worth flagging from the same query session: + +1. **`hrrr_climatology` is empty in dev mirror** — `mix hrrr_climatology` has + not been run (or its output is not synced). Doesn't affect scoring today + but blocks the climatology-anomaly factor on the algo.md roadmap. +2. **`rtma_observations` is empty** — `RtmaFetchWorker` not enabled in dev. + Currently only relevant as fallback for contacts whose HRRR enrichment + misses the ±1 h window; per the recalibration report HRRR matches 68,062 + of ~82,000 contacts so RTMA fallback would close ~14k of the 36k pending + gap. +3. **`metar_5min_observations` is empty** — schema exists but no ingestor. + Either wire an IEM 5-minute ASOS fetcher or drop the schema. +4. **HRRR backlog: 36,276 contacts pending enrichment, 45,718 complete + (55.8 %).** Year breakdown via the new section in + `scripts/recalibrate_algo.py` will identify whether this is stalled + historical backfill or stuck live queue. + +## Reproducibility + +Every query in this report is self-contained SQL against `prop_dev` (which +mirrors `10.0.15.24:5432/prop`). The schema-touch points are: + +``` +contacts (n=81,994) +hrrr_profiles (n=81,293,173) +hrrr_native_profiles (n=11,472) +soundings (n=27,058) +nexrad_observations (n=7,286) +narr_profiles (n=358 — known low) +commercial_links (n=7) +commercial_samples (n=38,443; 2026-03-30 → 2026-04-20) +``` + +Re-running the auto-generated companion is `python3 scripts/recalibrate_algo.py`. diff --git a/lib/microwaveprop/propagation/scorer.ex b/lib/microwaveprop/propagation/scorer.ex index 2eb8030b..ed4dda0c 100644 --- a/lib/microwaveprop/propagation/scorer.ex +++ b/lib/microwaveprop/propagation/scorer.ex @@ -183,10 +183,15 @@ defmodule Microwaveprop.Propagation.Scorer do amplify trapping (surface inversion → steep gradient → ducting), deep boundary layers indicate convective mixing that disrupts ducts. - Empirical basis (Apr 13 2026 refresh, n=680 10 GHz contacts joined to - HRRR profiles): 230 km avg distance for HPBL <200 m vs 100 km for HPBL - ≥2000 m — a 2.3× difference monotonic across bins. The multiplier is - applied to *both* the threshold-matched score and the default fallback. + As of the 2026-04-25 algo revisions the HPBL boundary-layer multiplier + is retired — `bl_depth_m` is accepted (and the schema still stores it + for diagnostics) but it does not modify the score. See + `docs/algo-reports/2026-04-25-algo-revisions.md` Recommendation 2: on + the n=47,418 10 GHz HRRR-matched corpus rho_hpbl was +0.004, well below + the 0.05 noise floor; the bin tables agreed within ±5 % across + 200–2,000 m. The previously reported "shallow-BL → longer-distance" + effect was a small-sample artefact and disappeared once the matched + corpus grew past ~5,000 contacts. """ @spec score_refractivity(number() | nil, number() | nil, map()) :: integer() def score_refractivity(min_gradient, bl_depth_m, band_config) do @@ -194,17 +199,15 @@ defmodule Microwaveprop.Propagation.Scorer do end @doc """ - Scores refractivity with optional native-profile duct info. + Scores refractivity, ignoring `best_duct_band_ghz`. - `best_duct_band_ghz` comes from `hrrr_native_profiles` and represents the - highest frequency the cell's native-resolution duct can trap. When it's - ≥ the target band's frequency the base score is boosted 1.15× because - HRRR pressure-level gradients systematically under-read thin ducts the - native profile can resolve (see Part 2c Apr 13 2026 findings). - - A duct that only supports lower frequencies does NOT boost the score — - it's a signal that the gradient we have is *all there is* at the target - band. + Accepts the native-profile duct band so existing call sites + (`Scorer.score_grid_point/2`, `Recalibrator`, the Rust comparator) + continue to compile, but the previous 1.15× boost was retired in the + 2026-04-25 revisions: at 10 GHz on n=52,341 matched contacts, cells + whose native duct supported the band ran *shorter* (198 km, n=173) + than no-duct cells (211 km, n=44,658). Refractivity is now the + gradient-only score. """ @spec score_refractivity(number() | nil, number() | nil, number() | nil, map()) :: integer() def score_refractivity(min_gradient, bl_depth_m, best_duct_band_ghz, band_config) do @@ -212,82 +215,31 @@ defmodule Microwaveprop.Propagation.Scorer do end @doc """ - Scores refractivity with optional native-profile duct info, gated by - Bulk Richardson number. + Scores refractivity, ignoring `best_duct_band_ghz` and `bulk_richardson`. - `bulk_richardson` comes from `hrrr_native_profiles` and measures - dynamic stability (lower = more stable). Native duct cells average - 8.7-18.4 for ducting vs 38.3 for non-ducting, so a low - `best_duct_band_ghz` reading under turbulent conditions - (Richardson ≥ 25) is likely a duct that would be broken up by - mechanical mixing — in that case the 1.15× boost is *not* applied. - - `nil` Richardson falls back to the 4-arity behaviour (boost applies - whenever the duct band supports the target frequency). + The Bulk Richardson gate existed only to suppress the native-duct + boost during mechanically-mixed conditions. With the boost retired + (see 4-arity doc), Richardson is irrelevant — every input collapses + onto the gradient-only base score. Arity preserved so prod call + sites don't need an emergency rewrite. """ @spec score_refractivity(number() | nil, number() | nil, number() | nil, number() | nil, map()) :: integer() def score_refractivity(nil, _bl_depth_m, _best_duct_band_ghz, _bulk_richardson, _band_config), do: 50 - def score_refractivity(min_gradient, bl_depth_m, best_duct_band_ghz, bulk_richardson, %{ - freq_mhz: freq_mhz, - humidity_effect: effect - }) do + def score_refractivity(min_gradient, _bl_depth_m, _best_duct_band_ghz, _bulk_richardson, %{humidity_effect: effect}) do thresholds = BandConfig.refractivity_thresholds() - base = - case find_refractivity_threshold(min_gradient, thresholds, effect) do - {:ok, score} -> - score + case find_refractivity_threshold(min_gradient, thresholds, effect) do + {:ok, score} -> + score - :none -> - {beneficial_default, harmful_default} = BandConfig.refractivity_default() - if effect == :beneficial, do: beneficial_default, else: harmful_default - end - - base - |> apply_hpbl_multiplier(bl_depth_m) - |> apply_native_duct_boost(best_duct_band_ghz, bulk_richardson, freq_mhz) - end - - # HPBL multiplier — applied to the base refractivity score. Calibrated to the - # binned 10 GHz data: <200 m → 230 km avg, 200–500 → 186, 500–1500 → ~160, - # 1500–2000 → 137, ≥2000 → 100. nil HPBL keeps the score unchanged. - defp apply_hpbl_multiplier(base, nil), do: base - - defp apply_hpbl_multiplier(base, hpbl) when hpbl < 200, do: clamp_score(base * 1.10) - defp apply_hpbl_multiplier(base, hpbl) when hpbl < 500, do: clamp_score(base * 1.05) - defp apply_hpbl_multiplier(base, hpbl) when hpbl < 1500, do: base - defp apply_hpbl_multiplier(base, hpbl) when hpbl < 2000, do: clamp_score(base * 0.92) - defp apply_hpbl_multiplier(base, _hpbl), do: clamp_score(base * 0.78) - - # Native-profile duct boost — 1.15× when the cell's best duct supports the - # target band's frequency AND the Bulk Richardson number is in the stable - # regime (< 25). HRRR pressure-level gradients systematically under-read - # thin ducts, so when hrrr_native_profiles reports a duct band at or above - # the target frequency under stable conditions, the base gradient score is - # under-estimating the real channel. A duct reading with high Richardson - # is evidence of mechanical mixing that would shred the trapping layer — - # don't boost in that case. - @bulk_richardson_stable_max 25.0 - - defp apply_native_duct_boost(base, nil, _bulk_richardson, _freq_mhz), do: base - - defp apply_native_duct_boost(base, best_duct_band_ghz, bulk_richardson, freq_mhz) do - target_ghz = freq_mhz / 1_000 - - if best_duct_band_ghz >= target_ghz and stable_atmosphere?(bulk_richardson) do - clamp_score(base * 1.15) - else - base + :none -> + {beneficial_default, harmful_default} = BandConfig.refractivity_default() + if effect == :beneficial, do: beneficial_default, else: harmful_default end end - # nil Richardson is treated as stable (no information → don't penalise). - # Explicit Richardson ≥ 25 gates the boost. - defp stable_atmosphere?(nil), do: true - defp stable_atmosphere?(r) when is_number(r), do: r < @bulk_richardson_stable_max - @doc """ Inverse-sensor boost from commercial LOS link degradation. diff --git a/rust/prop_grid_rs/src/scorer.rs b/rust/prop_grid_rs/src/scorer.rs index 92808841..1feb415a 100644 --- a/rust/prop_grid_rs/src/scorer.rs +++ b/rust/prop_grid_rs/src/scorer.rs @@ -16,8 +16,6 @@ use crate::band_config::{ }; use crate::region; -const BULK_RICHARDSON_STABLE_MAX: f64 = 25.0; - /// Inputs to `composite_score`. All f64 everywhere to stay close to /// Erlang's single-float world. #[derive(Debug, Clone, Default)] @@ -209,13 +207,20 @@ pub fn score_td_depression(temp_f: f64, dewpoint_f: f64, band: &BandConfig) -> i } } -// ── Factor 4: refractivity gradient (+ HPBL multiplier + native duct boost) ── +// ── Factor 4: refractivity gradient (gradient-only as of 2026-04-25) ─ +// +// HPBL boundary-layer multiplier and native-profile 1.15× duct boost both +// retired — see `docs/algo-reports/2026-04-25-algo-revisions.md`. The +// `bl_depth_m`, `best_duct_band_ghz`, and `bulk_richardson` parameters are +// retained on the call signature so callers (and the Elixir parity test +// suite under `mix rust.golden`) keep the same shape, but they no longer +// influence the score. pub fn score_refractivity( min_gradient: Option, - bl_depth_m: Option, - best_duct_band_ghz: Option, - bulk_richardson: Option, + _bl_depth_m: Option, + _best_duct_band_ghz: Option, + _bulk_richardson: Option, band: &BandConfig, ) -> i32 { let Some(gradient) = min_gradient else { @@ -226,7 +231,7 @@ pub fn score_refractivity( .iter() .find(|(max, _, _)| gradient < *max as f64); - let base = match base_match { + match base_match { Some((_, b, h)) => match band.humidity_effect { HumidityEffect::Beneficial => *b, HumidityEffect::Harmful => *h, @@ -239,47 +244,6 @@ pub fn score_refractivity( h } } - }; - - let with_hpbl = apply_hpbl_multiplier(base, bl_depth_m); - apply_native_duct_boost( - with_hpbl, - best_duct_band_ghz, - bulk_richardson, - band.freq_mhz, - ) -} - -fn apply_hpbl_multiplier(base: i32, hpbl: Option) -> i32 { - let mul = match hpbl { - None => return base, - Some(v) if v < 200.0 => 1.10, - Some(v) if v < 500.0 => 1.05, - Some(v) if v < 1500.0 => 1.0, - Some(v) if v < 2000.0 => 0.92, - Some(_) => 0.78, - }; - clamp_score_f(base as f64 * mul) -} - -fn apply_native_duct_boost( - base: i32, - best_duct_band_ghz: Option, - bulk_richardson: Option, - freq_mhz: u32, -) -> i32 { - let Some(duct_band_ghz) = best_duct_band_ghz else { - return base; - }; - let target_ghz = freq_mhz as f64 / 1000.0; - let stable = match bulk_richardson { - None => true, - Some(r) => r < BULK_RICHARDSON_STABLE_MAX, - }; - if duct_band_ghz >= target_ghz && stable { - clamp_score_f(base as f64 * 1.15) - } else { - base } } @@ -694,49 +658,49 @@ mod tests { ); } + // HPBL multiplier and native-profile duct boost retired in the + // 2026-04-25 algo revisions (see `docs/algo-reports/`). Both helpers + // are gone; the parameter slots remain so existing callers don't need + // a coordinated rewrite. Tests below pin the post-retirement + // behaviour: every variant collapses onto the gradient-only score. + #[test] - fn shallow_bl_boosts_score() { + fn hpbl_no_longer_affects_score() { let shallow = score_refractivity(Some(-100.0), Some(150.0), None, None, b10g()); let baseline = score_refractivity(Some(-100.0), BASELINE_BL, None, None, b10g()); - assert!(shallow > baseline); - assert!(shallow <= 100); - } - - #[test] - fn deep_bl_penalises_score() { let deep = score_refractivity(Some(-100.0), Some(2200.0), None, None, b10g()); - let baseline = score_refractivity(Some(-100.0), BASELINE_BL, None, None, b10g()); - assert!(deep < baseline); - assert!(deep >= 0); + assert_eq!(shallow, baseline); + assert_eq!(deep, baseline); } #[test] - fn shallow_bl_also_lifts_default_fallback() { + fn hpbl_no_longer_affects_default_fallback() { let shallow = score_refractivity(Some(-30.0), Some(150.0), None, None, b10g()); let baseline = score_refractivity(Some(-30.0), BASELINE_BL, None, None, b10g()); - assert!(shallow > baseline); + let deep = score_refractivity(Some(-30.0), Some(2200.0), None, None, b10g()); + assert_eq!(shallow, baseline); + assert_eq!(deep, baseline); } #[test] - fn native_duct_boosts_matching_band() { - let boosted = score_refractivity(Some(-80.0), Some(800.0), Some(15.0), None, b10g()); + fn native_duct_no_longer_boosts_matching_band() { + let with_duct = score_refractivity(Some(-80.0), Some(800.0), Some(15.0), None, b10g()); let plain = score_refractivity(Some(-80.0), Some(800.0), None, None, b10g()); - assert!(boosted > plain); - assert!(boosted <= 100); + assert_eq!(with_duct, plain); } #[test] - fn native_duct_below_target_no_boost() { + fn native_duct_below_target_still_matches_plain() { let unchanged = score_refractivity(Some(-80.0), Some(800.0), Some(3.0), None, b10g()); let plain = score_refractivity(Some(-80.0), Some(800.0), None, None, b10g()); assert_eq!(unchanged, plain); } #[test] - fn richardson_gates_the_boost() { + fn richardson_no_longer_gates_anything() { let stable = score_refractivity(Some(-80.0), Some(800.0), Some(15.0), Some(12.0), b10g()); let plain = score_refractivity(Some(-80.0), Some(800.0), None, None, b10g()); - assert!(stable > plain); + assert_eq!(stable, plain); let turbulent = score_refractivity(Some(-80.0), Some(800.0), Some(15.0), Some(40.0), b10g()); diff --git a/test/microwaveprop/propagation/scorer_test.exs b/test/microwaveprop/propagation/scorer_test.exs index d1129a50..f2208efc 100644 --- a/test/microwaveprop/propagation/scorer_test.exs +++ b/test/microwaveprop/propagation/scorer_test.exs @@ -270,8 +270,13 @@ defmodule Microwaveprop.Propagation.ScorerTest do # ── score_refractivity/3 ───────────────────────────────────────── describe "score_refractivity/3" do - # 750 m sits in the [500, 1500) baseline-multiplier band so existing - # threshold assertions don't pick up the HPBL adjustment. + # As of the 2026-04-25 algo revisions the HPBL multiplier and the + # native-profile 1.15× duct boost are both retired (`docs/algo-reports/ + # 2026-04-25-algo-revisions.md` Recommendations 1 and 2). HPBL had + # rho ≤ 0.092 at every band and +0.004 at 10 GHz on n=47,418 — well + # below the noise floor. The function arities are kept so callers + # continue compiling, but they all collapse onto the gradient-only + # base score. @baseline_bl 750 test "strong ducting gradient returns high score for beneficial" do @@ -293,51 +298,46 @@ defmodule Microwaveprop.Propagation.ScorerTest do assert Scorer.score_refractivity(nil, @baseline_bl, @band_10g) == 50 end - test "nil bl_depth_m treated as baseline (no HPBL multiplier)" do - # A nil HPBL must not crash and must produce the unmultiplied score. - assert Scorer.score_refractivity(-160, nil, @band_10g) == 92 + test "nil bl_depth_m matches any other bl_depth_m (HPBL retired)" do + assert Scorer.score_refractivity(-160, nil, @band_10g) == + Scorer.score_refractivity(-160, @baseline_bl, @band_10g) end - test "shallow boundary layer (<200 m) boosts the gradient score" do - # Same -100 gradient: shallow BL should clearly out-score a baseline BL. + test "shallow boundary layer is no longer a boost" do shallow = Scorer.score_refractivity(-100, 150, @band_10g) baseline = Scorer.score_refractivity(-100, @baseline_bl, @band_10g) - assert shallow > baseline - assert shallow <= 100 + assert shallow == baseline end - test "deep boundary layer (>=2000 m) penalises the gradient score" do - # Same -100 gradient: deep BL drags the score down. + test "deep boundary layer is no longer a penalty" do deep = Scorer.score_refractivity(-100, 2200, @band_10g) baseline = Scorer.score_refractivity(-100, @baseline_bl, @band_10g) - assert deep < baseline - assert deep >= 0 + assert deep == baseline end - test "shallow BL also lifts the weak-gradient default" do - # The HPBL multiplier applies to the default fallback too — the binned - # data shows shallow-BL paths run ~230 km even when HRRR gradient is weak. + test "HPBL has no effect on the default-fallback branch either" do shallow_default = Scorer.score_refractivity(-30, 150, @band_10g) baseline_default = Scorer.score_refractivity(-30, @baseline_bl, @band_10g) - assert shallow_default > baseline_default + deep_default = Scorer.score_refractivity(-30, 2200, @band_10g) + assert shallow_default == baseline_default + assert deep_default == baseline_default end end describe "score_refractivity/4 with native-profile duct support" do - # New 4-arity variant: adds best_duct_band_ghz from hrrr_native_profiles so - # cells with a thin native duct that supports the target band get a boost - # the HRRR pressure-level gradient alone would miss. Cells where the native - # duct only supports sub-band frequencies do NOT get a boost. + # Native-duct 1.15× boost retired — see Recommendation 1 in + # `docs/algo-reports/2026-04-25-algo-revisions.md`. At 10 GHz on + # n=52,341, contacts where the native duct supports the band ran + # *shorter* (198 km) than no-duct contacts (211 km). Arity kept, + # behaviour collapses onto the 3-arity gradient-only score. - test "native duct supporting the target band boosts the score" do - boosted = Scorer.score_refractivity(-80, 800, 15.0, @band_10g) + test "native duct supporting the target band no longer boosts" do + with_duct = Scorer.score_refractivity(-80, 800, 15.0, @band_10g) plain = Scorer.score_refractivity(-80, 800, nil, @band_10g) - assert boosted > plain - assert boosted <= 100 + assert with_duct == plain end - test "native duct below the target band does NOT boost" do - # A 3 GHz duct is useless for 10 GHz. + test "native duct below the target band still produces the same plain score" do unchanged = Scorer.score_refractivity(-80, 800, 3.0, @band_10g) plain = Scorer.score_refractivity(-80, 800, nil, @band_10g) assert unchanged == plain @@ -352,42 +352,42 @@ defmodule Microwaveprop.Propagation.ScorerTest do assert Scorer.score_refractivity(nil, 600, 15.0, @band_10g) == 50 end - test "24 GHz is boosted only when native duct supports ≥24 GHz" do + test "24 GHz is no longer boosted by any native duct band" do weak = Scorer.score_refractivity(-80, 800, 15.0, @band_24g) strong = Scorer.score_refractivity(-80, 800, 30.0, @band_24g) plain = Scorer.score_refractivity(-80, 800, nil, @band_24g) assert weak == plain - assert strong > plain + assert strong == plain end end describe "score_refractivity/5 with bulk-Richardson gating" do - # 5-arity variant: the native-duct boost only applies when Bulk - # Richardson number is in the stable regime (< 25). Native duct - # cells average 8.7–18.4 for ducting vs 38.3 for non-ducting, so a - # low best_duct_band_ghz reading with high Richardson is likely a - # duct that would be shredded by mechanical mixing — we should not - # boost it (Part 2c of algo.md). + # The Richardson gate existed only to suppress the native-duct + # boost during mechanically-mixed conditions. Now that the boost + # itself is retired, Richardson is irrelevant — every input maps + # to the same plain gradient-only score. Arity preserved so prod + # call sites in propagation/recalibrator and the Rust comparator + # don't need an emergency rewrite. - test "stable Richardson (<25) + matching duct band → boost applies" do - boosted = Scorer.score_refractivity(-80, 800, 15.0, 12.0, @band_10g) + test "stable Richardson + matching duct band ≡ plain" do + stable = Scorer.score_refractivity(-80, 800, 15.0, 12.0, @band_10g) plain = Scorer.score_refractivity(-80, 800, nil, nil, @band_10g) - assert boosted > plain + assert stable == plain end - test "turbulent Richardson (≥25) + matching duct band → boost suppressed" do + test "turbulent Richardson + matching duct band ≡ plain" do gated = Scorer.score_refractivity(-80, 800, 15.0, 40.0, @band_10g) plain = Scorer.score_refractivity(-80, 800, nil, nil, @band_10g) assert gated == plain end - test "nil Richardson preserves 4-arity behaviour (boost still applies)" do + test "nil Richardson ≡ plain" do with_nil_r = Scorer.score_refractivity(-80, 800, 15.0, nil, @band_10g) four_arity = Scorer.score_refractivity(-80, 800, 15.0, @band_10g) assert with_nil_r == four_arity end - test "boundary case: Richardson at exactly 25 suppresses the boost" do + test "Richardson at the old 25 boundary ≡ plain" do gated = Scorer.score_refractivity(-80, 800, 15.0, 25.0, @band_10g) plain = Scorer.score_refractivity(-80, 800, nil, nil, @band_10g) assert gated == plain