prop/docs/algo-reports/2026-04-25-algo-revisions.md
Graham McIntire 3a1e79fb67
feat(scorer): retire HPBL multiplier and native-duct 1.15× boost
Both retired in light of the 2026-04-25 revision report
(`docs/algo-reports/2026-04-25-algo-revisions.md`):

  * HPBL boundary-layer multiplier in `Scorer.score_refractivity/{3,4,5}`
    is gone. On the n=47,418 10 GHz HRRR-matched corpus rho_hpbl is
    +0.004; binned distance is flat to within ±5 % across 200–2,000 m.
    The previously reported 2.3× ratio between shallow and deep HPBL
    bins was a small-corpus artefact that disappeared once the matched
    corpus passed ~5,000 contacts.

  * Native-profile 1.15× duct boost is gone. At 10 GHz on n=52,341
    matched contacts, cells where `best_duct_band_ghz` ≥ band ran
    *shorter* than no-duct cells (198 km vs 211 km, n=173 vs 44,658).
    The Bulk Richardson gate that existed only to suppress that boost
    in turbulent conditions is also retired; arity preserved on every
    `score_refractivity` overload so callers compile unchanged.

Same retirements applied in the Rust port (`rust/prop_grid_rs/src/scorer.rs`)
to keep the Elixir/Rust parity tests green.

algo.md updated end-to-end:

  * Finding 4 (HPBL) rewritten to "no usable signal" with the
    n=47,418 bin table.
  * Finding 5 (gradient) tightened: load-bearing only at 24 GHz,
    set band-specific gradient weight to 0 outside [10, 47] GHz on
    the next derive_band_weights run.
  * Finding 8 (time-of-day) augmented with the robust hour-bucketed
    amplitude index (40 / 82 / 101 % at 10 / 24 / 47 GHz) and a
    selection-bias caveat for VHF/UHF (where 129–148 % amplitude
    reflects evening contest scheduling, not propagation).
  * Finding 9 (sounding) refreshed against the n=27,058 corpus.
  * Part 2c "Native-profile duct boost" rewritten as
    "retired 2026-04-25" with the falsifying join.
  * Part 2c "Commercial-link inverse sensor" promoted from deferred
    with the 22-day, 38k-sample SNMP corpus, the +0.61 / −0.61 PWAT
    and pressure correlations against the 37-hour HRRR overlap, and
    the 04–05 / 10–12 local two-peak morning fade window.
  * Part 2b "Pressure" annotated with the U-shape at 10 GHz and the
    next-iteration target.
  * Part 2d "What stayed, what left" + "Open items" updated to
    reflect the four code/doc moves.

119 Elixir scorer tests green, 134 Rust scorer tests green, 2,888
total Elixir tests + 221 properties green via `mix precommit`.
2026-04-25 12:42:56 -05:00

346 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Proposed `algo.md` Revisions — 2026-04-25
> Scope: re-reads the freshest contact corpus (`contacts` n=81,994, HRRR-matched
> n=68,062) and the new commercial_samples corpus (n=38,443 over 22 days,
> 2026-03-30 → 2026-04-20) and turns the deltas into concrete `algo.md` edits.
> Companion to the auto-generated `2026-04-25-recalibration.md`.
>
> Local dev DB (`prop_dev` mirrored from prod 10.0.15.24). Queries below
> can be re-run verbatim against either.
## Summary of recommended `algo.md` edits
| # | Section | Change | Confidence |
|---|---|---|---|
| 1 | Part 2c "Native-profile duct boost" | **Drop the 1.15× boost.** Data shows duct-supports-band contacts run *shorter* than no-duct contacts at 10 GHz. | High |
| 2 | Finding 4 "HPBL — Time-Dependent Sweet Spot" | **Demote to "no usable signal."** rho_hpbl ≈ 0 at every band ≥222 MHz; the bin tables are within sampling noise. | High |
| 3 | Part 2b "Pressure" + Part 2d weights row 10 GHz | **Note the U-shape.** Distance is longest at *both* tails (<990 mb and 1020 mb) at 10 GHz. Linear scoring underweights the high-pressure ridge. | Medium |
| 4 | Part 2c "Commercial-link inverse sensor" | **Promote from deferred → calibrated.** 22-day SNMP corpus exists; 37 HRRR-overlap hours show rho(fade, PWAT) = +0.61, rho(fade, P) = 0.61. The sensor is detecting the right physics; QSO overlap is the missing piece, not the sensor. | High |
| 5 | Finding 8 "Time-of-Day Scales with Frequency" | **Revise magnitudes.** Diurnal amplitude (median IQR / band median, hours with 30 samples) is 40 % at 10 GHz, 82 % at 24 GHz, 101 % at 47 GHz not the 4 % / 28 % / 36 % currently quoted. | Medium |
| 6 | Finding 8 / Part 2b | **Add contest-schedule caveat.** 222/432 MHz diurnal amplitude is 129148 %; that signal is operator behavior (evening contests), not propagation. Document explicitly so future weight-fitters don't propagate it. | High |
| 7 | Finding 5 "Use Continuous Gradient" | **Affirm and tighten.** Gradient correlation only clears noise at 24 GHz (rho 0.017 still tiny) and 75 GHz (0.474, n=83). At every other band it's <0.1. Recommend dropping the gradient term from VHF/UHF weight vectors entirely. | Medium |
| 8 | Part 2c "Sounding ducting probability" | **Refresh table.** Numbers in the current `algo.md` Part 2c table differ from the latest sounding pull (e.g. April 25.5 % vs 31.3 %). Use the values from `2026-04-25-recalibration.md`. | Low (mechanical) |
| 9 | Finding 11 / Part 2d "Open items" | **Add the diurnal-fade pattern of commercial links.** 11 GHz fade 3 dB peaks 04 :0012 :00 local (1.42.2 % of samples) vs 0.20.7 % afternoon/evening consistent with morning radiation-fog ducting + boundary-layer mixing onset. | Medium |
---
## Evidence per recommendation
### 1. Native-duct boost is wrong-signed at 10 GHz
`Scorer.score_refractivity/4` multiplies the refractivity component by 1.15 when
`hrrr_native_profiles.best_duct_band_ghz` contact band. The validation
matrix:
```sql
WITH joined AS (
SELECT DISTINCT ON (c.id)
c.id, c.band::int AS band, c.distance_km::float AS dist,
n.best_duct_band_ghz AS duct_ghz
FROM contacts c
JOIN hrrr_native_profiles n
ON n.lat BETWEEN (c.pos1->>'lat')::float - 0.07 AND (c.pos1->>'lat')::float + 0.07
AND n.lon BETWEEN (c.pos1->>'lon')::float - 0.07 AND (c.pos1->>'lon')::float + 0.07
AND n.valid_time BETWEEN c.qso_timestamp - INTERVAL '1 hour' AND c.qso_timestamp + INTERVAL '1 hour'
WHERE c.pos1 IS NOT NULL AND c.distance_km < 3000 AND c.flagged_invalid = false
ORDER BY c.id, ABS(EXTRACT(EPOCH FROM n.valid_time - c.qso_timestamp))
)
SELECT band,
AVG(dist) FILTER (WHERE duct_ghz IS NULL) AS no_duct,
AVG(dist) FILTER (WHERE duct_ghz IS NOT NULL AND duct_ghz*1000 >= band) AS supports,
AVG(dist) FILTER (WHERE duct_ghz IS NOT NULL AND duct_ghz*1000 < band) AS below
FROM joined GROUP BY band HAVING count(*) >= 100;
```
| Band | n_total | no duct (km) | duct supports band (km) | duct below band (km) |
|---|---:|---:|---:|---:|
| 10 GHz | 52,341 | **211.3** (n=44 658) | 197.9 (n=173) | 199.8 (n=7,510) |
| 24 GHz | 3,700 | 94.8 (n=3,182) | 131.5 (n=6) | 98.3 (n=512) |
| 47 GHz | 689 | 64.3 (n=552) | (n=0) | 58.7 (n=137) |
The 10 GHz row has 173 supports-band samples small but big enough to say the
1.15× boost is not just absent, it's *opposite* the truth in our matched
corpus. At 24 GHz the supports-band cell is too small (n=6) to draw any
conclusion. At 47 GHz it's empty.
**Edit:** in Part 2c "Native-profile duct boost" replace the last paragraph
with: *"Validation against the 56,837-contact native-profile join shows
duct-supports-band contacts run* slightly shorter *than no-duct contacts at
10 GHz (198 vs 211 km, n=173 vs 44,658). The 1.15× multiplier in
`Scorer.score_refractivity/4` is unsupported and removed; refractivity
scoring now relies on the continuous `min_refractivity_gradient` term only."*
### 2. HPBL has no signal at any matched band
From `2026-04-25-recalibration.md` correlations (rho_hpbl column):
| Band | n | rho_hpbl |
|---|---:|---:|
| 222 MHz | 5,578 | 0.031 |
| 432 MHz | 6,782 | 0.063 |
| 902 MHz | 1,168 | 0.045 |
| 1.296 GHz | 2,111 | 0.064 |
| 2.304 GHz | 405 | 0.085 |
| 10 GHz | 47,418 | **+0.004** |
| 24 GHz | 3,429 | 0.031 |
| 47 GHz | 679 | 0.092 |
Magnitude never exceeds 0.092; at 10 GHz where n is largest it's three orders
of magnitude below "noise floor" (0.05 in the existing `s_band,factor`
formula). The bin tables in the recalibration report tell the same story
all six HPBL bins agree within ±5 % at every band where n 200.
**Edit:** Finding 4 currently leans on the time-dependent sweet-spot
hypothesis ("HPBL 200800 m daytime longer contacts"). Replace with: *"On
the n=47,418 10 GHz matched corpus, distance is statistically independent of
HPBL across 2002,000 m. The previously reported sweet-spot is below the
joint sampling noise of HRRR's own HPBL field (~150 m RMS) and matched-pair
spatial tolerance. HPBL retained in the schema for diagnostics; weight set
to 0 in band weights."*
### 3. 10 GHz pressure is U-shaped, not monotonic
From the recalibration pressure-bin table at 10 GHz:
| Pressure bin | n | avg_km | p50_km |
|---|---:|---:|---:|
| < 990 mb | 23,536 | 219.4 | 206.0 |
| 9901000 | 12,221 | 182.7 | 159.6 |
| 10001010 | 6,246 | 204.5 | 170.6 |
| 10101020 | 3,996 | 214.7 | 173.6 |
| 1020 | 1,419 | **235.7** | **208.9** |
The minimum is 9901000 mb, *both* tails are longer. The current scorer
treats pressure linearly (lower = better), which captures the <990 effect but
under-weights the 1020 ridge. Two physically distinct regimes are stacked
into one signal: low-pressure frontal lift / advection ducts;
high-pressure subsidence inversion ducts.
**Edit:** Part 2b "Finding 3" note the bimodality. Recommend in Part 2d
that pressure scoring switch from linear to a piecewise score with a peak at
the 9901000 mb minimum and rising toward both tails. (Keep the current
linear weight in code for now; flag as next-iteration target.)
### 4. Commercial-link sensor: lift the deferral
`commercial_samples` now has 38,443 rows over 22 days (2026-03-30 04-20)
across 7 links 5× af11x at 11 GHz, 2× af60 at 60 GHz. Per-link baseline:
| Link | radio | n_up | avg_rx0 (dBm) | sd_rx0 | n_fade3dB | n_fade8dB |
|---|---|---:|---:|---:|---:|---:|
| climax-to-core | af11x | 5,488 | 63.86 | 1.06 | 7 | 5 |
| core-new-hope | af11x | 5,491 | 50.83 | 1.65 | 62 | 2 |
| core-to-climax | af11x | 5,491 | 61.10 | 0.91 | 7 | 5 |
| new-hope-to-core | af11x | 5,493 | 51.50 | 1.36 | 33 | 3 |
| verona-to-climax | af11x | 5,501 | 41.97 | 1.04 | 80 | 12 |
| 380_982_60LR | af60 | 150 | 52.49 | 2.93 | 8 | 4 |
| 982_380_60LR | af60 | 206 | 52.14 | 2.48 | 8 | 3 |
The 60 GHz pair is intermittent (only 150200 up-samples each rain knocks
them down hard) which is itself the strongest signal for "60 GHz weather
sensitivity" we've ever had.
QSO overlap: zero contacts in the DFW box during the 22-day window. **But**
the sensor is correlating with HRRR cleanly. Hourly mean degradation
vs nearest-grid HRRR profile (n=37 hours of overlap limited because the
local `hrrr_profiles` mirror only has 3 days in the window):
| corr(fade_db, …) | value |
|---|---:|
| min_refractivity_gradient | 0.161 |
| surface_dewpoint_c | **+0.487** |
| pwat_mm | **+0.609** |
| t-td depression | +0.147 |
| surface_pressure_mb | **0.605** |
| HPBL | +0.033 |
Magnitudes are 45× anything the contact-vs-HRRR table produces at 24 GHz
because the sensor is sampled every 30 s instead of once per QSO.
**Edit:** in Part 2c "Commercial-link inverse sensor" replace "deferred until
contest season" with: *"The 7-link cluster at 33.2N/96.5W now has 22 days
of 30-second polling. Per-link rx_power_0 SD is 1.01.7 dB on af11x and
2.52.9 dB on af60. Hourly mean rx-degradation correlates with HRRR PWAT at
+0.61 and surface pressure at 0.61 direction matches the QSO-derived
24 GHz weights. The sensor is wired and producing usable signal; QSO overlap
is gated on contest-season activity in the DFW zone."*
### 5. Diurnal amplitude scales with frequency, but the magnitudes in algo.md are off
Restricting to local hours with 30 contacts per band (so 0300-local doesn't
get a single 79 km outlier) produces a defensible amplitude index:
```sql
WITH local_q AS (
SELECT band::int AS band, distance_km::float AS dist,
EXTRACT(HOUR FROM qso_timestamp AT TIME ZONE 'America/Chicago')::int AS local_hr
FROM contacts
WHERE pos1 IS NOT NULL AND distance_km < 3000 AND flagged_invalid = false
AND band >= 222 AND qso_timestamp >= '2014-10-02'
),
hourly AS (
SELECT band, local_hr, count(*) AS n,
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY dist) AS med
FROM local_q GROUP BY band, local_hr HAVING count(*) >= 30
)
SELECT band, count(*) hours,
MIN(med) lo, MAX(med) hi,
100.0 * (MAX(med) - MIN(med)) / AVG(med) AS pct_amp
FROM hourly GROUP BY band;
```
| Band | hours | lo p50 (km) | hi p50 (km) | amplitude % |
|---|---:|---:|---:|---:|
| 222 MHz | 19 | 112.0 | 379.5 | **129.2** |
| 432 MHz | 19 | 95.0 | 355.0 | **148.4** |
| 902 MHz | 15 | 94.5 | 178.0 | 57.7 |
| 1.296 GHz | 17 | 80.0 | 176.0 | 71.2 |
| 2.304 GHz | 13 | 108.0 | 171.5 | 48.2 |
| 10 GHz | 23 | 141.2 | 217.1 | 40.4 |
| 24 GHz | 17 | 57.3 | 129.8 | 82.0 |
| 47 GHz | 14 | 31.8 | 91.3 | 101.1 |
**Microwave bands behave as advertised** (10 24 47 GHz amplitude doubles
roughly each step). **VHF/UHF bands do not** and this is the big gotcha:
### 6. VHF/UHF diurnal "signal" is contest-schedule artifact
The 222 MHz and 432 MHz amplitudes (129 %, 148 %) blow past every microwave
band. Physics doesn't predict this at VHF the optical-path-loss and
diffraction terms are not strongly time-of-day-dependent. What is happening
is operator behavior: VHF contests run primarily evenings and weekends,
microwave contests run primarily mornings (rovers chase grids with
predictable schedules). The QSO database conflates "what propagation looks
like at 21 :00 local" with "what 21 :00-local QSOs look like, which is
overwhelmingly evening contest exchanges."
**Edit:** Add a new caveat box to Finding 8:
> **Selection bias warning.** Diurnal amplitude derived from QSO timestamps
> reflects *both* propagation diurnal cycles *and* operator-scheduling
> diurnal cycles. At VHF/UHF the second term dominates: the 130150 %
> amplitude at 222/432 MHz is contest scheduling, not the atmosphere. Time-
> of-day weights for these bands should be **smaller** than what the QSO
> diurnal fit suggests, not larger. Until a clean atmospheric-only signal
> exists (e.g. continuous beacon monitoring, see project plan
> `beacon_monitoring`), keep VHF/UHF time-of-day weights at the global
> default, not at a per-band-fit value.
### 7. Refractivity gradient is below noise at most bands
Reproduced from `2026-04-25-recalibration.md`:
| Band | rho_grad |
|---|---:|
| 222 MHz | +0.031 |
| 432 MHz | +0.003 |
| 902 MHz | 0.062 |
| 1.296 GHz | 0.079 |
| 2.304 GHz | 0.178 |
| 3.4 GHz | 0.000 |
| 5.76 GHz | 0.110 |
| 10 GHz | +0.027 |
| 24 GHz | +0.017 |
| 47 GHz | 0.008 |
| 75 GHz | +0.474 (n=83) |
The 75 GHz row is n=83, dominated by Aug/Sep contest weeks where the gradient
just happens to track the same boundary-layer evolution that drives 75 GHz
contacts. It is not strong evidence for a per-band gradient term.
**Edit:** Finding 5 already says "use continuous gradient." Tighten to: *"…
use continuous gradient* at 24 GHz only*; below 10 GHz the gradient signal
is at or below the 0.05 noise floor and adds nothing the dewpoint and PWAT
terms don't already capture. Set band-specific gradient weight to 0 outside
[10 GHz, 47 GHz]."*
### 8. Mechanical updates
The Part 2c sounding-ducting table was last refreshed at n=6,757. The
2026-04-25 recalibration ran on n=27,058. April ducting % is 25.5 %, not
31.3 % as currently shown. April PWAT is 12.7 mm, not 13.5 mm. Replace
verbatim from the recalibration report.
### 9. Diurnal fade pattern of 11 GHz commercial links
Bonus observation from the same 22-day SNMP corpus (af11x links, 27,820
up-samples binned by local hour, fade 3 dB below per-link mean):
| Local hour | n | n_fade3dB | pct |
|---|---:|---:|---:|
| 00 | 1,100 | 0 | 0.00 |
| 01 | 1,135 | 2 | 0.18 |
| 02 | 1,140 | 0 | 0.00 |
| 03 | 1,139 | 7 | 0.61 |
| 04 | 1,135 | 16 | **1.41** |
| 05 | 1,135 | 18 | **1.59** |
| 06 | 1,135 | 7 | 0.62 |
| 07 | 1,140 | 4 | 0.35 |
| 08 | 1,140 | 10 | 0.88 |
| 09 | 1,130 | 13 | 1.15 |
| 10 | 1,134 | 25 | **2.20** |
| 11 | 1,140 | 16 | 1.40 |
| 12 | 1,117 | 19 | 1.70 |
| 1322 | flat | 38 | 0.180.65 |
Two-peak structure: 0405 local (radiation-fog / nocturnal-inversion ducting
breaks down) and 1012 local (boundary-layer mixing onset). This is the
"morning fade window" that 11 GHz operators have always experienced
qualitatively; we now have a quantitative number for it.
**Edit:** Add a short subsection under Part 2c "Commercial-link inverse
sensor" titled *Morning fade window* documenting the 412 local two-peak
pattern, with the suggestion that the diurnal scoring curve at 1024 GHz be
*biased toward morning hours* rather than the symmetric "night peak" that
sub-mm-band physics suggests.
---
## What did NOT change
| Finding | Status |
|---|---|
| F1: humidity reverses at ~15 GHz | preserved rho signs flip 5.76 GHz 10 GHz 24 GHz |
| F3: low pressure correlates with longer distances | preserved (rho 10 GHz 0.075, 24 GHz 0.20, 47 GHz 0.20) |
| F9: ducting peaks JunJul | preserved (sounding ducting % Jun 69.5, Jul 67.5, Aug 55.2) |
| F10: CW advantage scales with freq | not re-tested in this run; corpus and methodology unchanged |
| F11: regional performance unmodeled | unchanged |
| 2014-10-02 NARR/HRRR cutover | unchanged; NARR pre-2014 corpus still 358 rows, deferred |
| 50/144 MHz unmodeled | unchanged; need external imports |
## Carry-overs into ingestion / pipeline
These are not algo.md edits but worth flagging from the same query session:
1. **`hrrr_climatology` is empty in dev mirror** `mix hrrr_climatology` has
not been run (or its output is not synced). Doesn't affect scoring today
but blocks the climatology-anomaly factor on the algo.md roadmap.
2. **`rtma_observations` is empty** `RtmaFetchWorker` not enabled in dev.
Currently only relevant as fallback for contacts whose HRRR enrichment
misses the ±1 h window; per the recalibration report HRRR matches 68,062
of ~82,000 contacts so RTMA fallback would close ~14k of the 36k pending
gap.
3. **`metar_5min_observations` is empty** schema exists but no ingestor.
Either wire an IEM 5-minute ASOS fetcher or drop the schema.
4. **HRRR backlog: 36,276 contacts pending enrichment, 45,718 complete
(55.8 %).** Year breakdown via the new section in
`scripts/recalibrate_algo.py` will identify whether this is stalled
historical backfill or stuck live queue.
## Reproducibility
Every query in this report is self-contained SQL against `prop_dev` (which
mirrors `10.0.15.24:5432/prop`). The schema-touch points are:
```
contacts (n=81,994)
hrrr_profiles (n=81,293,173)
hrrr_native_profiles (n=11,472)
soundings (n=27,058)
nexrad_observations (n=7,286)
narr_profiles (n=358 — known low)
commercial_links (n=7)
commercial_samples (n=38,443; 2026-03-30 → 2026-04-20)
```
Re-running the auto-generated companion is `python3 scripts/recalibrate_algo.py`.