prop/docs/algo-reports/2026-04-25-algo-revisions.md

Proposed algo.md Revisions — 2026-04-25

Scope: re-reads the freshest contact corpus (contacts n=81,994, HRRR-matched n=68,062) and the new commercial_samples corpus (n=38,443 over 22 days, 2026-03-30 → 2026-04-20) and turns the deltas into concrete algo.md edits. Companion to the auto-generated 2026-04-25-recalibration.md.

Local dev DB (prop_dev mirrored from prod 10.0.15.24). Queries below can be re-run verbatim against either.

Summary of recommended algo.md edits

#	Section	Change	Confidence
1	Part 2c "Native-profile duct boost"	Drop the 1.15× boost. Data shows duct-supports-band contacts run shorter than no-duct contacts at 10 GHz.	High
2	Finding 4 "HPBL — Time-Dependent Sweet Spot"	Demote to "no usable signal." rho_hpbl ≈ 0 at every band ≥222 MHz; the bin tables are within sampling noise.	High
3	Part 2b "Pressure" + Part 2d weights row 10 GHz	Note the U-shape. Distance is longest at both tails (<990 mb and ≥1020 mb) at 10 GHz. Linear scoring underweights the high-pressure ridge.	Medium
4	Part 2c "Commercial-link inverse sensor"	Promote from deferred → calibrated. 22-day SNMP corpus exists; 37 HRRR-overlap hours show rho(fade, PWAT) = +0.61, rho(fade, P) = −0.61. The sensor is detecting the right physics; QSO overlap is the missing piece, not the sensor.	High
5	Finding 8 "Time-of-Day Scales with Frequency"	Revise magnitudes. Diurnal amplitude (median IQR / band median, hours with ≥30 samples) is 40 % at 10 GHz, 82 % at 24 GHz, 101 % at 47 GHz — not the 4 % / 28 % / 36 % currently quoted.	Medium
6	Finding 8 / Part 2b	Add contest-schedule caveat. 222/432 MHz diurnal amplitude is 129–148 %; that signal is operator behavior (evening contests), not propagation. Document explicitly so future weight-fitters don't propagate it.	High
7	Finding 5 "Use Continuous Gradient"	Affirm and tighten. Gradient correlation only clears noise at 24 GHz (rho ≈ 0.017 — still tiny) and 75 GHz (0.474, n=83). At every other band it's <0.1. Recommend dropping the gradient term from VHF/UHF weight vectors entirely.	Medium
8	Part 2c "Sounding ducting probability"	Refresh table. Numbers in the current algo.md Part 2c table differ from the latest sounding pull (e.g. April 25.5 % vs 31.3 %). Use the values from 2026-04-25-recalibration.md.	Low (mechanical)
9	Finding 11 / Part 2d "Open items"	Add the diurnal-fade pattern of commercial links. 11 GHz fade ≥3 dB peaks 04 :00–12 :00 local (1.4–2.2 % of samples) vs 0.2–0.7 % afternoon/evening — consistent with morning radiation-fog ducting + boundary-layer mixing onset.	Medium

---

Evidence per recommendation

1. Native-duct boost is wrong-signed at 10 GHz

Scorer.score_refractivity/4 multiplies the refractivity component by 1.15 when hrrr_native_profiles.best_duct_band_ghz ≥ contact band. The validation matrix:

WITH joined AS (
  SELECT DISTINCT ON (c.id)
    c.id, c.band::int AS band, c.distance_km::float AS dist,
    n.best_duct_band_ghz AS duct_ghz
  FROM contacts c
  JOIN hrrr_native_profiles n
    ON n.lat BETWEEN (c.pos1->>'lat')::float - 0.07 AND (c.pos1->>'lat')::float + 0.07
   AND n.lon BETWEEN (c.pos1->>'lon')::float - 0.07 AND (c.pos1->>'lon')::float + 0.07
   AND n.valid_time BETWEEN c.qso_timestamp - INTERVAL '1 hour' AND c.qso_timestamp + INTERVAL '1 hour'
  WHERE c.pos1 IS NOT NULL AND c.distance_km < 3000 AND c.flagged_invalid = false
  ORDER BY c.id, ABS(EXTRACT(EPOCH FROM n.valid_time - c.qso_timestamp))
)
SELECT band,
       AVG(dist) FILTER (WHERE duct_ghz IS NULL)                              AS no_duct,
       AVG(dist) FILTER (WHERE duct_ghz IS NOT NULL AND duct_ghz*1000 >= band) AS supports,
       AVG(dist) FILTER (WHERE duct_ghz IS NOT NULL AND duct_ghz*1000 <  band) AS below
FROM joined GROUP BY band HAVING count(*) >= 100;

Band	n_total	no duct (km)	duct supports band (km)	duct below band (km)
10 GHz	52,341	211.3 (n=44 658)	197.9 (n=173)	199.8 (n=7,510)
24 GHz	3,700	94.8 (n=3,182)	131.5 (n=6)	98.3 (n=512)
47 GHz	689	64.3 (n=552)	– (n=0)	58.7 (n=137)

The 10 GHz row has 173 supports-band samples — small but big enough to say the 1.15× boost is not just absent, it's opposite the truth in our matched corpus. At 24 GHz the supports-band cell is too small (n=6) to draw any conclusion. At 47 GHz it's empty.

Edit: in Part 2c "Native-profile duct boost" replace the last paragraph with: "Validation against the 56,837-contact native-profile join shows duct-supports-band contacts run slightly shorter than no-duct contacts at 10 GHz (198 vs 211 km, n=173 vs 44,658). The 1.15× multiplier in Scorer.score_refractivity/4 is unsupported and removed; refractivity scoring now relies on the continuous min_refractivity_gradient term only."

2. HPBL has no signal at any matched band

From 2026-04-25-recalibration.md correlations (rho_hpbl column):

Band	n	rho_hpbl
222 MHz	5,578	−0.031
432 MHz	6,782	−0.063
902 MHz	1,168	−0.045
1.296 GHz	2,111	−0.064
2.304 GHz	405	−0.085
10 GHz	47,418	+0.004
24 GHz	3,429	−0.031
47 GHz	679	−0.092

Magnitude never exceeds 0.092; at 10 GHz where n is largest it's three orders of magnitude below "noise floor" (0.05 in the existing s_band,factor formula). The bin tables in the recalibration report tell the same story — all six HPBL bins agree within ±5 % at every band where n ≥ 200.

Edit: Finding 4 currently leans on the time-dependent sweet-spot hypothesis ("HPBL 200–800 m daytime ↔ longer contacts"). Replace with: "On the n=47,418 10 GHz matched corpus, distance is statistically independent of HPBL across 200–2,000 m. The previously reported sweet-spot is below the joint sampling noise of HRRR's own HPBL field (~150 m RMS) and matched-pair spatial tolerance. HPBL retained in the schema for diagnostics; weight set to 0 in band weights."

3. 10 GHz pressure is U-shaped, not monotonic

From the recalibration pressure-bin table at 10 GHz:

Pressure bin	n	avg_km	p50_km
< 990 mb	23,536	219.4	206.0
990–1000	12,221	182.7	159.6
1000–1010	6,246	204.5	170.6
1010–1020	3,996	214.7	173.6
≥ 1020	1,419	235.7	208.9

The minimum is 990–1000 mb, both tails are longer. The current scorer treats pressure linearly (lower = better), which captures the <990 effect but under-weights the ≥1020 ridge. Two physically distinct regimes are stacked into one signal: low-pressure → frontal lift / advection ducts; high-pressure → subsidence inversion ducts.

Edit: Part 2b "Finding 3" — note the bimodality. Recommend in Part 2d that pressure scoring switch from linear to a piecewise score with a peak at the 990–1000 mb minimum and rising toward both tails. (Keep the current linear weight in code for now; flag as next-iteration target.)

4. Commercial-link sensor: lift the deferral

commercial_samples now has 38,443 rows over 22 days (2026-03-30 → 04-20) across 7 links — 5× af11x at 11 GHz, 2× af60 at 60 GHz. Per-link baseline:

Link	radio	n_up	avg_rx0 (dBm)	sd_rx0	n_fade≥3dB	n_fade≥8dB
climax-to-core	af11x	5,488	−63.86	1.06	7	5
core-new-hope	af11x	5,491	−50.83	1.65	62	2
core-to-climax	af11x	5,491	−61.10	0.91	7	5
new-hope-to-core	af11x	5,493	−51.50	1.36	33	3
verona-to-climax	af11x	5,501	−41.97	1.04	80	12
380_982_60LR	af60	150	−52.49	2.93	8	4
982_380_60LR	af60	206	−52.14	2.48	8	3

The 60 GHz pair is intermittent (only 150–200 up-samples each — rain knocks them down hard) which is itself the strongest signal for "60 GHz weather sensitivity" we've ever had.

QSO overlap: zero contacts in the DFW box during the 22-day window. But the sensor is correlating with HRRR cleanly. Hourly mean degradation vs nearest-grid HRRR profile (n=37 hours of overlap — limited because the local hrrr_profiles mirror only has 3 days in the window):

corr(fade_db, …)	value
min_refractivity_gradient	−0.161
surface_dewpoint_c	+0.487
pwat_mm	+0.609
t-td depression	+0.147
surface_pressure_mb	−0.605
HPBL	+0.033

Magnitudes are 4–5× anything the contact-vs-HRRR table produces at 24 GHz because the sensor is sampled every 30 s instead of once per QSO.

Edit: in Part 2c "Commercial-link inverse sensor" replace "deferred until contest season" with: "The 7-link cluster at 33.2N/−96.5W now has 22 days of 30-second polling. Per-link rx_power_0 SD is 1.0–1.7 dB on af11x and 2.5–2.9 dB on af60. Hourly mean rx-degradation correlates with HRRR PWAT at +0.61 and surface pressure at −0.61 — direction matches the QSO-derived 24 GHz weights. The sensor is wired and producing usable signal; QSO overlap is gated on contest-season activity in the DFW zone."

5. Diurnal amplitude scales with frequency, but the magnitudes in algo.md are off

Restricting to local hours with ≥30 contacts per band (so 0300-local doesn't get a single 79 km outlier) produces a defensible amplitude index:

WITH local_q AS (
  SELECT band::int AS band, distance_km::float AS dist,
         EXTRACT(HOUR FROM qso_timestamp AT TIME ZONE 'America/Chicago')::int AS local_hr
  FROM contacts
  WHERE pos1 IS NOT NULL AND distance_km < 3000 AND flagged_invalid = false
    AND band >= 222 AND qso_timestamp >= '2014-10-02'
),
hourly AS (
  SELECT band, local_hr, count(*) AS n,
         PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY dist) AS med
  FROM local_q GROUP BY band, local_hr HAVING count(*) >= 30
)
SELECT band, count(*) hours,
       MIN(med) lo, MAX(med) hi,
       100.0 * (MAX(med) - MIN(med)) / AVG(med) AS pct_amp
FROM hourly GROUP BY band;

Band	hours	lo p50 (km)	hi p50 (km)	amplitude %
222 MHz	19	112.0	379.5	129.2
432 MHz	19	95.0	355.0	148.4
902 MHz	15	94.5	178.0	57.7
1.296 GHz	17	80.0	176.0	71.2
2.304 GHz	13	108.0	171.5	48.2
10 GHz	23	141.2	217.1	40.4
24 GHz	17	57.3	129.8	82.0
47 GHz	14	31.8	91.3	101.1

Microwave bands behave as advertised (10 → 24 → 47 GHz amplitude doubles roughly each step). VHF/UHF bands do not — and this is the big gotcha:

6. VHF/UHF diurnal "signal" is contest-schedule artifact

The 222 MHz and 432 MHz amplitudes (129 %, 148 %) blow past every microwave band. Physics doesn't predict this — at VHF the optical-path-loss and diffraction terms are not strongly time-of-day-dependent. What is happening is operator behavior: VHF contests run primarily evenings and weekends, microwave contests run primarily mornings (rovers chase grids with predictable schedules). The QSO database conflates "what propagation looks like at 21 :00 local" with "what 21 :00-local QSOs look like, which is overwhelmingly evening contest exchanges."

Edit: Add a new caveat box to Finding 8:

Selection bias warning. Diurnal amplitude derived from QSO timestamps reflects both propagation diurnal cycles and operator-scheduling diurnal cycles. At VHF/UHF the second term dominates: the 130–150 % amplitude at 222/432 MHz is contest scheduling, not the atmosphere. Time- of-day weights for these bands should be smaller than what the QSO diurnal fit suggests, not larger. Until a clean atmospheric-only signal exists (e.g. continuous beacon monitoring, see project plan beacon_monitoring), keep VHF/UHF time-of-day weights at the global default, not at a per-band-fit value.

7. Refractivity gradient is below noise at most bands

Reproduced from 2026-04-25-recalibration.md:

Band	rho_grad
222 MHz	+0.031
432 MHz	+0.003
902 MHz	−0.062
1.296 GHz	−0.079
2.304 GHz	−0.178
3.4 GHz	0.000
5.76 GHz	−0.110
10 GHz	+0.027
24 GHz	+0.017
47 GHz	−0.008
75 GHz	+0.474 (n=83)

The 75 GHz row is n=83, dominated by Aug/Sep contest weeks where the gradient just happens to track the same boundary-layer evolution that drives 75 GHz contacts. It is not strong evidence for a per-band gradient term.

Edit: Finding 5 already says "use continuous gradient." Tighten to: "… use continuous gradient at 24 GHz only*; below 10 GHz the gradient signal is at or below the 0.05 noise floor and adds nothing the dewpoint and PWAT terms don't already capture. Set band-specific gradient weight to 0 outside [10 GHz, 47 GHz]."*

8. Mechanical updates

The Part 2c sounding-ducting table was last refreshed at n=6,757. The 2026-04-25 recalibration ran on n=27,058. April ducting % is 25.5 %, not 31.3 % as currently shown. April PWAT is 12.7 mm, not 13.5 mm. Replace verbatim from the recalibration report.

9. Diurnal fade pattern of 11 GHz commercial links

Bonus observation from the same 22-day SNMP corpus (af11x links, 27,820 up-samples binned by local hour, fade ≥3 dB below per-link mean):

Local hour	n	n_fade≥3dB	pct
00	1,100	0	0.00
01	1,135	2	0.18
02	1,140	0	0.00
03	1,139	7	0.61
04	1,135	16	1.41
05	1,135	18	1.59
06	1,135	7	0.62
07	1,140	4	0.35
08	1,140	10	0.88
09	1,130	13	1.15
10	1,134	25	2.20
11	1,140	16	1.40
12	1,117	19	1.70
13–22	flat	3–8	0.18–0.65

Two-peak structure: 04–05 local (radiation-fog / nocturnal-inversion ducting breaks down) and 10–12 local (boundary-layer mixing onset). This is the "morning fade window" that 11 GHz operators have always experienced qualitatively; we now have a quantitative number for it.

Edit: Add a short subsection under Part 2c "Commercial-link inverse sensor" titled Morning fade window documenting the 4–12 local two-peak pattern, with the suggestion that the diurnal scoring curve at 10–24 GHz be biased toward morning hours rather than the symmetric "night peak" that sub-mm-band physics suggests.

---

What did NOT change

Finding	Status
F1: humidity reverses at ~15 GHz	preserved — rho signs flip 5.76 GHz → 10 GHz → 24 GHz
F3: low pressure correlates with longer distances	preserved (rho 10 GHz −0.075, 24 GHz −0.20, 47 GHz −0.20)
F9: ducting peaks Jun–Jul	preserved (sounding ducting % Jun 69.5, Jul 67.5, Aug 55.2)
F10: CW advantage scales with freq	not re-tested in this run; corpus and methodology unchanged
F11: regional performance unmodeled	unchanged
2014-10-02 NARR/HRRR cutover	unchanged; NARR pre-2014 corpus still 358 rows, deferred
50/144 MHz unmodeled	unchanged; need external imports

Carry-overs into ingestion / pipeline

These are not algo.md edits but worth flagging from the same query session:

1. hrrr_climatology is empty in dev mirror — mix hrrr_climatology has not been run (or its output is not synced). Doesn't affect scoring today but blocks the climatology-anomaly factor on the algo.md roadmap.
2. rtma_observations is empty — RtmaFetchWorker not enabled in dev. Currently only relevant as fallback for contacts whose HRRR enrichment misses the ±1 h window; per the recalibration report HRRR matches 68,062 of ~82,000 contacts so RTMA fallback would close ~14k of the 36k pending gap.
3. metar_5min_observations is empty — schema exists but no ingestor. Either wire an IEM 5-minute ASOS fetcher or drop the schema.
4. HRRR backlog: 36,276 contacts pending enrichment, 45,718 complete (55.8 %). Year breakdown via the new section in scripts/recalibrate_algo.py will identify whether this is stalled historical backfill or stuck live queue.

Reproducibility

Every query in this report is self-contained SQL against prop_dev (which mirrors 10.0.15.24:5432/prop). The schema-touch points are:

contacts                       (n=81,994)
hrrr_profiles                  (n=81,293,173)
hrrr_native_profiles           (n=11,472)
soundings                      (n=27,058)
nexrad_observations            (n=7,286)
narr_profiles                  (n=358 — known low)
commercial_links               (n=7)
commercial_samples             (n=38,443; 2026-03-30 → 2026-04-20)

Re-running the auto-generated companion is python3 scripts/recalibrate_algo.py.