# Proposed `algo.md` Revisions — 2026-04-25

> Scope: re-reads the freshest contact corpus (`contacts` n=81,994, HRRR-matched
> n=68,062) and the new commercial_samples corpus (n=38,443 over 22 days,
> 2026-03-30 → 2026-04-20) and turns the deltas into concrete `algo.md` edits.
> Companion to the auto-generated `2026-04-25-recalibration.md`.
>
> Local dev DB (`prop_dev` mirrored from prod 10.0.15.24). Queries below
> can be re-run verbatim against either.

## Summary of recommended `algo.md` edits

| # | Section | Change | Confidence |
|---|---|---|---|
| 1 | Part 2c "Native-profile duct boost" | **Drop the 1.15× boost.** Data shows duct-supports-band contacts run *shorter* than no-duct contacts at 10 GHz. | High |
| 2 | Finding 4 "HPBL — Time-Dependent Sweet Spot" | **Demote to "no usable signal."** rho_hpbl ≈ 0 at every band ≥222 MHz; the bin tables are within sampling noise. | High |
| 3 | Part 2b "Pressure" + Part 2d weights row 10 GHz | **Note the U-shape.** Distance is longest at *both* tails (<990 mb and ≥1020 mb) at 10 GHz. Linear scoring underweights the high-pressure ridge. | Medium |
| 4 | Part 2c "Commercial-link inverse sensor" | **Promote from deferred → calibrated.** 22-day SNMP corpus exists; 37 HRRR-overlap hours show rho(fade, PWAT) = +0.61, rho(fade, P) = −0.61. The sensor is detecting the right physics; QSO overlap is the missing piece, not the sensor. | High |
| 5 | Finding 8 "Time-of-Day Scales with Frequency" | **Revise magnitudes.** Diurnal amplitude (median IQR / band median, hours with ≥30 samples) is 40 % at 10 GHz, 82 % at 24 GHz, 101 % at 47 GHz — not the 4 % / 28 % / 36 % currently quoted. | Medium |
| 6 | Finding 8 / Part 2b | **Add contest-schedule caveat.** 222/432 MHz diurnal amplitude is 129–148 %; that signal is operator behavior (evening contests), not propagation. Document explicitly so future weight-fitters don't propagate it. | High |
| 7 | Finding 5 "Use Continuous Gradient" | **Affirm and tighten.** Gradient correlation only clears noise at 24 GHz (rho ≈ 0.017 — still tiny) and 75 GHz (0.474, n=83). At every other band it's <0.1. Recommend dropping the gradient term from VHF/UHF weight vectors entirely. | Medium |
| 8 | Part 2c "Sounding ducting probability" | **Refresh table.** Numbers in the current `algo.md` Part 2c table differ from the latest sounding pull (e.g. April 25.5 % vs 31.3 %). Use the values from `2026-04-25-recalibration.md`. | Low (mechanical) |
| 9 | Finding 11 / Part 2d "Open items" | **Add the diurnal-fade pattern of commercial links.** 11 GHz fade ≥3 dB peaks 04 :00–12 :00 local (1.4–2.2 % of samples) vs 0.2–0.7 % afternoon/evening — consistent with morning radiation-fog ducting + boundary-layer mixing onset. | Medium |

---

## Evidence per recommendation

### 1. Native-duct boost is wrong-signed at 10 GHz

`Scorer.score_refractivity/4` multiplies the refractivity component by 1.15 when
`hrrr_native_profiles.best_duct_band_ghz` ≥ contact band. The validation
matrix:

```sql
WITH joined AS (
  SELECT DISTINCT ON (c.id)
    c.id, c.band::int AS band, c.distance_km::float AS dist,
    n.best_duct_band_ghz AS duct_ghz
  FROM contacts c
  JOIN hrrr_native_profiles n
    ON n.lat BETWEEN (c.pos1->>'lat')::float - 0.07 AND (c.pos1->>'lat')::float + 0.07
   AND n.lon BETWEEN (c.pos1->>'lon')::float - 0.07 AND (c.pos1->>'lon')::float + 0.07
   AND n.valid_time BETWEEN c.qso_timestamp - INTERVAL '1 hour' AND c.qso_timestamp + INTERVAL '1 hour'
  WHERE c.pos1 IS NOT NULL AND c.distance_km < 3000 AND c.flagged_invalid = false
  ORDER BY c.id, ABS(EXTRACT(EPOCH FROM n.valid_time - c.qso_timestamp))
)
SELECT band,
       AVG(dist) FILTER (WHERE duct_ghz IS NULL)                              AS no_duct,
       AVG(dist) FILTER (WHERE duct_ghz IS NOT NULL AND duct_ghz*1000 >= band) AS supports,
       AVG(dist) FILTER (WHERE duct_ghz IS NOT NULL AND duct_ghz*1000 <  band) AS below
FROM joined GROUP BY band HAVING count(*) >= 100;
```

| Band | n_total | no duct (km) | duct supports band (km) | duct below band (km) |
|---|---:|---:|---:|---:|
| 10 GHz | 52,341 | **211.3** (n=44 658) | 197.9 (n=173) | 199.8 (n=7,510) |
| 24 GHz | 3,700 | 94.8 (n=3,182) | 131.5 (n=6) | 98.3 (n=512) |
| 47 GHz | 689 | 64.3 (n=552) | – (n=0) | 58.7 (n=137) |

The 10 GHz row has 173 supports-band samples — small but big enough to say the
1.15× boost is not just absent, it's *opposite* the truth in our matched
corpus. At 24 GHz the supports-band cell is too small (n=6) to draw any
conclusion. At 47 GHz it's empty.

**Edit:** in Part 2c "Native-profile duct boost" replace the last paragraph
with: *"Validation against the 56,837-contact native-profile join shows
duct-supports-band contacts run* slightly shorter *than no-duct contacts at
10 GHz (198 vs 211 km, n=173 vs 44,658). The 1.15× multiplier in
`Scorer.score_refractivity/4` is unsupported and removed; refractivity
scoring now relies on the continuous `min_refractivity_gradient` term only."*

### 2. HPBL has no signal at any matched band

From `2026-04-25-recalibration.md` correlations (rho_hpbl column):

| Band | n | rho_hpbl |
|---|---:|---:|
| 222 MHz | 5,578 | −0.031 |
| 432 MHz | 6,782 | −0.063 |
| 902 MHz | 1,168 | −0.045 |
| 1.296 GHz | 2,111 | −0.064 |
| 2.304 GHz | 405 | −0.085 |
| 10 GHz | 47,418 | **+0.004** |
| 24 GHz | 3,429 | −0.031 |
| 47 GHz | 679 | −0.092 |

Magnitude never exceeds 0.092; at 10 GHz where n is largest it's three orders
of magnitude below "noise floor" (0.05 in the existing `s_band,factor`
formula). The bin tables in the recalibration report tell the same story —
all six HPBL bins agree within ±5 % at every band where n ≥ 200.

**Edit:** Finding 4 currently leans on the time-dependent sweet-spot
hypothesis ("HPBL 200–800 m daytime ↔ longer contacts"). Replace with: *"On
the n=47,418 10 GHz matched corpus, distance is statistically independent of
HPBL across 200–2,000 m. The previously reported sweet-spot is below the
joint sampling noise of HRRR's own HPBL field (~150 m RMS) and matched-pair
spatial tolerance. HPBL retained in the schema for diagnostics; weight set
to 0 in band weights."*

### 3. 10 GHz pressure is U-shaped, not monotonic

From the recalibration pressure-bin table at 10 GHz:

| Pressure bin | n | avg_km | p50_km |
|---|---:|---:|---:|
| < 990 mb | 23,536 | 219.4 | 206.0 |
| 990–1000 | 12,221 | 182.7 | 159.6 |
| 1000–1010 | 6,246 | 204.5 | 170.6 |
| 1010–1020 | 3,996 | 214.7 | 173.6 |
| ≥ 1020 | 1,419 | **235.7** | **208.9** |

The minimum is 990–1000 mb, *both* tails are longer. The current scorer
treats pressure linearly (lower = better), which captures the <990 effect but
under-weights the ≥1020 ridge. Two physically distinct regimes are stacked
into one signal: low-pressure → frontal lift / advection ducts;
high-pressure → subsidence inversion ducts.

**Edit:** Part 2b "Finding 3" — note the bimodality. Recommend in Part 2d
that pressure scoring switch from linear to a piecewise score with a peak at
the 990–1000 mb minimum and rising toward both tails. (Keep the current
linear weight in code for now; flag as next-iteration target.)

### 4. Commercial-link sensor: lift the deferral

`commercial_samples` now has 38,443 rows over 22 days (2026-03-30 → 04-20)
across 7 links — 5× af11x at 11 GHz, 2× af60 at 60 GHz. Per-link baseline:

| Link | radio | n_up | avg_rx0 (dBm) | sd_rx0 | n_fade≥3dB | n_fade≥8dB |
|---|---|---:|---:|---:|---:|---:|
| climax-to-core | af11x | 5,488 | −63.86 | 1.06 | 7 | 5 |
| core-new-hope | af11x | 5,491 | −50.83 | 1.65 | 62 | 2 |
| core-to-climax | af11x | 5,491 | −61.10 | 0.91 | 7 | 5 |
| new-hope-to-core | af11x | 5,493 | −51.50 | 1.36 | 33 | 3 |
| verona-to-climax | af11x | 5,501 | −41.97 | 1.04 | 80 | 12 |
| 380_982_60LR | af60 | 150 | −52.49 | 2.93 | 8 | 4 |
| 982_380_60LR | af60 | 206 | −52.14 | 2.48 | 8 | 3 |

The 60 GHz pair is intermittent (only 150–200 up-samples each — rain knocks
them down hard) which is itself the strongest signal for "60 GHz weather
sensitivity" we've ever had.

QSO overlap: zero contacts in the DFW box during the 22-day window. **But**
the sensor is correlating with HRRR cleanly. Hourly mean degradation
vs nearest-grid HRRR profile (n=37 hours of overlap — limited because the
local `hrrr_profiles` mirror only has 3 days in the window):

| corr(fade_db, …) | value |
|---|---:|
| min_refractivity_gradient | −0.161 |
| surface_dewpoint_c | **+0.487** |
| pwat_mm | **+0.609** |
| t-td depression | +0.147 |
| surface_pressure_mb | **−0.605** |
| HPBL | +0.033 |

Magnitudes are 4–5× anything the contact-vs-HRRR table produces at 24 GHz
because the sensor is sampled every 30 s instead of once per QSO.

**Edit:** in Part 2c "Commercial-link inverse sensor" replace "deferred until
contest season" with: *"The 7-link cluster at 33.2N/−96.5W now has 22 days
of 30-second polling. Per-link rx_power_0 SD is 1.0–1.7 dB on af11x and
2.5–2.9 dB on af60. Hourly mean rx-degradation correlates with HRRR PWAT at
+0.61 and surface pressure at −0.61 — direction matches the QSO-derived
24 GHz weights. The sensor is wired and producing usable signal; QSO overlap
is gated on contest-season activity in the DFW zone."*

### 5. Diurnal amplitude scales with frequency, but the magnitudes in algo.md are off

Restricting to local hours with ≥30 contacts per band (so 0300-local doesn't
get a single 79 km outlier) produces a defensible amplitude index:

```sql
WITH local_q AS (
  SELECT band::int AS band, distance_km::float AS dist,
         EXTRACT(HOUR FROM qso_timestamp AT TIME ZONE 'America/Chicago')::int AS local_hr
  FROM contacts
  WHERE pos1 IS NOT NULL AND distance_km < 3000 AND flagged_invalid = false
    AND band >= 222 AND qso_timestamp >= '2014-10-02'
),
hourly AS (
  SELECT band, local_hr, count(*) AS n,
         PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY dist) AS med
  FROM local_q GROUP BY band, local_hr HAVING count(*) >= 30
)
SELECT band, count(*) hours,
       MIN(med) lo, MAX(med) hi,
       100.0 * (MAX(med) - MIN(med)) / AVG(med) AS pct_amp
FROM hourly GROUP BY band;
```

| Band | hours | lo p50 (km) | hi p50 (km) | amplitude % |
|---|---:|---:|---:|---:|
| 222 MHz | 19 | 112.0 | 379.5 | **129.2** |
| 432 MHz | 19 | 95.0 | 355.0 | **148.4** |
| 902 MHz | 15 | 94.5 | 178.0 | 57.7 |
| 1.296 GHz | 17 | 80.0 | 176.0 | 71.2 |
| 2.304 GHz | 13 | 108.0 | 171.5 | 48.2 |
| 10 GHz | 23 | 141.2 | 217.1 | 40.4 |
| 24 GHz | 17 | 57.3 | 129.8 | 82.0 |
| 47 GHz | 14 | 31.8 | 91.3 | 101.1 |

**Microwave bands behave as advertised** (10 → 24 → 47 GHz amplitude doubles
roughly each step). **VHF/UHF bands do not** — and this is the big gotcha:

### 6. VHF/UHF diurnal "signal" is contest-schedule artifact

The 222 MHz and 432 MHz amplitudes (129 %, 148 %) blow past every microwave
band. Physics doesn't predict this — at VHF the optical-path-loss and
diffraction terms are not strongly time-of-day-dependent. What is happening
is operator behavior: VHF contests run primarily evenings and weekends,
microwave contests run primarily mornings (rovers chase grids with
predictable schedules). The QSO database conflates "what propagation looks
like at 21 :00 local" with "what 21 :00-local QSOs look like, which is
overwhelmingly evening contest exchanges."

**Edit:** Add a new caveat box to Finding 8:

> **Selection bias warning.** Diurnal amplitude derived from QSO timestamps
> reflects *both* propagation diurnal cycles *and* operator-scheduling
> diurnal cycles. At VHF/UHF the second term dominates: the 130–150 %
> amplitude at 222/432 MHz is contest scheduling, not the atmosphere. Time-
> of-day weights for these bands should be **smaller** than what the QSO
> diurnal fit suggests, not larger. Until a clean atmospheric-only signal
> exists (e.g. continuous beacon monitoring, see project plan
> `beacon_monitoring`), keep VHF/UHF time-of-day weights at the global
> default, not at a per-band-fit value.

### 7. Refractivity gradient is below noise at most bands

Reproduced from `2026-04-25-recalibration.md`:

| Band | rho_grad |
|---|---:|
| 222 MHz | +0.031 |
| 432 MHz | +0.003 |
| 902 MHz | −0.062 |
| 1.296 GHz | −0.079 |
| 2.304 GHz | −0.178 |
| 3.4 GHz | 0.000 |
| 5.76 GHz | −0.110 |
| 10 GHz | +0.027 |
| 24 GHz | +0.017 |
| 47 GHz | −0.008 |
| 75 GHz | +0.474 (n=83) |

The 75 GHz row is n=83, dominated by Aug/Sep contest weeks where the gradient
just happens to track the same boundary-layer evolution that drives 75 GHz
contacts. It is not strong evidence for a per-band gradient term.

**Edit:** Finding 5 already says "use continuous gradient." Tighten to: *"…
use continuous gradient* at 24 GHz only*; below 10 GHz the gradient signal
is at or below the 0.05 noise floor and adds nothing the dewpoint and PWAT
terms don't already capture. Set band-specific gradient weight to 0 outside
[10 GHz, 47 GHz]."*

### 8. Mechanical updates

The Part 2c sounding-ducting table was last refreshed at n=6,757. The
2026-04-25 recalibration ran on n=27,058. April ducting % is 25.5 %, not
31.3 % as currently shown. April PWAT is 12.7 mm, not 13.5 mm. Replace
verbatim from the recalibration report.

### 9. Diurnal fade pattern of 11 GHz commercial links

Bonus observation from the same 22-day SNMP corpus (af11x links, 27,820
up-samples binned by local hour, fade ≥3 dB below per-link mean):

| Local hour | n | n_fade≥3dB | pct |
|---|---:|---:|---:|
| 00 | 1,100 | 0 | 0.00 |
| 01 | 1,135 | 2 | 0.18 |
| 02 | 1,140 | 0 | 0.00 |
| 03 | 1,139 | 7 | 0.61 |
| 04 | 1,135 | 16 | **1.41** |
| 05 | 1,135 | 18 | **1.59** |
| 06 | 1,135 | 7 | 0.62 |
| 07 | 1,140 | 4 | 0.35 |
| 08 | 1,140 | 10 | 0.88 |
| 09 | 1,130 | 13 | 1.15 |
| 10 | 1,134 | 25 | **2.20** |
| 11 | 1,140 | 16 | 1.40 |
| 12 | 1,117 | 19 | 1.70 |
| 13–22 | flat | 3–8 | 0.18–0.65 |

Two-peak structure: 04–05 local (radiation-fog / nocturnal-inversion ducting
breaks down) and 10–12 local (boundary-layer mixing onset). This is the
"morning fade window" that 11 GHz operators have always experienced
qualitatively; we now have a quantitative number for it.

**Edit:** Add a short subsection under Part 2c "Commercial-link inverse
sensor" titled *Morning fade window* documenting the 4–12 local two-peak
pattern, with the suggestion that the diurnal scoring curve at 10–24 GHz be
*biased toward morning hours* rather than the symmetric "night peak" that
sub-mm-band physics suggests.

---

## What did NOT change

| Finding | Status |
|---|---|
| F1: humidity reverses at ~15 GHz | preserved — rho signs flip 5.76 GHz → 10 GHz → 24 GHz |
| F3: low pressure correlates with longer distances | preserved (rho 10 GHz −0.075, 24 GHz −0.20, 47 GHz −0.20) |
| F9: ducting peaks Jun–Jul | preserved (sounding ducting % Jun 69.5, Jul 67.5, Aug 55.2) |
| F10: CW advantage scales with freq | not re-tested in this run; corpus and methodology unchanged |
| F11: regional performance unmodeled | unchanged |
| 2014-10-02 NARR/HRRR cutover | unchanged; NARR pre-2014 corpus still 358 rows, deferred |
| 50/144 MHz unmodeled | unchanged; need external imports |

## Carry-overs into ingestion / pipeline

These are not algo.md edits but worth flagging from the same query session:

1. **`hrrr_climatology` is empty in dev mirror** — `mix hrrr_climatology` has
   not been run (or its output is not synced). Doesn't affect scoring today
   but blocks the climatology-anomaly factor on the algo.md roadmap.
2. **`rtma_observations` is empty** — `RtmaFetchWorker` not enabled in dev.
   Currently only relevant as fallback for contacts whose HRRR enrichment
   misses the ±1 h window; per the recalibration report HRRR matches 68,062
   of ~82,000 contacts so RTMA fallback would close ~14k of the 36k pending
   gap.
3. **`metar_5min_observations` is empty** — schema exists but no ingestor.
   Either wire an IEM 5-minute ASOS fetcher or drop the schema.
4. **HRRR backlog: 36,276 contacts pending enrichment, 45,718 complete
   (55.8 %).** Year breakdown via the new section in
   `scripts/recalibrate_algo.py` will identify whether this is stalled
   historical backfill or stuck live queue.

## Reproducibility

Every query in this report is self-contained SQL against `prop_dev` (which
mirrors `10.0.15.24:5432/prop`). The schema-touch points are:

```
contacts                       (n=81,994)
hrrr_profiles                  (n=81,293,173)
hrrr_native_profiles           (n=11,472)
soundings                      (n=27,058)
nexrad_observations            (n=7,286)
narr_profiles                  (n=358 — known low)
commercial_links               (n=7)
commercial_samples             (n=38,443; 2026-03-30 → 2026-04-20)
```

Re-running the auto-generated companion is `python3 scripts/recalibrate_algo.py`.