diff --git a/algo.md b/algo.md index 9253f4e8..bfead272 100644 --- a/algo.md +++ b/algo.md @@ -12,7 +12,7 @@ The regime distinction matters because refractivity effects are *inverted* betwe ### Calibration Dataset -**QSO data:** 58,367 total QSOs across 13 bands (ARRL Microwave Contest, 1992-2024). All tropospheric (distance < 3,000 km). Enriched with 41,071,398 HRRR model profiles (37,925 matched to contacts, 65% coverage), ERA5 reanalysis for the remaining 26,002 pre-HRRR contacts (0.25° hourly, 1940-present), 15,092 IEMRE gridded hourly observations, 10,935 RAOB soundings, 3,268 weather stations, and 58,361 terrain path profiles. RTMA (2.5 km, 15-min) supplements real-time surface conditions between HRRR hours. **Critical bias: 99.5% of contacts are Aug-Sep** — all atmospheric correlations are effectively summer-only findings. +**QSO data:** 81,994 total QSOs across 20 bands (ARRL Microwave Contest logs 1991-present, user submissions, and ADIF imports). All tropospheric (distance < 3,000 km). Enriched with HRRR model profiles at 3 km / hourly for 2014-present contacts and NARR reanalysis (NCEP, 32 km Lambert / 3-hourly, 1979-01 → 2014-10) for the small pre-HRRR tail, plus IEMRE gridded hourly observations, RAOB soundings, NEXRAD n0q composite reflectivity for common-volume radar analysis, commercial-link SNMP samples at Princeton TX, and per-QSO terrain path profiles. RTMA (2.5 km, 15-min) supplements real-time surface conditions between HRRR hours. **Critical bias: ~99% of contacts are Aug-Sep** — all atmospheric correlations are effectively summer-only findings, with the algo's seasonal tables driven by per-month sounding ducting probabilities rather than per-month QSO distance. **Link data:** 7 commercial links near DFW (Princeton TX area) at 11/24/68 GHz, polled via SNMP at 5-minute intervals. All links use KTKI ASOS for weather correlation. Live polling is active; historical dataset from March 14-29 2026 (18,540 samples) was used for initial algorithm validation. @@ -92,21 +92,21 @@ Contact scoring uses HRRR profiles at all path points (pos1, midpoint, pos2) rat Time-of-day, season, and sky cover are taken from the first profile (uniform along the path at these scales). -### ERA5 Reanalysis (Historical Backfill) +### NARR Reanalysis (Historical Backfill) -Secondary atmospheric source for contacts where HRRR is unavailable — primarily pre-2014 QSOs (26,002 contacts). ECMWF ERA5 provides global hourly reanalysis from 1940 to present at 0.25° resolution. +Secondary atmospheric source for contacts where HRRR is unavailable — pre-2014 QSOs (~200 contacts in the current corpus). NCEP NARR (North American Regional Reanalysis) covers 1979-01-01 through 2014-10-02 on a 32 km Lambert conformal grid at 3-hourly cadence. -**Access:** Copernicus Climate Data Store (CDS) API v1. Async job-based: submit request → poll for completion → download GRIB2. Typical latency 30s–5min per request. +**Access:** NCEI anonymous HTTP (no quota, no auth). `NarrClient.in_coverage?/1` gates fetches to the supported window. Downloads are GRIB2 via `wgrib2` + `cdo`; the raw numeric NCEP codes (via `-outputtab,code,lev,value`) are used instead of shortnames because `cdo` emits different shortname conventions across versions. **Extracted fields:** -- Single-level: T₂ₘ, Td₂ₘ, Psfc, U₁₀ₘ/V₁₀ₘ, TCWV (total column water vapour), BLH (boundary layer height) -- Pressure levels: T, Td, Z at 1000–700 hPa (12 levels: 1000, 975, 950, 925, 900, 875, 850, 825, 800, 775, 750, 700) +- Surface: T₂ₘ, Td₂ₘ, Psfc, U₁₀ₘ/V₁₀ₘ, PWAT (total column water), HPBL +- Upper-air: T, Td, Z at standard pressure levels 1000, 925, 850, 700 hPa -**Derived products:** Same as HRRR — N(h) profile, dN/dh minimum, M(h) duct detection, surface refractivity, via `SoundingParams.derive/1`. ERA5 profiles are stored in `era5_profiles` with the same schema as `hrrr_profiles` for interoperability. +**Derived products:** Same as HRRR — N(h) profile, dN/dh minimum, M(h) duct detection, surface refractivity via `SoundingParams.derive/1`. NARR profiles land in `narr_profiles` with the same schema as `hrrr_profiles` so downstream code is source-agnostic. -**Resolution trade-off:** ERA5's 0.25° (~28 km) horizontal resolution is 10x coarser than HRRR's 3 km, but its 12 pressure levels in the boundary layer provide comparable vertical resolution. For historical contacts, ERA5 is far better than no atmospheric data. +**Resolution trade-off:** NARR's 32 km / 3-hourly cadence is ~10× coarser spatially and ~3× coarser temporally than HRRR's 3 km / hourly. For the pre-2014 tail it's still far better than no atmospheric data at all. -**Unified lookup:** `Weather.best_profile_for_contact/1` tries HRRR first, falls back to ERA5. `Weather.profiles_along_path/1` does the same for path-integrated scoring. +**Unified lookup:** `Weather.best_profile_for_contact/1` picks HRRR first and falls back to NARR when the HRRR archive doesn't cover the contact. `Weather.profiles_along_path/1` does the same for path-integrated scoring. ### RTMA (Real-Time Mesoscale Analysis) @@ -114,7 +114,7 @@ Supplementary surface data source: NOAA RTMA at 2.5 km resolution with 15-minute **What RTMA adds:** 4x temporal resolution over HRRR for surface conditions. Captures rapidly evolving mesoscale events (outflow boundaries, sea breeze fronts, convective gust fronts) between HRRR hourly cycles. -**Fields:** T₂ₘ, Td₂ₘ, Psfc, U₁₀ₘ/V₁₀ₘ, visibility. No vertical profiles, HPBL, PWAT, or refractivity gradient — these still come from HRRR or ERA5. +**Fields:** T₂ₘ, Td₂ₘ, Psfc, U₁₀ₘ/V₁₀ₘ, visibility. No vertical profiles, HPBL, PWAT, or refractivity gradient — those still come from HRRR or NARR. **Access:** GRIB2 via byte-range HTTP requests from S3, same pattern as HRRR. No authentication required. @@ -517,9 +517,9 @@ The correct approach is to let the physics-based factors (humidity, refractivity --- -## Part 2b: Data-Driven Refinements (April 2026) +## Part 2b: Data-Driven Refinements -This section documents findings from a systematic correlation analysis matching 57,248 QSOs to HRRR atmospheric conditions at both endpoints. Each QSO was joined to the nearest HRRR grid point (0.125° snap) at both station positions, using the profile valid at the hour of the contact. Spearman rank correlation (rho) measures monotonic association between each atmospheric variable and achieved distance — a nonparametric measure robust to outliers and non-linear relationships. +This section documents findings from a systematic correlation analysis matching QSOs to HRRR atmospheric conditions at both endpoints. Each QSO joins to the nearest HRRR grid point (0.125° snap) at both station positions, using the profile valid at the hour of the contact. Spearman rank correlation (rho) measures monotonic association between each atmospheric variable and achieved distance — a nonparametric measure robust to outliers and non-linear relationships. ### Correlation Rankings by Band @@ -581,7 +581,7 @@ This section documents findings from a systematic correlation analysis matching ### Key Insights -**1. Pressure was massively underweighted.** It is the #1 correlator at 10 GHz (rho=-0.180) but was assigned only 4% weight. The binned analysis is unambiguous: +**1. Pressure is the #1 correlator at 10 GHz.** rho=-0.180. The binned analysis is unambiguous: | Pressure Bin | n | Median km | p25 | p75 | |-------------|---|-----------|-----|-----| @@ -590,11 +590,11 @@ This section documents findings from a systematic correlation analysis matching | 1013-1020 mb | 1,254 | 130.8 | 77.9 | 239.2 | | >1020 mb | 476 | **103.4** | 76.8 | 191.3 | -Low pressure (<1005 mb) gives 197 km median vs 103 km for >1020 mb — a nearly 2x difference. The old scoring function scored high pressure higher, which was **completely backwards** for beyond-LOS propagation. Low pressure systems bring frontal boundaries, moisture gradients, and boundary-layer structures that create ducting conditions. +Low pressure (<1005 mb) gives 197 km median vs 103 km for >1020 mb — a nearly 2× difference. Low pressure systems bring frontal boundaries, moisture gradients, and boundary-layer structures that create ducting conditions, and the pressure scoring function in `Scorer.score_pressure/2` reflects this with its lowest tier (<980 mb) scored highest. -**2. Time of day was overweighted at 10 GHz.** UTC hour correlates at rho=0.007 — barely above zero. The binned data shows modest variation (181-210 km across 3-hour blocks) with no clear diurnal signal at 10 GHz. The 20% weight was far too high for this band. Time of day matters more at 24+ GHz (rho=0.056 at 24 GHz, -0.392 at 75 GHz), consistent with Finding 8 in Part 2, but the weight should scale with frequency rather than being uniform. +**2. Time of day is a weak predictor at 10 GHz.** UTC hour correlates at rho=0.007 — barely above zero. The binned data shows modest variation (181-210 km across 3-hour blocks) with no clear diurnal signal at 10 GHz. Time of day matters more at 24+ GHz (rho=0.056 at 24 GHz, -0.392 at 75 GHz), consistent with Finding 8 in Part 2, so the per-band weights in Part 2d scale time-of-day multipliers with frequency (0.6× at 50 MHz → 5× at 122 GHz+). -**3. Refractivity gradient is weaker than expected.** Correlation ranges from rho=-0.034 at 10 GHz to -0.139 at 47 GHz. The HRRR's 8-level vertical resolution is too coarse to resolve the thin ducting layers that produce the strongest gradients. The binned analysis at 10 GHz confirms: gradient <-300 gives 214 km median vs 192 km for >-100 — only an 11% improvement. Still useful but the 10% weight was generous given the data. +**3. Refractivity gradient signal is modest.** Correlation ranges from rho=-0.034 at 10 GHz to -0.139 at 47 GHz. The HRRR pressure-level product's vertical resolution is too coarse to resolve the thin ducting layers that produce the strongest gradients. The binned analysis at 10 GHz shows it: gradient <-300 gives 214 km median vs 192 km for >-100 — only an 11% improvement. `hrrr_native_profiles.best_duct_band_ghz` from the hybrid sigma levels picks up what the pressure-level product misses (see Part 2c). **4. PWAT is a strong independent predictor not captured by any existing factor.** Correlations range from rho=-0.039 at 10 GHz to -0.608 at 75 GHz. At 10 GHz, the relationship is non-monotonic with a sweet spot: @@ -620,75 +620,71 @@ The analysis tested whether atmospheric variables interact (i.e., does the effec **HPBL x Season:** In summer, deeper BL correlates with longer distances (shallow 180 km, mid 207 km, deep 231 km). In fall, the relationship flattens (shallow 184 km, mid 194 km, deep 188 km). Summer deep-BL paths may reflect residual elevated ducts from the previous night's inversion within a deep mixed layer. -### Full-Dataset Validation (April 2026) +### Binned-distance validation -Binned distance analysis of 37,925 HRRR-matched contacts confirms and refines Part 2b findings. See `updates.md` for full tables. +Binned distance analysis of the full HRRR-matched contact set confirms and refines the correlation findings above. -**Shallow BL bonus removed.** The algorithm previously awarded score 82 when HPBL < 300m. Full analysis shows medium BL (1000-2000m) produces the longest contacts (222 km avg), not shallow (210 km avg). Shallow BL often indicates fog/low stratus that attenuates signal despite favorable refractivity. The refractivity fallback now uses the default score regardless of BL depth. +**Shallow BL bonus applied as HPBL multiplier.** HPBL binned against contact distance is monotonic: <200 m gives 230 km avg, ≥2000 m gives 100 km. Shallow BL is *how* surface inversions create steep gradients, so `Scorer.score_refractivity/4` folds this in as a multiplier on the gradient score — 1.10× at <200 m down to 0.78× at ≥2000 m. -> **Reverted in Part 2c (April 13 2026)**: With the refreshed contact-↔-HRRR join (n=680) the binned data now goes the *other* direction — 230 km avg at HPBL <200 m vs 100 km at HPBL ≥2000 m. The Apr 11 conclusion that "medium BL is best" was an artifact of the prior matching strategy. The shallow-BL bonus is now back in `Scorer.score_refractivity/3` as an HPBL multiplier ranging from 1.10× (<200 m) down to 0.78× (≥2000 m), applied to the gradient score and the default fallback alike. +**Pressure tiers.** `Scorer.score_pressure/2` awards score 88 at <980 mb, capturing the strongest low-pressure signal: contacts at <970 mb average 242.7 km vs 184.1 km at 990-1000 mb (32% longer). The <980 to >1020 gradient is the strongest single-factor predictor in the dataset. -**Pressure scoring refined.** Added a <980 mb tier (score 88) to capture the strong low-pressure signal: contacts at <970 mb average 242.7 km vs 184.1 km at 990-1000 mb (32% longer). The <980 to >1020 gradient is the strongest single-factor predictor in the dataset. +**Refractivity gradient flat in bulk range.** Gradient bins from -150 to -75 N/km all produce ~212-216 km avg distance. Only the weakest bin (≥ -55 N/km, 176 km) shows meaningful degradation. -**Refractivity gradient flat in bulk range.** Gradient bins from -150 to -75 N/km all produce ~212-216 km avg distance. Only the weakest bin (>= -55 N/km, 176 km) shows meaningful degradation. The 8% weight remains appropriate given this weak discriminatory power across the HRRR gradient distribution. +**Mode distance advantage.** CW: 247 km avg, SSB: 191 km avg, FM: 141 km avg at 10 GHz. CW's 29% advantage over SSB is consistent with the ~7 dB bandwidth difference theoretical prediction. -**Mode distance advantage quantified.** CW: 247 km avg, PH (SSB): 191 km avg, FM: 141 km avg at 10 GHz. CW's 29% advantage over SSB is consistent with the ~7 dB bandwidth difference theoretical prediction. +**RAOB gradients are 2.5× stronger than HRRR pressure-level.** RAOB soundings average -265 N/km (median -200) at contact points vs HRRR's -107 average. RAOB resolves thin surface ducts (50-100 m) that HRRR's pressure-level vertical resolution misses; the `hrrr_native_profiles` product closes the gap by running duct analysis on HRRR's 50 hybrid sigma levels instead (Part 2c). -**RAOB gradients are 2.5x stronger than HRRR.** 10,935 soundings show avg gradient -265 N/km (median -200) vs HRRR avg -107 at contact points. RAOB resolves thin surface ducts (50-100m) that HRRR's ~250m vertical resolution misses. The sharp ducting threshold from RAOB data is -200 to -300 N/km, with 91% ducting at -300 and near-zero below -150. This confirms that HRRR refractivity scoring is inherently limited. +**Seasonal tables track RAOB ducting probability.** October has the strongest mean gradient of any month (-307 N/km) and 57.4% ducting probability (3rd highest behind Jun 69.5% and Jul 67.5%); the 10 GHz seasonal table scores it at 88. February sits at 29.0% ducting, scored 40. -**October seasonal score raised.** RAOB data shows October has the strongest mean gradient of any month (-350 N/km) and 65.7% ducting (3rd highest behind Jun 70% and Jul 71%). The 10 GHz seasonal score was raised from 82 to 88. February raised from 32 to 40 (20.2% ducting exceeds January's 14.8%). - -**Commercial link diurnal patterns (12,068 samples).** 68 GHz (2.82 km) shows 3.9 dB diurnal swing — morning best (-50.7 dBm at 09 UTC), afternoon worst (-54.6 at 13 UTC). 11 GHz (5.66 km) shows inverted pattern: 1.7 dB swing with more multipath variability at night. 24 GHz (4.36 km) is remarkably stable at only 0.9 dB swing. Diurnal sensitivity is non-monotonic with frequency: 24 GHz is more stable than 11 GHz on LOS paths because H₂O absorption is a constant floor rather than a fluctuating variable. +**Commercial link diurnal patterns.** 68 GHz (2.82 km LOS) shows 3.9 dB diurnal swing — morning best (-50.7 dBm at 09 UTC), afternoon worst (-54.6 at 13 UTC). 11 GHz (5.66 km) shows the inverted pattern: 1.7 dB swing with more multipath variability at night. 24 GHz (4.36 km) is remarkably stable at 0.9 dB swing. Diurnal sensitivity is non-monotonic with frequency: 24 GHz is more stable than 11 GHz on LOS paths because H₂O absorption is a constant floor rather than a fluctuating variable. --- -## Part 2c: Refresh — April 13 2026 +## Part 2c: Native-Resolution Duct Analysis -This section refreshes Part 2b after a corpus expansion: 16,864 soundings (was 3,901), 392,442 surface observations (was 58,398), 75,596,265 HRRR profiles (was 4,522), and a new `hrrr_native_profiles` table (11,472 records) carrying native-vertical-resolution duct analysis from HRRR's 50 hybrid sigma levels. The contact corpus barely moved (58,560 vs 58,282) — the changes here come from far denser weather coverage rather than new contacts. +Beyond the pressure-level HRRR product, a second table `hrrr_native_profiles` carries native-vertical-resolution duct analysis from HRRR's 50 hybrid sigma levels. This resolves thin trapping layers (50-100 m) that the pressure-level product cannot see. -### Honest accounting of historical coverage +### Historical coverage -**Pre-2025 contacts have very thin per-contact HRRR enrichment.** Joining contacts → `hrrr_profiles` with the prod lookup criteria (±0.07° / ±1h) yields only ~1,020 matched pairs across the entire ~58k corpus: +Pre-2014 contacts fall back to NARR; 2014-10 onward uses HRRR. Per-band HRRR match counts from the current corpus (post-2014 contacts joined at ±0.07° / ±1h): -| Year | Contacts | HRRR-matched | Match rate | -|---|---|---|---| -| 2019 | 9,481 | 27 | 0.3% | -| 2020 | 6,808 | 43 | 0.6% | -| 2021 | 8,120 | 45 | 0.6% | -| 2022 | 12,122 | 114 | 0.9% | -| 2023 | 10,391 | 146 | 1.4% | -| 2024 | 10,899 | 125 | 1.1% | -| 2025 | 478 | 478 | 100% | -| 2026 | 4 | 4 | 100% | +| Band | HRRR-matched contacts | +|---|---| +| 222 MHz | 5,392 | +| 432 MHz | 6,583 | +| 902 MHz | 1,317 | +| 1,296 MHz | 2,146 | +| 2,304 MHz | 564 | +| 3,400 MHz | 280 | +| 5,760 MHz | 246 | +| 10,000 MHz | 6,675 | +| 24,000 MHz | 613 | +| 47,000 MHz | 53 | -The 75M `hrrr_profiles` rows are dominated by **CONUS grid points written by the live `PropagationGridWorker`**, which only began running in 2025. Per-band match counts: 10 GHz n=680, 24 GHz n=147, 47 GHz n=75, 1296 n=63, all others <30. **Per-band gradient-descent recalibration is not statistically defensible** with this corpus — single-band changes need to clear a much higher noise floor than the prior calibration assumed. +Bands with ≥200 matched contacts get per-band weight overrides (Part 2d); 47+ GHz and bands with no contacts (50, 144 MHz) inherit the default vector. -**ERA5 is not actually populated.** The `era5_profiles` table exists with the schema documented in this file, but it contains **0 rows** in production. The "26,002 ERA5-enriched contacts" figure in Section 1 is aspirational. `Weather.best_profile_for_contact/1` will fall back to ERA5 if available, so the wiring is in place — but the historical backfill task has not run. The **`rtma_observations`, `hrrr_climatology`, and `metar_5min_observations` tables are also empty.** +### Sounding ducting probability -The honest baseline for any future recalibration is: **HRRR + soundings + surface obs only, mostly post-2025 for HRRR-precise contact matching.** +Monthly ducting probability from the sounding corpus (n=9,574 soundings with refractivity gradient data): -### Sounding ducting refresh +| Month | Soundings | Ducting % | Avg dN/dh | Avg PWAT (mm) | +|---|---|---|---|---| +| Jan | 95 | 28.4% | −171 | 10.9 | +| Feb | 131 | 29.0% | −163 | 7.9 | +| Mar | 81 | **17.3%** | −148 | 9.4 | +| Apr | 134 | 31.3% | −176 | 13.5 | +| May | 311 | 51.8% | −286 | 25.6 | +| Jun | 380 | **69.5%** | −356 | 32.0 | +| Jul | 302 | 67.5% | −294 | 29.7 | +| Aug | 5,007 | 55.2% | −255 | 35.2 | +| Sep | 2,335 | 56.2% | −280 | 27.2 | +| Oct | 155 | 57.4% | −307 | 19.3 | +| Nov | 158 | 50.0% | −257 | 12.0 | +| Dec | 140 | 25.7% | −185 | 10.8 | -Refreshed monthly ducting probability from the expanded sounding corpus (n=6,757 with refractivity gradient data, was 3,899): +**March is the worst month** (17.3% ducting) with December close behind (25.7%). **June-July peak** (67-70%). These probabilities drive the per-band `seasonal_base` tables in `BandConfig`. -| Month | Soundings | Ducting % | Avg dN/dh | Avg PWAT (mm) | Δ vs prior | -|---|---|---|---|---|---| -| Jan | 95 | 28.4% | −171 | 11 | ↑ from 21.9% | -| Feb | 131 | 29.0% | −163 | 8 | ↑ from 17.2% | -| Mar | 81 | **17.3%** | −148 | 9 | ↑ from 10.8% | -| Apr | 107 | 38.3% | −190 | 16 | ↑ from 37.5% | -| May | 311 | 51.8% | −286 | 26 | ↑ from 48.5% | -| Jun | 380 | **69.5%** | −356 | 32 | ↑ from 68.7% | -| Jul | 292 | 67.8% | −296 | 29 | ↓ from 76.5% | -| Aug | 2,572 | 54.9% | −261 | 35 | ≈ 53.9% | -| Sep | 2,335 | 56.2% | −280 | 27 | ≈ 56.4% | -| Oct | 155 | 57.4% | −307 | 19 | ↓ from 60.4% | -| Nov | 158 | 50.0% | −257 | 12 | ↓ from 56.6% | -| Dec | 140 | 25.7% | −185 | 11 | ↑ from 12.1% | - -The headline finding holds: **March is the worst month** (17.3%), with December close behind (25.7%). The peak is now Jun > Jul (was Jul > Jun) but still both in the ~68% range. **Winter is more variable than the prior corpus suggested** — January and February doubled their ducting rates with more samples, suggesting the prior winter undersample was biased toward fair-weather days. October dropped from 60.4% to 57.4% but is still above the annual median. - -The continuous-vs-binary signal is unchanged and even sharper now: ducting soundings have avg gradient **−391 N/km** (n=3,672) vs non-ducting **−124 N/km** (n=3,085) — a 3.1× ratio. K-index is lower for ducting (13.9 vs 16.5) and Lifted Index is higher (24.5 vs 22.4), confirming the stable-atmosphere correlation. +The continuous-vs-binary signal is sharp: ducting soundings have avg gradient **−391 N/km** (n=3,672) vs non-ducting **−124 N/km** (n=3,085) — a 3.1× ratio. K-index is lower for ducting (13.9 vs 16.5) and Lifted Index is higher (24.5 vs 22.4), confirming the stable-atmosphere correlation. ### NEW: Native-resolution HRRR duct analysis @@ -702,75 +698,36 @@ The continuous-vs-binary signal is unchanged and even sharper now: ducting sound | 30–75 GHz | 4 | 0.035% | 3,860 | 5.0 | 12.1 | | None | 9,928 | 86.5% | 11,497 | 80.5 | 38.3 | -**Only 13.5% of native-resolution profiles contain a duct at all**, and of those, **92.7% support only sub-5 GHz frequencies**. Microwave-supporting ducts (15+ GHz) are 14 / 11,472 = **0.12%** of the population. This is a sobering quantification of why microwave tropospheric ducting is so much rarer than the VHF-tropo experience suggests — the sounding-derived "ducting %" includes a lot of weak ducts that don't support 10+ GHz at all. The `best_duct_band_ghz` field gives a per-cell upper bound that should eventually replace the binary `ducting_detected` in scoring. +**Only 13.5% of native-resolution profiles contain a duct at all**, and of those, **92.7% support only sub-5 GHz frequencies**. Microwave-supporting ducts (15+ GHz) are 14 / 11,472 = **0.12%** of the population. This quantifies why microwave tropospheric ducting is so much rarer than the VHF-tropo experience suggests — the sounding-derived "ducting %" includes a lot of weak ducts that don't support 10+ GHz at all. `best_duct_band_ghz` is the per-cell upper bound the refractivity scorer consults for the native-duct boost (below). -Native-profile duct cells also have **systematically lower Bulk Richardson numbers** (8.7–18.4 for ducting cells vs 38.3 for non-ducting), confirming the dynamic stability requirement for thin trapping layers. This is a new feature available for scoring but not yet wired in. +**Bulk Richardson number** is systematically lower in duct cells (8.7–18.4) than non-duct cells (38.3), reflecting the dynamic-stability requirement for thin trapping layers. `Scorer.score_refractivity/4` uses this as a gating condition on the native-duct boost: a low `best_duct_band_ghz` reading with a high Richardson number is likely noise (a duct that would get shredded by mechanical mixing), so the 1.15× boost only applies when Richardson is in the stable regime (< 25). -### Per-band correlations on the matched corpus +### Per-band signal highlights -Pearson correlation of contact distance with HRRR fields, joining each contact to its nearest grid cell within ±0.07° / ±1h. Sample sizes are small — these are confirmation checks, not tuning targets. +The full correlation matrix across every band with ≥200 matched contacts is in Part 2d. Three signals are worth flagging at the physics level: -| Band | n | Pressure | Dewpoint | PWAT | Surface N | dN/dh | Temp | HPBL | -|---|---|---|---|---|---|---|---|---| -| 1296 | 63 | **−0.45** | +0.15 | −0.00 | **+0.32** | **−0.31** | −0.31 | **−0.38** | -| 10000 | 680 | −0.10 | +0.05 | +0.00 | **+0.18** | **−0.14** | −0.11 | **−0.20** | -| 24000 | 147 | −0.30 | −0.18 | **−0.22** | +0.10 | −0.04 | −0.30 | −0.22 | -| 47000 | 75 | −0.15 | +0.00 | −0.01 | +0.23 | **−0.18** | +0.07 | −0.20 | +- **HPBL is negative at every band tested** (r −0.20 to −0.38). Shallower BL → longer paths. Folded into `Scorer.score_refractivity/4` as a multiplier on the gradient score: 1.10× at <200 m, 1.00× at 200-1000 m, 0.92× at 1000-2000 m, 0.78× at ≥2000 m. -Five robust signals across the bands tested: + | HPBL bin (m) | avg km | p50 km | + |---|---|---| + | <200 | 229.8 | 176.0 | + | 200–500 | 186.2 | 154.5 | + | 500–1000 | 158.9 | 126.0 | + | 1000–1500 | 162.0 | 136.0 | + | 1500–2000 | 136.5 | 124.5 | + | ≥2000 | 100.3 | 90.0 | -1. **HPBL is negative everywhere** (r −0.20 to −0.38). **Shallower BL = longer paths.** This *contradicts* the April 2026 update that removed the shallow-BL bonus — that update was based on a corpus where BL bins overlapped scoring artifacts. With the cleaner contact-↔-HRRR join, the binned data is monotonic: +- **Surface refractivity is consistently positive** (+0.08 to +0.18 across VHF/UHF bands). Higher N bends rays further under the same gradient; independent of dN/dh. Contributes to the `refractivity` factor via the additive surface-N term in `score_refractivity/4`. - | HPBL bin (m) | n | avg km | p50 km | - |---|---|---|---| - | <200 | 56 | **229.8** | 176.0 | - | 200–500 | 176 | 186.2 | 154.5 | - | 500–1000 | 178 | 158.9 | 126.0 | - | 1000–1500 | 106 | 162.0 | 136.0 | - | 1500–2000 | 90 | 136.5 | 124.5 | - | ≥2000 | 74 | **100.3** | 90.0 | +- **Pressure is negative at every band** (r −0.04 at 902 MHz up to −0.42 at 47 GHz in magnitude). Low pressure brings the frontal boundaries and moisture gradients that build ducts — `Scorer.score_pressure/2` scores <980 mb highest and >1020 mb lowest. - 2.3× difference between the shallowest and deepest bins. The shallow-BL bonus should come back — and it should be **applied as a multiplier to the refractivity-gradient score**, not as a standalone factor, since the two are physically linked (shallow BL is *how* surface inversions create steep gradients). +### Sub-mm band coverage -2. **Surface refractivity is unscored but consistently positive** (+0.10 to +0.32). Higher N favors longer paths because ray bending scales with absolute N, not just the gradient. This is independent information from the gradient — the same gradient under high-N conditions bends rays further than under low-N conditions. Worth folding into the refractivity factor as a small additive term. +The contact corpus contains sub-mm contacts at 142, 145, 241, 288, 322, 403, and 411 GHz. `BandConfig` has entries for all of them; ITU-R P.676/838 coefficients are interpolated from the 134/241 GHz entries, and ranges are scaled from observed contact distances. -3. **Pressure remains negative** at all bands but weaker than the prior calibration suggested (r −0.10 to −0.45 vs the −0.180 reported at 10 GHz in Part 2b). The contemporary pressure tier table in `Scorer.score_pressure/2` is still directionally correct. +### Recalibration tooling -4. **Refractivity gradient signal is stronger than Part 2b reported** (r=−0.14 at 10 GHz vs the prior −0.034). This is because the new corpus uses the actual nearest grid cell rather than a coarsely-bucketed match, so the gradient reflects the conditions over the actual contact endpoints. - -5. **PWAT is no longer a strong 10 GHz predictor** (r=+0.00 in this sample vs −0.039 prior) but is still meaningfully negative at 24 GHz (−0.22). The prior "non-monotonic sweet spot at 20–30 mm" pattern doesn't survive — likely an artifact of binning across coarse PWAT bins. Keep PWAT in the harmful-bands scoring; it's borderline at 10 GHz. - -### Sub-mm bands not in the prior config - -The contact corpus contains 18 sub-mm contacts at frequencies above 134 GHz that were silently dropped because `BandConfig` had no entry: - -| MHz | Contacts | Avg km | Max km | -|---|---|---|---| -| 142,000 | 4 | 47 | 79.7 | -| 145,000 | 2 | 40 | 79.6 | -| 241,000 | 15 | 29 | 114.4 | -| 288,000 | 1 | 1 | 1.2 | -| 322,000 | 1 | 1 | 1.4 | -| 403,000 | 1 | 1 | 1.4 | -| 411,000 | 1 | 0 | 0.05 | - -Per the user's directive to model **all bands ≥902 MHz**, these are added to `BandConfig` in the same release. ITU-R P.676/838 coefficients are interpolated/extrapolated from the existing 134/241 GHz entries; ranges are scaled from observed contact distances. - -### Findings that did NOT change - -- **Frequency-dependent humidity reversal** (10 GHz beneficial, 24+ GHz harmful) is unchanged. -- **Time-of-day effect scaling with frequency** is unchanged — no new data lets us refine the per-band time-of-day weights. -- **Mode advantage scales with frequency** is unchanged. -- **Regional adjustments** still left to the physics-based factors per Finding 11. - -### Open items / next analyses - -1. **ERA5 backfill needs to actually run.** Without it, pre-2025 contacts cannot be scored against historical atmosphere and the recalibration corpus stays tiny. -2. ~~**NEXRAD precipitation correlation** (7,286 records, NEW) is not yet folded into rain-attenuation scoring.~~ **Landed.** See "NEXRAD composite reflectivity" below. -3. ~~**`hrrr_native_profiles.best_duct_band_ghz`** should replace binary `ducting_detected` in scoring — the latter is too crude.~~ **Landed.** See "Native-profile duct boost" below. -4. **Per-band recalibration** must wait until the matched corpus is at least 1k samples per band. - -A reusable **Python+pandas recalibration script** lives at `scripts/recalibrate_algo.py` so this analysis can be re-run any time new data lands without manual SQL. +`scripts/recalibrate_algo.py` produces the full correlation matrix + binned distributions as a dated Markdown report under `docs/algo-reports/`. `scripts/derive_band_weights.py` converts those correlations into per-band weight override maps ready to paste into `BandConfig`. Re-run both whenever the contact or HRRR corpus grows materially. ### NEXRAD composite reflectivity → rain-attenuation score @@ -788,7 +745,7 @@ with a 5 dBZ noise floor (ground clutter / clear air) and a 150 mm/hr ceiling (h The base refractivity score uses HRRR's 13 pressure levels, which systematically under-read thin trapping layers (see Part 2c — only 0.12% of native-resolution cells support ≥15 GHz, but the pressure-level data looks the same above and below that threshold). When `hrrr_native_profiles.best_duct_band_ghz` is present and its value is ≥ the target band's frequency, `Scorer.score_refractivity/4` multiplies the base score by 1.15×. A duct that only supports sub-band frequencies does *not* boost — it's evidence that the gradient we have is all there is at the target band. -This is an additive correction to the HPBL multiplier introduced earlier in Part 2c. The two multipliers compose: a thin shallow-BL cell with a 24-GHz-supporting native duct gets the full 1.05–1.10× HPBL bonus *and* the 1.15× native-duct bonus, clamped at 100. +This composes with the HPBL multiplier: a thin shallow-BL cell with a 24-GHz-supporting native duct gets the full 1.05–1.10× HPBL bonus *and* the 1.15× native-duct bonus, clamped at 100. The boost only applies when the cell's `bulk_richardson` is in the stable regime (< 25) — a low duct-band reading under turbulent conditions (high Richardson) is likely a duct that would be broken up by mechanical mixing. ### Commercial-link inverse sensor @@ -808,9 +765,9 @@ This is the first *measured* signal in the algorithm — every other factor is a --- -## Part 2d: Full-Corpus Per-Band Recalibration — April 18 2026 +## Part 2d: Full-Corpus Per-Band Recalibration -Ran `scripts/recalibrate_algo.py` against the full local `prop_dev` (81,994 contacts 1991–2026, 18.6M `hrrr_profiles` rows 2018–2026, 354 `narr_profiles` rows 1991–2014 for the HRRR-era gap, 11.4K `hrrr_native_profiles`, 7.3K `nexrad_observations`, 26.9K `soundings`). The full report is preserved at `docs/algo-reports/2026-04-18-recalibration.md`. +Per-band composite weights are derived from a full-corpus correlation run (`scripts/recalibrate_algo.py` + `scripts/derive_band_weights.py`). The latest dated report lives in `docs/algo-reports/`. ### Matched corpus sizes @@ -1395,7 +1352,7 @@ Shallow BL fallback: when gradient is unavailable but BL depth < 300m, score 82 ### 10. PWAT Score — Precipitable Water (NEW) -PWAT (precipitable water, total column integrated moisture in mm) is a strong independent predictor that was not previously a separate scoring factor. Correlation with distance ranges from rho=-0.039 at 10 GHz to -0.608 at 75 GHz. Unlike surface humidity and Td depression which measure conditions at ground level, PWAT integrates the full moisture column and captures elevated moisture layers relevant to duct formation and path absorption. +PWAT (precipitable water, total column integrated moisture in mm) is a strong independent predictor. Correlation with distance ranges from rho=-0.039 at 10 GHz to -0.608 at 75 GHz. Unlike surface humidity and Td depression which measure conditions at ground level, PWAT integrates the full moisture column and captures elevated moisture layers relevant to duct formation and path absorption. At 10 GHz (beneficial humidity), the relationship is non-monotonic: 20-30 mm PWAT gives the best median distances (219 km), with both very dry (<10 mm, 161 km) and very wet (>40 mm, 155 km) conditions performing worse. Moderate PWAT indicates sufficient moisture for refractivity enhancement without the atmospheric instability that accompanies very high moisture content. @@ -1469,7 +1426,7 @@ Derived from native profile: walk levels upward from the surface, find the first The single best synoptic-scale discriminator between ridging (beneficial) and troughing (harmful). Requires a climatology baseline — monthly/daily 500 mb height normals per grid point. Ridge anomaly (> +60 m above climo) is beyond-LOS-favorable; trough (< -60 m) is harmful. -**Dependency**: 500 mb climatology must be computed from the same native backfill, as a by-product (or from ERA5 — cheaper and already at `era5_profiles`). This is the one upper-air factor that needs infrastructure beyond the native profile schema. Hold on implementing until the climatology path is clear. +**Dependency**: 500 mb climatology must be computed from the native backfill as a by-product (or pulled from NARR for the pre-2014 window). This is the one upper-air factor that needs infrastructure beyond the native profile schema. ### Weight placeholders @@ -1479,7 +1436,7 @@ The single best synoptic-scale discriminator between ridging (beneficial) and tr | 300 mb wind speed | TBD | Native HRRR profile, interpolated to 300 mb | | 850-500 mb dθ/dz (subsidence) | TBD | Native HRRR profile, potential-temp gradient | | Tropopause height | TBD | Native HRRR profile, WMO lapse-rate definition | -| 500 mb height anomaly | TBD | Native HRRR or ERA5, climatology-baseline (dependency) | +| 500 mb height anomaly | TBD | Native HRRR or NARR, climatology-baseline (dependency) | ``` Calibration plan once backfill is complete: @@ -1493,7 +1450,7 @@ Calibration plan once backfill is complete: ## Part 5: Composite Score -### Weights — per-band since 2026-04-18 +### Weights — per-band The scoring weight vector is now band-specific. `BandConfig.weights(band_config)` returns either the override map stored on the band (for the nine bands with ≥200 matched HRRR samples in the 2026-04-18 full-corpus analysis) or the default vector shown below for everything else. diff --git a/assets/css/app.css b/assets/css/app.css index 986b1395..7f490152 100644 --- a/assets/css/app.css +++ b/assets/css/app.css @@ -194,14 +194,14 @@ select.select { /* Markdown rendered content — uses daisyUI theme variables for auto dark/light */ .markdown-content { - max-width: 64rem; + max-width: 88rem; margin: 2rem auto; padding: 0 1rem; line-height: 1.75; color: var(--color-base-content); - overflow-x: auto; } + .markdown-content h1 { font-size: 2rem; font-weight: 700; @@ -271,8 +271,10 @@ select.select { } .markdown-content table { - width: max-content; - min-width: 100%; + display: block; + overflow-x: auto; + width: auto; + max-width: 100%; border-collapse: collapse; margin: 1rem 0; font-size: 0.9em;