From a49b233e4dea2864f455052f4f35b3b758c2d645 Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Tue, 7 Apr 2026 12:37:50 -0500 Subject: [PATCH] Document ERA5, RTMA, and path-integrated HRRR in algo.md Add sections for all three new data sources in Meteorological Foundations: path-integrated scoring strategy, ERA5 reanalysis for pre-2014 contacts, RTMA 15-minute surface analysis. Update calibration dataset summary with ERA5/RAOB/RTMA coverage. --- algo.md | 38 +++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/algo.md b/algo.md index f4c29fe4..5ba0a85c 100644 --- a/algo.md +++ b/algo.md @@ -12,7 +12,7 @@ The regime distinction matters because refractivity effects are *inverted* betwe ### Calibration Dataset -**QSO data:** 58,367 total QSOs across 13 bands (ARRL Microwave Contest, 1992-2024). All tropospheric (distance < 3,000 km). Enriched with 41,071,398 HRRR model profiles (37,925 matched to contacts, 65% coverage), 15,092 IEMRE gridded hourly observations, 3,268 weather stations, and 58,361 terrain path profiles. **Critical bias: 99.5% of contacts are Aug-Sep** — all atmospheric correlations are effectively summer-only findings. +**QSO data:** 58,367 total QSOs across 13 bands (ARRL Microwave Contest, 1992-2024). All tropospheric (distance < 3,000 km). Enriched with 41,071,398 HRRR model profiles (37,925 matched to contacts, 65% coverage), ERA5 reanalysis for the remaining 26,002 pre-HRRR contacts (0.25° hourly, 1940-present), 15,092 IEMRE gridded hourly observations, 10,935 RAOB soundings, 3,268 weather stations, and 58,361 terrain path profiles. RTMA (2.5 km, 15-min) supplements real-time surface conditions between HRRR hours. **Critical bias: 99.5% of contacts are Aug-Sep** — all atmospheric correlations are effectively summer-only findings. **Link data:** 7 commercial links near DFW (Princeton TX area) at 11/24/68 GHz, polled via SNMP at 5-minute intervals. All links use KTKI ASOS for weather correlation. Live polling is active; historical dataset from March 14-29 2026 (18,540 samples) was used for initial algorithm validation. @@ -82,6 +82,42 @@ Primary atmospheric data source: NOAA HRRR v4 (3 km, hourly, 18h forecast cycle) **Vertical resolution limitation:** The 25 hPa pressure level spacing resolves features ≥100m thick. Thin surface ducts (50–100m) detectable by RAOB at ~10m resolution may be missed. Comparative statistics: HRRR gradients cluster −40 to −130 N/km (median −70); collocated RAOB gradients extend to −500+ N/km. Scoring thresholds are calibrated to HRRR-derived gradient distributions, with RAOB duct detections used as supplementary data where available (3,901 profiles from 112 stations). +### Path-Integrated HRRR Scoring + +Contact scoring uses HRRR profiles at all path points (pos1, midpoint, pos2) rather than just the transmitter location. The aggregation strategy reflects physical reality: + +- **Best along path** for beneficial factors (refractivity gradient, surface pressure): a duct or frontal boundary at any point along the path can enable propagation. +- **Worst along path** for harmful factors (rain, wind): a rain cell or turbulence at any segment degrades the entire link. +- **Average along path** for neutral factors (temperature, dewpoint, PWAT, BL depth): path-integrated moisture and stability represent the bulk atmospheric state. + +Time-of-day, season, and sky cover are taken from the first profile (uniform along the path at these scales). + +### ERA5 Reanalysis (Historical Backfill) + +Secondary atmospheric source for contacts where HRRR is unavailable — primarily pre-2014 QSOs (26,002 contacts). ECMWF ERA5 provides global hourly reanalysis from 1940 to present at 0.25° resolution. + +**Access:** Copernicus Climate Data Store (CDS) API v1. Async job-based: submit request → poll for completion → download GRIB2. Typical latency 30s–5min per request. + +**Extracted fields:** +- Single-level: T₂ₘ, Td₂ₘ, Psfc, U₁₀ₘ/V₁₀ₘ, TCWV (total column water vapour), BLH (boundary layer height) +- Pressure levels: T, Td, Z at 1000–700 hPa (12 levels: 1000, 975, 950, 925, 900, 875, 850, 825, 800, 775, 750, 700) + +**Derived products:** Same as HRRR — N(h) profile, dN/dh minimum, M(h) duct detection, surface refractivity, via `SoundingParams.derive/1`. ERA5 profiles are stored in `era5_profiles` with the same schema as `hrrr_profiles` for interoperability. + +**Resolution trade-off:** ERA5's 0.25° (~28 km) horizontal resolution is 10x coarser than HRRR's 3 km, but its 12 pressure levels in the boundary layer provide comparable vertical resolution. For historical contacts, ERA5 is far better than no atmospheric data. + +**Unified lookup:** `Weather.best_profile_for_contact/1` tries HRRR first, falls back to ERA5. `Weather.profiles_along_path/1` does the same for path-integrated scoring. + +### RTMA (Real-Time Mesoscale Analysis) + +Supplementary surface data source: NOAA RTMA at 2.5 km resolution with 15-minute analysis cycles. Available on AWS S3 at `s3://noaa-rtma-pds/`. + +**What RTMA adds:** 4x temporal resolution over HRRR for surface conditions. Captures rapidly evolving mesoscale events (outflow boundaries, sea breeze fronts, convective gust fronts) between HRRR hourly cycles. + +**Fields:** T₂ₘ, Td₂ₘ, Psfc, U₁₀ₘ/V₁₀ₘ, visibility. No vertical profiles, HPBL, PWAT, or refractivity gradient — these still come from HRRR or ERA5. + +**Access:** GRIB2 via byte-range HTTP requests from S3, same pattern as HRRR. No authentication required. + ### Surface Observations (ASOS) ASOS provides in-situ validation and additional parameters not in HRRR output: