diff --git a/algo.md b/algo.md index 67ff07fd..ee159613 100644 --- a/algo.md +++ b/algo.md @@ -1297,25 +1297,125 @@ For each band: | Signal prediction floor | +/- 2-3 dB | Measured from link data | | CW bandwidth advantage | ~7 dB | 10*log10(2700/500); 24% range increase | -## ITU-R References +## ITU-R Recommendations -- P.453-14: Radio refractivity -- P.525: Free-space path loss -- P.676-13: Gaseous absorption (O2 + H2O) -- P.838-3: Specific rain attenuation -- P.526: Diffraction -- P.452: Interference between stations +The following ITU-R Recommendations provide the physics models underlying the scoring algorithm. All are publicly available from the ITU Radiocommunication Sector (https://www.itu.int/rec/R-REC-P/en). -## Data References +| Recommendation | Title | Used For | +|----------------|-------|----------| +| ITU-R P.453-14 | The radio refractive index: its formula and refractivity data | Surface refractivity N calculation, refractivity gradient | +| ITU-R P.525-4 | Calculation of free-space attenuation | Free-space path loss baseline | +| ITU-R P.676-13 | Attenuation by atmospheric gases and related effects | O2 and H2O absorption coefficients per band | +| ITU-R P.838-3 | Specific attenuation model for rain for use in prediction methods | Rain attenuation coefficients (k, alpha) per band | +| ITU-R P.526-15 | Propagation by diffraction | Knife-edge and terrain diffraction loss | +| ITU-R P.452-17 | Prediction procedure for the evaluation of interference between stations on the surface of the Earth | Clear-air propagation modeling framework | +| ITU-R P.835-6 | Reference standard atmospheres | Standard atmosphere profiles for baseline | +| ITU-R P.530-18 | Propagation data and prediction methods for terrestrial line-of-sight systems | Multipath fading and enhancement statistics | -- ARRL Microwave Contest QSO database: 58,282 total, 57,488 tropospheric with distance (1992-2024) -- IEM ASOS observations: 1,299 stations, 58,398 surface observations -- IEM RAOB soundings: 112 stations, 3,901 soundings -- IEMRE gridded hourly: 3,675 observations at 0.125° resolution -- HRRR model profiles: 4,522 profiles with refractivity, BL depth, PWAT -- Terrain profiles: 58,276 QSO paths (97.2% blocked, avg 36.2 dB diffraction) -- Commercial link data: 7 links at 11/24/68 GHz near DFW, SNMP-polled -- Solar indices: 9,586 daily values (1998-2026) — reserved for future use +## Data Sources + +### Primary QSO Dataset + +**ARRL Microwave Contest Results (1992-2024)** +- Source: Contest log submissions compiled from ARRL contest results +- Volume: 58,282 total QSOs across 13+ bands (10 GHz through 403 GHz) +- Usable subset: 57,488 tropospheric QSOs after filtering 4 EME contacts (QRA64D/JT4F modes >3,000 km) +- Fields: station callsigns, Maidenhead grid squares, timestamp, mode (CW/SSB/FM/FT8), band +- Grid-to-coordinate conversion: gridmap.org API for Maidenhead → lat/lon +- Distance: Haversine great-circle calculation from grid square centers + +### Surface Weather Observations (ASOS) + +**Iowa Environmental Mesonet (IEM) — Automated Surface Observing System** +- API: `https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py` +- Network discovery: `https://mesonet.agron.iastate.edu/api/1/network.py` +- Station count: 2,922 total, 1,299 with observations matched to QSOs +- Observation count: 58,398 surface observations +- Fields: temperature (°F), dewpoint (°F), relative humidity (%), wind speed (kts), wind direction (°), sea level pressure (mb), sky condition (CLR/FEW/SCT/BKN/OVC), precipitation (inches/hour), weather codes +- Temporal matching: ±2 hours around QSO timestamp +- Spatial matching: nearest station within 150 km of QSO path endpoints +- No authentication required + +### Upper-Air Soundings (RAOB) + +**Iowa Environmental Mesonet (IEM) — Radiosonde Observations** +- API: `https://mesonet.agron.iastate.edu/json/raob.py` +- Station count: 346 sounding stations total, 112 with data matched to QSOs +- Sounding count: 3,901 vertical profiles +- Standard times: 00Z and 12Z (bracketing QSO timestamps) +- Spatial matching: nearest station within 300 km of QSO path +- Raw profile: pressure, temperature, dewpoint, height per level +- Derived parameters (computed at ingestion by `SoundingParams.derive/1`): + - Surface refractivity (ITU-R P.453-14 formula) + - Minimum refractivity gradient (N/km) — primary ducting indicator + - Boundary layer depth (m) + - Precipitable water (mm) + - K-index, Lifted index — atmospheric stability + - Ducting detection and duct characteristics (height, strength in M-units) + +### HRRR Model Data + +**NOAA High-Resolution Rapid Refresh (HRRR)** +- Source: AWS S3 public bucket `https://noaa-hrrr-bdp-pds.s3.amazonaws.com` +- Format: GRIB2 files, hourly cadence, 3km horizontal resolution +- Profile count: 4,522 profiles matched to QSO/grid locations +- Pressure levels extracted: 1000, 975, 950, 925, 900, 850, 800, 700 mb +- Surface fields: temperature (°C), dewpoint (°C), pressure (mb), HPBL (boundary layer height, m), PWAT (precipitable water, mm), 10m wind components (u, v), cloud cover (%), precipitation (mm) +- Per-level fields: temperature, dewpoint, geopotential height +- Derived: refractivity profile, min gradient, ducting detection (same as sounding derivation) +- Batch optimization: groups multiple grid points by HRRR hour to download GRIB2 once per time step + +### IEMRE Gridded Hourly Weather + +**Iowa Environmental Mesonet Reanalysis (IEMRE)** +- API: `https://mesonet.agron.iastate.edu/iemre/hourly/{date}/{lat}/{lon}/json` +- Resolution: 0.125° (~14 km) gridded +- Observation count: 3,675 gridded hourly observations +- Fields per hour: air temperature (°F), dewpoint (°F), sky cover (%), wind components (u/v, m/s), hourly precipitation (inches), solar radiation +- Used for: weather at QSO endpoint grid points, more granular than nearest-ASOS matching +- Status: ingested but not yet integrated into scoring factors + +### Terrain Elevation Data + +**SRTM (Shuttle Radar Topography Mission)** +- Primary source: AWS S3 `https://elevation-tiles-prod.s3.amazonaws.com/skadi` (local tile cache) +- Resolution: 90m (SRTM3, 3 arc-second) +- Tile format: `.hgt` binary, 3601×3601 samples, 16-bit signed big-endian +- Fallback APIs (when tiles unavailable): + - Open-Meteo: `https://api.open-meteo.com/v1/elevation` + - OpenTopography: `https://api.opentopodata.org/v1/srtm90m` +- Profile method: 64 elevation samples per QSO path (great-circle interpolation) +- Path count: 58,276 QSO paths profiled +- Results: 56,658 BLOCKED (97.2%, avg 36.2 dB diffraction), 1,277 CLEAR (2.2%), 341 FRESNEL_PARTIAL (0.6%) +- Analysis: max elevation, min Fresnel clearance, knife-edge diffraction loss, obstruction count + +### Commercial Link Validation Data + +**Ubiquiti airFiber and airFiber 60 links near Princeton, TX** +- Link count: 7 commercial microwave links +- Frequencies: 11 GHz (AF11X, dual-chain), 24 GHz (AF11X), 68 GHz (AF60, single-chain) +- Polling: SNMP at 5-minute intervals (rx_power_0, rx_power_1 for dual-chain; rx_power for single) +- Weather correlation: KTKI ASOS station +- Historical dataset: March 14-29, 2026 (18,540 samples) +- Validated coefficients: 11 GHz and 24 GHz gaseous absorption, 68 GHz O2 band wing absorption (0.1 dB/km per g/m³ measured on 2.8 km path) +- Live polling: active for ongoing validation + +### Solar Indices + +**GFZ German Research Centre for Geosciences** +- Source: `https://kp.gfz.de/app/files/Kp_ap_Ap_SN_F107_since_1932.txt` +- Volume: 9,586 daily values (1998-2026) +- Fields: Solar Flux Index (F10.7), adjusted SFI, sunspot number, Ap index, Kp values (3-hourly) +- Status: ingested but NOT used in tropospheric scoring — reserved for potential VHF/sporadic-E extension + +### Live Scoring Data + +**Propagation Grid Scores** +- Coverage: CONUS grid at 0.125° resolution +- Update cadence: hourly (HRRR-based via `PropagationGridWorker`) + 10-minute ASOS adjustments (`AsosAdjustmentWorker`) +- Per grid point: composite score (0-100), 9 individual factor scores, valid_time +- Bands scored: all configured bands (10, 24, 47, 68, 75, 122, 134, 142, 241 GHz) +- Retention: 2 most recent valid_times kept, older data pruned automatically ## Known Data Quality Issues @@ -1324,3 +1424,5 @@ For each band: - **Sounding data recency**: Latest soundings are from Sep 2024. Ingestion pipeline may need restart for live enrichment. - **Surface obs density**: Historical obs are sparse (~1 per 4.7 days per station) because they were fetched per-QSO-window. Live polling is now active for continuous coverage. - **Weather codes (wx_codes)**: Stored in surface_observations but unused by scoring. Direct fog/thunderstorm/freezing-rain detection could supplement indirect inference from Td depression and sky condition. +- **Contest bias**: 97% of QSOs are from August-September ARRL Microwave Contest. Seasonal and regional statistics may not generalize to year-round conditions. +- **Rain model unvalidated**: ITU-R P.838-3 rain attenuation coefficients are theoretical — no rain events occurred in the commercial link validation dataset.