graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Expand algo.md sources: full data provenance, APIs, and ITU-R references

Browse source 
- Expand ITU-R section with recommendation numbers, titles, and usage
- Document all 8 external data sources with API URLs, volumes, fields
- Add derived parameter documentation (sounding params, HRRR profiles)
- Document live scoring pipeline (grid resolution, update cadence, retention)
- Add contest bias and rain model caveats to known issues
This commit is contained in:
Graham McIntire 2026-03-31 10:52:45 -05:00
parent a18cf48493
commit d458c23782
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
1 changed files with 118 additions and 16 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

134
algo.md
View file

@ -1297,25 +1297,125 @@ For each band:
| Signal prediction floor | +/- 2-3 dB | Measured from link data |
| CW bandwidth advantage | ~7 dB | 10*log10(2700/500); 24% range increase |
## ITU-R References
## ITU-R Recommendations
- P.453-14: Radio refractivity
- P.525: Free-space path loss
- P.676-13: Gaseous absorption (O2 + H2O)
- P.838-3: Specific rain attenuation
- P.526: Diffraction
- P.452: Interference between stations
The following ITU-R Recommendations provide the physics models underlying the scoring algorithm. All are publicly available from the ITU Radiocommunication Sector (https://www.itu.int/rec/R-REC-P/en).
## Data References
| Recommendation | Title | Used For |
|----------------|-------|----------|
| ITU-R P.453-14 | The radio refractive index: its formula and refractivity data | Surface refractivity N calculation, refractivity gradient |
| ITU-R P.525-4 | Calculation of free-space attenuation | Free-space path loss baseline |
| ITU-R P.676-13 | Attenuation by atmospheric gases and related effects | O2 and H2O absorption coefficients per band |
| ITU-R P.838-3 | Specific attenuation model for rain for use in prediction methods | Rain attenuation coefficients (k, alpha) per band |
| ITU-R P.526-15 | Propagation by diffraction | Knife-edge and terrain diffraction loss |
| ITU-R P.452-17 | Prediction procedure for the evaluation of interference between stations on the surface of the Earth | Clear-air propagation modeling framework |
| ITU-R P.835-6 | Reference standard atmospheres | Standard atmosphere profiles for baseline |
| ITU-R P.530-18 | Propagation data and prediction methods for terrestrial line-of-sight systems | Multipath fading and enhancement statistics |
- ARRL Microwave Contest QSO database: 58,282 total, 57,488 tropospheric with distance (1992-2024)
- IEM ASOS observations: 1,299 stations, 58,398 surface observations
- IEM RAOB soundings: 112 stations, 3,901 soundings
- IEMRE gridded hourly: 3,675 observations at 0.125° resolution
- HRRR model profiles: 4,522 profiles with refractivity, BL depth, PWAT
- Terrain profiles: 58,276 QSO paths (97.2% blocked, avg 36.2 dB diffraction)
- Commercial link data: 7 links at 11/24/68 GHz near DFW, SNMP-polled
- Solar indices: 9,586 daily values (1998-2026) — reserved for future use
## Data Sources
### Primary QSO Dataset
**ARRL Microwave Contest Results (1992-2024)**
- Source: Contest log submissions compiled from ARRL contest results
- Volume: 58,282 total QSOs across 13+ bands (10 GHz through 403 GHz)
- Usable subset: 57,488 tropospheric QSOs after filtering 4 EME contacts (QRA64D/JT4F modes >3,000 km)
- Fields: station callsigns, Maidenhead grid squares, timestamp, mode (CW/SSB/FM/FT8), band
- Grid-to-coordinate conversion: gridmap.org API for Maidenhead → lat/lon
- Distance: Haversine great-circle calculation from grid square centers
### Surface Weather Observations (ASOS)
**Iowa Environmental Mesonet (IEM) — Automated Surface Observing System**
- API: `https://mesonet.agron.iastate.edu/cgi-bin/request/asos.py`
- Network discovery: `https://mesonet.agron.iastate.edu/api/1/network.py`
- Station count: 2,922 total, 1,299 with observations matched to QSOs
- Observation count: 58,398 surface observations
- Fields: temperature (°F), dewpoint (°F), relative humidity (%), wind speed (kts), wind direction (°), sea level pressure (mb), sky condition (CLR/FEW/SCT/BKN/OVC), precipitation (inches/hour), weather codes
- Temporal matching: ±2 hours around QSO timestamp
- Spatial matching: nearest station within 150 km of QSO path endpoints
- No authentication required
### Upper-Air Soundings (RAOB)
**Iowa Environmental Mesonet (IEM) — Radiosonde Observations**
- API: `https://mesonet.agron.iastate.edu/json/raob.py`
- Station count: 346 sounding stations total, 112 with data matched to QSOs
- Sounding count: 3,901 vertical profiles
- Standard times: 00Z and 12Z (bracketing QSO timestamps)
- Spatial matching: nearest station within 300 km of QSO path
- Raw profile: pressure, temperature, dewpoint, height per level
- Derived parameters (computed at ingestion by `SoundingParams.derive/1`):
- Surface refractivity (ITU-R P.453-14 formula)
- Minimum refractivity gradient (N/km) — primary ducting indicator
- Boundary layer depth (m)
- Precipitable water (mm)
- K-index, Lifted index — atmospheric stability
- Ducting detection and duct characteristics (height, strength in M-units)
### HRRR Model Data
**NOAA High-Resolution Rapid Refresh (HRRR)**
- Source: AWS S3 public bucket `https://noaa-hrrr-bdp-pds.s3.amazonaws.com`
- Format: GRIB2 files, hourly cadence, 3km horizontal resolution
- Profile count: 4,522 profiles matched to QSO/grid locations
- Pressure levels extracted: 1000, 975, 950, 925, 900, 850, 800, 700 mb
- Surface fields: temperature (°C), dewpoint (°C), pressure (mb), HPBL (boundary layer height, m), PWAT (precipitable water, mm), 10m wind components (u, v), cloud cover (%), precipitation (mm)
- Per-level fields: temperature, dewpoint, geopotential height
- Derived: refractivity profile, min gradient, ducting detection (same as sounding derivation)
- Batch optimization: groups multiple grid points by HRRR hour to download GRIB2 once per time step
### IEMRE Gridded Hourly Weather
**Iowa Environmental Mesonet Reanalysis (IEMRE)**
- API: `https://mesonet.agron.iastate.edu/iemre/hourly/{date}/{lat}/{lon}/json`
- Resolution: 0.125° (~14 km) gridded
- Observation count: 3,675 gridded hourly observations
- Fields per hour: air temperature (°F), dewpoint (°F), sky cover (%), wind components (u/v, m/s), hourly precipitation (inches), solar radiation
- Used for: weather at QSO endpoint grid points, more granular than nearest-ASOS matching
- Status: ingested but not yet integrated into scoring factors
### Terrain Elevation Data
**SRTM (Shuttle Radar Topography Mission)**
- Primary source: AWS S3 `https://elevation-tiles-prod.s3.amazonaws.com/skadi` (local tile cache)
- Resolution: 90m (SRTM3, 3 arc-second)
- Tile format: `.hgt` binary, 3601×3601 samples, 16-bit signed big-endian
- Fallback APIs (when tiles unavailable):
- Open-Meteo: `https://api.open-meteo.com/v1/elevation`
- OpenTopography: `https://api.opentopodata.org/v1/srtm90m`
- Profile method: 64 elevation samples per QSO path (great-circle interpolation)
- Path count: 58,276 QSO paths profiled
- Results: 56,658 BLOCKED (97.2%, avg 36.2 dB diffraction), 1,277 CLEAR (2.2%), 341 FRESNEL_PARTIAL (0.6%)
- Analysis: max elevation, min Fresnel clearance, knife-edge diffraction loss, obstruction count
### Commercial Link Validation Data
**Ubiquiti airFiber and airFiber 60 links near Princeton, TX**
- Link count: 7 commercial microwave links
- Frequencies: 11 GHz (AF11X, dual-chain), 24 GHz (AF11X), 68 GHz (AF60, single-chain)
- Polling: SNMP at 5-minute intervals (rx_power_0, rx_power_1 for dual-chain; rx_power for single)
- Weather correlation: KTKI ASOS station
- Historical dataset: March 14-29, 2026 (18,540 samples)
- Validated coefficients: 11 GHz and 24 GHz gaseous absorption, 68 GHz O2 band wing absorption (0.1 dB/km per g/m³ measured on 2.8 km path)
- Live polling: active for ongoing validation
### Solar Indices
**GFZ German Research Centre for Geosciences**
- Source: `https://kp.gfz.de/app/files/Kp_ap_Ap_SN_F107_since_1932.txt`
- Volume: 9,586 daily values (1998-2026)
- Fields: Solar Flux Index (F10.7), adjusted SFI, sunspot number, Ap index, Kp values (3-hourly)
- Status: ingested but NOT used in tropospheric scoring — reserved for potential VHF/sporadic-E extension
### Live Scoring Data
**Propagation Grid Scores**
- Coverage: CONUS grid at 0.125° resolution
- Update cadence: hourly (HRRR-based via `PropagationGridWorker`) + 10-minute ASOS adjustments (`AsosAdjustmentWorker`)
- Per grid point: composite score (0-100), 9 individual factor scores, valid_time
- Bands scored: all configured bands (10, 24, 47, 68, 75, 122, 134, 142, 241 GHz)
- Retention: 2 most recent valid_times kept, older data pruned automatically
## Known Data Quality Issues
@ -1324,3 +1424,5 @@ For each band:
- **Sounding data recency**: Latest soundings are from Sep 2024. Ingestion pipeline may need restart for live enrichment.
- **Surface obs density**: Historical obs are sparse (~1 per 4.7 days per station) because they were fetched per-QSO-window. Live polling is now active for continuous coverage.
- **Weather codes (wx_codes)**: Stored in surface_observations but unused by scoring. Direct fog/thunderstorm/freezing-rain detection could supplement indirect inference from Td depression and sky condition.
- **Contest bias**: 97% of QSOs are from August-September ARRL Microwave Contest. Seasonal and regional statistics may not generalize to year-round conditions.
- **Rain model unvalidated**: ITU-R P.838-3 rain attenuation coefficients are theoretical — no rain events occurred in the commercial link validation dataset.