Rewrite meteorological foundations for technical audience

algo.md

@@ -31,165 +31,118 @@ The regime distinction matters because refractivity effects are *inverted* betwe
 
 ## Meteorological Foundations
 
-This section describes the atmospheric science underlying the propagation predictions, written for review by meteorologists. The system predicts microwave radio propagation conditions (10-241 GHz) using NWP model output and surface observations, focusing on tropospheric ducting and gaseous absorption.
+This section documents the atmospheric physics and NWP integration underlying the propagation prediction system. The system forecasts tropospheric microwave propagation conditions (10–241 GHz) from HRRR model output, RAOB soundings, and ASOS surface observations. The primary mechanisms of interest are tropospheric ducting via refractivity gradients and frequency-dependent gaseous/hydrometeor attenuation.
 
-### Why Meteorology Matters for Microwave Radio
+### Refractivity Framework
 
-Microwave radio signals (wavelengths 1 mm to 3 cm) travel through the troposphere and are affected by the same atmospheric structure that meteorologists analyze daily. Unlike HF radio which depends on the ionosphere, microwave propagation is entirely a tropospheric phenomenon governed by:
-
-- **Temperature and moisture profiles** in the lowest 2 km AGL
-- **Boundary layer structure** (mixed layer depth, capping inversions, nocturnal stable layers)
-- **Synoptic-scale weather patterns** (frontal boundaries, subsidence, marine layers)
-- **Gaseous composition** (water vapor and O2 density along the path)
-
-Under standard atmospheric conditions, microwave signals are limited to roughly line-of-sight range (~50 km for ground-level antennas). Enhanced propagation occurs when atmospheric structure bends the radio beam downward (super-refraction) or traps it in a waveguide (ducting), extending range to hundreds or thousands of kilometers.
-
-### The Refractivity-Propagation Connection
-
-Radio refractivity N is the key bridge between meteorology and propagation. N is computed from standard meteorological variables:
+Radio refractivity N (ITU-R P.453-14):
 
 ```
-N = 77.6 * P/T + 3.73×10⁵ * e/T²
+N = 77.6·P/T + 3.73×10⁵·e/T²
 ```
 
-where P is pressure (hPa), T is temperature (K), and e is water vapor pressure (hPa). The first term is the "dry" component (pressure/temperature), the second is the "wet" component (moisture). At microwave frequencies, both terms matter — the wet term contributes roughly 20-40% of total N in the boundary layer.
+P (hPa), T (K), e = water vapor pressure (hPa) via Buck equation from Td. The wet term contributes 20–40% of total N in the ABL. Modified refractivity M = N + 0.157·h (h in m AGL); ducting where dM/dh < 0.
 
-The **vertical gradient of N** (dN/dh) determines how radio beams bend:
+The vertical gradient dN/dh governs ray curvature via the effective earth radius factor k = 1/(1 + R·dN/dh·10⁻⁶):
 
-| dN/dh (N-units/km) | Propagation Regime | Meteorological Condition |
+| dN/dh (N/km) | k | Regime |
 |---|---|---|
-| > 0 | Sub-refraction (beam bends upward) | Strong lapse, well-mixed dry BL |
-| 0 to −79 | Normal refraction | Standard atmosphere |
-| −79 to −157 | Super-refraction (beam bends toward earth) | Moderate inversion or moisture lapse |
-| < −157 | Ducting (beam trapped in waveguide) | Strong temperature inversion with moisture contrast |
-
-The critical threshold is **−157 N/km**: below this, the radio beam curves faster than the Earth's surface, creating a tropospheric duct.
+| > 0 | < 1 | Sub-refraction |
+| 0 to −79 | 1.0–1.33 | Standard |
+| −79 to −157 | 1.33–∞ | Super-refraction |
+| < −157 | negative | Ducting (ray curvature exceeds earth curvature) |
 
 ### Ducting Mechanisms
 
-Four meteorological mechanisms produce ducting conditions relevant to this system. Each creates a sharp negative refractivity gradient through a different combination of temperature increase and/or moisture decrease with height:
+Four mechanisms produce the negative dN/dh gradients that enable beyond-LOS propagation. Each creates a sharp temperature increase and/or moisture decrease with height:
 
-**1. Radiation/Nocturnal Inversion Ducts**
-- **Mechanism**: Radiative cooling of the surface after sunset creates a strong surface-based temperature inversion. Moisture is often trapped below the inversion cap.
-- **Meteorological signature**: Clear skies, light winds, high pressure. Surface temperature drops rapidly after sunset while air aloft remains warm.
-- **Timing**: Forms 1-3 hours after sunset, strongest just before sunrise. Dissipates within 1-2 hours of solar heating.
-- **Vertical extent**: Typically 50-300m AGL (shallow boundary layer).
-- **dN/dh**: −100 to −300 N/km in the inversion layer.
-- **This is the primary ducting mechanism in our dataset** — the majority of amateur microwave contacts occur during dawn/sunrise windows specifically to exploit these ducts.
+**1. Radiation (Nocturnal) Ducts** — Surface-based temperature inversion from radiative cooling under clear skies, light winds. Moisture trapped below the inversion cap. Forms 1–3h post-sunset, peaks pre-dawn, erodes within 1–2h of insolation. Typical depth 50–300m AGL, dN/dh −100 to −300 N/km. **Primary mechanism in our dataset** — operators target dawn windows specifically.
 
-**2. Advection Ducts (Marine/Coastal)**
-- **Mechanism**: Warm, dry air advects over a cooler surface (ocean, lake, or cold ground). Temperature increases with height while moisture decreases sharply at the air-mass boundary.
-- **Meteorological signature**: Onshore warm-air advection, sea breeze fronts, warm sector of extratropical cyclones passing over cool water.
-- **Timing**: Can persist for hours to days, not limited to diurnal cycle.
-- **Vertical extent**: 50-500m, can be elevated.
-- **dN/dh**: −200 to −500+ N/km. Among the strongest ducts observed.
-- **Relevance**: Gulf Coast, California coast, and Great Lakes regions. Our dataset shows Florida/Gulf Coast (EL) and California (CM) outperform inland regions at 10 GHz.
+**2. Advection Ducts** — Warm, dry air mass overriding a cooler surface (SST discontinuity, lake, post-frontal cold ground). Strong temperature increase with moisture decreasing sharply at the air-mass interface. Persistent (hours to days), independent of diurnal cycle. Depth 50–500m, often elevated. dN/dh −200 to −500+ N/km. Dominant along Gulf Coast, California coast, Great Lakes.
 
-**3. Subsidence Inversion Ducts**
-- **Mechanism**: Large-scale sinking motion (subsidence) in high-pressure systems creates a temperature inversion aloft. Moisture below the subsidence inversion is trapped, creating a sharp dN/dh gradient at the inversion base.
-- **Meteorological signature**: Ridge axis or high-pressure center, clear skies above inversion, possibly hazy/humid below. Visible on soundings as a sharp temperature increase at 800-900 hPa with dewpoint drop above.
-- **Timing**: Persistent, follows synoptic-scale ridging.
-- **Vertical extent**: 500-2000m AGL (elevated duct).
-- **dN/dh**: −100 to −200 N/km (weaker than advection ducts but larger scale).
+**3. Subsidence Ducts** — Synoptic-scale subsidence inversion (ridge axis, subtropical high) with trapped moisture below the inversion base. Visible on soundings as sharp temperature increase at 800–900 hPa with coincident Td drop. Persistent with ridging. Depth 500–2000m AGL (elevated duct). dN/dh −100 to −200 N/km.
 
-**4. Frontal/Boundary Ducts**
-- **Mechanism**: Temperature and moisture gradients along frontal boundaries, outflow boundaries, and drylines create localized refractivity gradients.
-- **Meteorological signature**: Cold fronts (sharp moisture drop behind front), warm fronts (overrunning warm moist air), dryline passages.
-- **Timing**: Transient, associated with frontal passage.
-- **Our data confirms**: Low-pressure systems correlate with longer propagation distances (262 km avg vs 196 km for high pressure at 10 GHz), likely due to frontal boundary structure rather than pressure itself.
+**4. Frontal/Boundary Ducts** — Mesoscale refractivity gradients along cold fronts (post-frontal moisture drop), warm fronts (overrunning moist air), outflow boundaries, and drylines. Transient. Our data confirms low-pressure systems correlate with extended propagation (262 km avg vs 196 km at 1025+ hPa for 10 GHz), consistent with frontal boundary structure.
 
-### HRRR Model Data in the Prediction System
+### NWP Integration (HRRR)
 
-The system ingests NOAA's High-Resolution Rapid Refresh (HRRR) model output as its primary atmospheric data source:
+Primary atmospheric data source: NOAA HRRR v4 (3 km, hourly, 18h forecast cycle).
 
-- **Model**: HRRR v4, 3 km horizontal resolution, hourly runs, 18-hour forecasts
-- **Source**: NOAA AWS S3 archive (GRIB2 format)
-- **Extracted fields**:
-  - Surface: 2m temperature, 2m dewpoint, surface pressure, 10m wind (u/v), total cloud cover, precipitation
-  - Boundary layer: HPBL (planetary boundary layer height), PWAT (precipitable water)
-  - Upper air: temperature, dewpoint, and geopotential height at 1000, 975, 950, 925, 900, 850, 800, 700 hPa
+**Extracted fields (GRIB2 via byte-range requests from AWS S3):**
+- Surface: T₂ₘ, Td₂ₘ, Psfc, U₁₀ₘ/V₁₀ₘ, TCDC, APCP
+- ABL: HPBL (diagnosed PBL height), PWAT (column-integrated precipitable water)
+- Pressure levels: T, Td, Z at every 25 hPa from 1000–700 hPa (13 levels, ~80m vertical spacing below 900 hPa)
 
-From these fields, the system derives:
+**Derived products:**
+1. N(h) profile at each pressure level → dN/dh minimum (primary ducting discriminant)
+2. M(h) profile → explicit duct detection (dM/dh < 0 layers with strength Δ M > 2 M-units)
+3. Surface N from Psfc, T₂ₘ, Td₂ₘ
+4. Absolute humidity ρ = 217·es(Td)/T
+5. Dynamic k-factor for terrain diffraction (ITU-R P.526-16)
 
-1. **Surface refractivity N** (ITU-R P.453-14 formula)
-2. **Refractivity profile** N(h) at each pressure level
-3. **Minimum refractivity gradient** dN/dh — the primary ducting indicator
-4. **Modified refractivity** M(h) = N + 0.157h for duct detection (ducting where dM/dh < 0)
-5. **Absolute humidity** from temperature and dewpoint
-6. **Effective earth radius factor k** for terrain diffraction analysis
-
-**HRRR vs radiosonde comparison**: HRRR profiles have 8 standard pressure levels in the lowest 3 km, while radiosondes report at ~10m vertical resolution. This means HRRR can detect large-scale inversions but may miss thin ducting layers (<100m). Our gradient statistics reflect this: HRRR-derived gradients cluster between −40 and −130 N/km (median −70), while radiosonde-derived gradients extend to −500+ N/km. The scoring thresholds are calibrated separately for HRRR data.
+**Vertical resolution limitation:** The 25 hPa pressure level spacing resolves features ≥100m thick. Thin surface ducts (50–100m) detectable by RAOB at ~10m resolution may be missed. Comparative statistics: HRRR gradients cluster −40 to −130 N/km (median −70); collocated RAOB gradients extend to −500+ N/km. Scoring thresholds are calibrated to HRRR-derived gradient distributions, with RAOB duct detections used as supplementary data where available (3,901 profiles from 112 stations).
 
 ### Surface Observations (ASOS)
 
-ASOS stations provide ground-truth observations that complement the HRRR model:
+ASOS provides in-situ validation and additional parameters not in HRRR output:
+- T, Td → ρ(abs humidity), T-Td depression (inversion proxy)
+- Wind speed → mechanical mixing potential (light winds favor inversion persistence)
+- Sky condition → longwave radiation budget (CLR promotes nocturnal cooling)
+- Psfc → absolute N computation, barometric trend (pressure tendency)
+- Precipitation type/rate → hydrometeor attenuation (ITU-R P.838-3)
 
-- **Temperature and dewpoint**: Used to compute absolute humidity and T-Td depression
-- **Wind speed**: Light winds favor inversion persistence; strong winds indicate mechanical mixing
-- **Sky condition**: Clear skies promote radiative cooling (nocturnal inversions); overcast prevents it
-- **Pressure**: Used for absolute refractivity calculation and trend analysis
-- **Precipitation**: Rain causes direct path attenuation (ITU-R P.838-3), scaling with frequency
+### Frequency-Dependent Attenuation
 
-### Frequency-Dependent Atmospheric Effects
+The same thermodynamic state produces opposing propagation effects across the spectrum:
 
-A critical meteorological nuance: the same atmospheric state produces opposite effects at different frequencies.
+**10 GHz (λ = 3 cm):** Gaseous attenuation negligible (0.012 dB/km). Propagation governed entirely by refractivity structure. Increased ρ raises N, enhancing beam bending — moisture is beneficial. Rain attenuation mild (γR = 0.05 dB/km at 4 mm/hr).
 
-**At 10 GHz (3 cm wavelength)**:
-- Gaseous absorption is negligible (0.012 dB/km total)
-- Propagation is dominated by refractivity structure — ducts and inversions
-- More moisture increases N, increasing beam bending — **humidity helps**
-- Rain attenuation is mild (0.05 dB/km at 4 mm/hr)
+**24 GHz (λ = 1.25 cm):** Proximal to the 22.235 GHz H₂O rotational line. H₂O absorption coefficient 0.012 dB/km per g/m³ (10× the 10 GHz rate). Moisture degrades path budget despite refractivity benefit — net effect is harmful. Rain attenuation 6× that of 10 GHz.
 
-**At 24 GHz (1.25 cm wavelength)**:
-- Near the 22.235 GHz water vapor absorption line — humidity causes significant loss
-- 0.012 dB/km per g/m³ of water vapor (10x the rate at 10 GHz)
-- **Humidity hurts** — dry air allows longer paths despite reduced ducting benefit
-- Rain attenuation 6x higher than 10 GHz
+**47–75 GHz:** Transition regime. 47 GHz in a relative window (O₂ wing + mild H₂O). 68 GHz on the wing of the 60 GHz O₂ absorption complex (γO₂ = 0.9 dB/km). 75 GHz in a window band (γtotal ≈ 0.057 dB/km). Ducting is the sole mechanism enabling paths beyond ~20 km.
 
-**At 47+ GHz (sub-cm wavelength)**:
-- Increasingly dominated by gaseous absorption
-- 68 GHz is near the 60 GHz O2 absorption complex (0.9 dB/km baseline)
-- 241 GHz is between H2O lines at 183 and 325 GHz (0.3 dB/km per g/m³)
-- Propagation windows exist at 75 GHz and 134 GHz
-- **Ducting is the only mechanism** enabling contacts beyond ~20 km at these frequencies
+**122–241 GHz:** Dominated by gaseous absorption. 122 GHz on the 118.75 GHz O₂ line wing (0.8 dB/km). 134 GHz in the window between O₂ 118 and H₂O 183 lines. 241 GHz between H₂O 183 and 325 GHz lines (0.3 dB/km per g/m³). All contacts in the dataset above 122 GHz are CW mode — link budget is that tight.
 
-### Diurnal Cycle and Timing
+### ABL Diurnal Cycle
 
-The boundary layer diurnal cycle is the most predictable propagation driver:
+The ABL diurnal cycle is the most predictable propagation driver:
 
-1. **Sunset to sunrise**: Radiative cooling builds surface inversions. BL collapses from 1000-2000m (afternoon) to 100-300m. Ducting probability rises.
-2. **Dawn (sunrise ± 1.5h)**: Peak inversion strength. BL is shallowest. This is the preferred operating window for amateur microwave contacts.
-3. **Morning transition (sunrise + 1.5 to +3h)**: Solar heating erodes inversion from below. Ducting weakens.
-4. **Afternoon (sunrise + 6h onward)**: Full convective mixing. BL deepest. Worst propagation conditions. Turbulent scattering dominates any residual refractivity gradients.
+1. **Post-sunset → sunrise:** Radiative cooling develops surface-based inversion. HPBL collapses from O(10³m) to O(10²m). Super-refractive/ducting conditions develop.
+2. **Dawn (sunrise ± 1.5h):** Peak inversion strength, minimum HPBL. Maximum ducting probability.
+3. **Morning transition (+1.5 to +3h):** Shortwave heating erodes inversion from below. Convective mixing deepens ABL.
+4. **Afternoon (+6h):** Fully convective ABL, maximum HPBL. Minimum propagation. Turbulent scattering dominates residual refractivity gradients.
 
-Our data quantifies this diurnal effect by frequency: the night/dawn enhancement over afternoon baseline is +4% at 10 GHz, +28% at 24 GHz, +36% at 47 GHz, and +360% at 75 GHz. At the highest frequencies, **time of day is the dominant predictor** — more important than any single weather variable.
+Observed diurnal enhancement (night/dawn over afternoon baseline): +4% at 10 GHz, +28% at 24 GHz, +36% at 47 GHz, +360% at 75 GHz. At EHF, time of day dominates all other predictors. The system uses longitude-based solar time (hour + lon/15) rather than UTC for diurnal scoring — Spearman ρ with distance improves from 0.056 (UTC) to 0.188 (solar) at 24 GHz.
 
 ### Seasonal Cycle
 
-Ducting probability from our radiosonde analysis (3,901 profiles, continental US):
+Ducting probability from 3,901 RAOB profiles (CONUS):
 
-- **Peak**: June-July (69-77% ducting frequency). Warm, moist boundary layers with strong nocturnal inversions.
-- **Secondary peak**: October-November (55-65%). Autumn radiative cooling with residual moisture.
-- **Minimum**: March (10.8%). Transitional season with frequent mixing events.
-- **Winter**: December-February (12-20%). Cold, dry atmosphere limits refractivity gradients.
+| Month | Ducting % | Mean dN/dh min | Mean PWAT (mm) |
+|-------|-----------|----------------|----------------|
+| Jun | 68.7% | −323 | 28.6 |
+| Jul | 76.5% | −301 | 27.6 |
+| Aug | 53.9% | −261 | 33.3 |
+| Sep | 56.4% | −287 | 26.2 |
+| Oct | 60.4% | −314 | 17.9 |
+| Mar | 10.8% | −113 | 6.6 |
+| Dec–Feb | 12–22% | −130 | 7–10 |
 
-At 24+ GHz, the seasonal pattern inverts: **winter is best** because low absolute humidity minimizes absorption, even though ducting is less frequent. The system's band-specific seasonal scoring reflects this inversion.
+Peak Jun–Jul, secondary peak Oct–Nov (autumn radiative cooling with residual moisture). March minimum (frequent mixing events). At 24+ GHz the seasonal optimum inverts — winter minimizes H₂O absorption despite lower ducting probability. Band-specific seasonal weights account for this.
 
-### Known Limitations and Areas for Meteorologist Review
+### Known Limitations
 
-1. **HRRR vertical resolution**: 8 pressure levels below 700 hPa may miss thin ducting layers. Soundings show gradients 3-5x stronger than HRRR at the same locations. The system uses HRRR-calibrated thresholds, but this may underestimate ducting potential.
+1. **HRRR vertical resolution:** 25 hPa spacing (recently improved from ~100 hPa) may still miss thin surface ducts <100m. RAOB data used as supplementary duct detection source.
 
-2. **Mesoscale effects**: Sea breeze fronts, outflow boundaries, and terrain-induced convergence zones can create localized ducts not resolved by the 3 km HRRR grid. The system does not currently account for these sub-grid phenomena.
+2. **Sub-grid mesoscale:** Sea breeze fronts, outflow boundaries, terrain-induced convergence zones — ducting features below the 3 km HRRR grid are not resolved.
 
-3. **Precipitation model**: Rain attenuation uses ITU-R P.838-3 coefficients but has not been validated against measured data (no rain events in the commercial link dataset). Rainscatter (using rain as a scattering medium for propagation) is observed at 24 GHz but not modeled.
+3. **Hydrometeor attenuation:** ITU-R P.838-3 coefficients applied but unvalidated against measured data. Rainscatter propagation (observed at 24 GHz via FM mode) is not modeled.
 
-4. **Fog and cloud effects**: The system uses cloud cover percentage as a sky condition proxy but does not model fog/cloud attenuation directly. At 10 GHz, fog attenuation is negligible; at 75+ GHz, dense fog can add 1-5 dB/km.
+4. **Fog/cloud:** Cloud cover percentage is used as a proxy; direct fog/cloud attenuation modeling is not implemented. Relevant above 47 GHz where dense fog adds 1–5 dB/km.
 
-5. **Turbulence scintillation**: Not modeled. On long LOS paths, convective turbulence causes amplitude scintillation (rapid signal fluctuations) that is distinct from multipath fading.
+5. **Scintillation:** Amplitude scintillation from refractive turbulence on long LOS paths is not modeled.
 
-6. **Time resolution**: The system updates hourly using HRRR analysis/forecast. Rapidly evolving mesoscale events (convective outflows, sea breeze onset) may be captured by the forecast hours but with reduced accuracy beyond the analysis hour.
-
-We welcome feedback from operational meteorologists on these assumptions and any atmospheric processes we may be underweighting or overlooking.
+6. **Temporal resolution:** Hourly HRRR updates. Rapidly evolving mesoscale features (convective outflows, sea breeze onset) may lag reality between analysis cycles.
 
 ---