graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update algo.md to match current codebase

Browse source 
- Fix pressure scoring thresholds (finer granularity at <980-990 mb)
- Fix 10 GHz parameters (o2_db_km, h2o_coeff, seasonal Feb/Oct)
- Fix 24 GHz parameters (o2_db_km 0.02, h2o_coeff 0.002)
- Fix 47 GHz o2_db_km (0.04)
- Add 5 new UHF/microwave bands (902-5760 MHz)
- Add Part 12: HRRR native hybrid-sigma levels (50-level profiles)
- Add Part 13: Duct analysis (M-profile, trapped frequency)
- Add Part 14: Inversion analysis (Ri, theta-e, shear)
- Add Part 15: NEXRAD radar data integration
- Add Part 16: Backtest framework (21 features)
This commit is contained in:
Graham McIntire 2026-04-11 10:19:07 -05:00
parent f4546ad1f3
commit c8127fbb02
1 changed files with 380 additions and 17 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

397
algo.md
View file

@ -2,7 +2,7 @@
## Overview ## Overview
Propagation scoring and prediction for microwave amateur radio bands (10-241 GHz), calibrated against 57,488 tropospheric QSOs with distance data, validated against commercial terrestrial links at 11/24/68 GHz, and grounded in ITU-R atmospheric models. Propagation scoring and prediction for amateur radio bands from 902 MHz through 241 GHz, calibrated against 57,488 tropospheric QSOs with distance data, validated against commercial terrestrial links at 11/24/68 GHz, and grounded in ITU-R atmospheric models.
The algorithm has two operating regimes: The algorithm has two operating regimes:
1. **Beyond-LOS** — Ham radio paths (50-1000+ km) where atmospheric ducting and refraction are essential. This is the primary use case. Terrain analysis confirms **97.2% of all QSO paths are terrain-blocked** (average diffraction loss 36.2 dB), making atmospheric propagation mechanisms the sole enabler. 1. **Beyond-LOS** — Ham radio paths (50-1000+ km) where atmospheric ducting and refraction are essential. This is the primary use case. Terrain analysis confirms **97.2% of all QSO paths are terrain-blocked** (average diffraction loss 36.2 dB), making atmospheric propagation mechanisms the sole enabler.
@ -222,9 +222,14 @@ Total absorption per km = O2 component (fixed) + H2O component (humidity-depende
| Band | f (GHz) | O2 (dB/km) | H2O Coeff (dB/km per g/m^3) | Total @ 7.5 g/m^3 | Dominant Constraint | | Band | f (GHz) | O2 (dB/km) | H2O Coeff (dB/km per g/m^3) | Total @ 7.5 g/m^3 | Dominant Constraint |
|------|---------|------------|---------------------------|-------------------|---------------------| |------|---------|------------|---------------------------|-------------------|---------------------|
| 10G | 10.368 | 0.008 | 0.0005 | 0.012 | Negligible absorption | | 902M | 0.902 | 0.006 | 0.0 | 0.006 | Negligible absorption |
| 24G | 24.192 | 0.015 | 0.012 | 0.105 | 22.235 GHz H2O line | | 1296M | 1.296 | 0.006 | 0.0 | 0.006 | Negligible absorption |
| 47G | 47.088 | 0.045 | 0.003 | 0.068 | O2 wing + mild H2O | | 2304M | 2.304 | 0.006 | 0.0 | 0.006 | Negligible absorption |
| 3456M | 3.456 | 0.006 | 0.0 | 0.006 | Negligible absorption |
| 5760M | 5.760 | 0.007 | 0.0 | 0.007 | Negligible absorption |
| 10G | 10.368 | 0.007 | 0.0 | 0.007 | Negligible absorption |
| 24G | 24.192 | 0.02 | 0.002 | 0.035 | 22.235 GHz H2O line |
| 47G | 47.088 | 0.04 | 0.003 | 0.063 | O2 wing + mild H2O |
| 68G | 68.040 | 0.90 | 0.007 | 0.95 | 60 GHz O2 band wing | | 68G | 68.040 | 0.90 | 0.007 | 0.95 | 60 GHz O2 band wing |
| 75G | 76.032 | 0.012 | 0.006 | 0.057 | Window band | | 75G | 76.032 | 0.012 | 0.006 | 0.057 | Window band |
| 122G | 122.250 | 0.80 | 0.010 | 0.875 | 118.75 GHz O2 wing | | 122G | 122.250 | 0.80 | 0.010 | 0.875 | 118.75 GHz O2 wing |
@ -240,6 +245,11 @@ gamma_R = k * R^alpha (dB/km), R = rain rate (mm/hr):
| Band | k_H | alpha_H | Light 4mm/hr | Moderate 10mm/hr | Heavy 25mm/hr | | Band | k_H | alpha_H | Light 4mm/hr | Moderate 10mm/hr | Heavy 25mm/hr |
|------|------|------|-------------|-----------------|--------------| |------|------|------|-------------|-----------------|--------------|
| 902M | 0.000 | 1.00 | 0.00 | 0.00 | 0.00 |
| 1296M | 0.000 | 1.00 | 0.00 | 0.00 | 0.00 |
| 2304M | 0.001 | 1.15 | 0.005 | 0.01 | 0.04 |
| 3456M | 0.002 | 1.20 | 0.01 | 0.03 | 0.08 |
| 5760M | 0.005 | 1.25 | 0.02 | 0.09 | 0.24 |
| 10G | 0.010 | 1.28 | 0.05 | 0.19 | 0.56 | | 10G | 0.010 | 1.28 | 0.05 | 0.19 | 0.56 |
| 24G | 0.070 | 1.07 | 0.31 | 0.81 | 2.04 | | 24G | 0.070 | 1.07 | 0.31 | 0.81 | 2.04 |
| 47G | 0.187 | 0.93 | 0.68 | 1.58 | 3.69 | | 47G | 0.187 | 0.93 | 0.68 | 1.58 | 3.69 |
@ -620,18 +630,93 @@ Binned distance analysis of 37,925 HRRR-matched contacts confirms and refines Pa
```elixir ```elixir
@band_configs %{ @band_configs %{
902 => %{
label: "902 MHz",
o2_db_km: 0.006,
h2o_coeff: 0.0,
humidity_effect: :beneficial, # Sub-GHz: refractivity dominates, absorption negligible
humidity_penalty: 0.0,
rain_k: 0.000, rain_alpha: 1.0, # Rain attenuation negligible at 900 MHz
seasonal_base: %{1 => 38, 2 => 40, 3 => 22, 4 => 55, 5 => 68,
6 => 90, 7 => 95, 8 => 75, 9 => 78, 10 => 88,
11 => 78, 12 => 25},
seasonal_adj: %{},
typical_range_km: 400,
extended_range_km: 800,
exceptional_range_km: 1500
},
1_296 => %{
label: "1296 MHz",
o2_db_km: 0.006,
h2o_coeff: 0.0,
humidity_effect: :beneficial,
humidity_penalty: 0.0,
rain_k: 0.000, rain_alpha: 1.0,
seasonal_base: %{1 => 38, 2 => 40, 3 => 22, 4 => 55, 5 => 68,
6 => 90, 7 => 95, 8 => 75, 9 => 78, 10 => 88,
11 => 78, 12 => 25},
seasonal_adj: %{},
typical_range_km: 350,
extended_range_km: 700,
exceptional_range_km: 1200
},
2_304 => %{
label: "2304 MHz",
o2_db_km: 0.006,
h2o_coeff: 0.0,
humidity_effect: :beneficial,
humidity_penalty: 0.0,
rain_k: 0.001, rain_alpha: 1.15, # Onset of rain sensitivity
seasonal_base: %{1 => 38, 2 => 40, 3 => 22, 4 => 55, 5 => 68,
6 => 90, 7 => 95, 8 => 75, 9 => 78, 10 => 88,
11 => 78, 12 => 25},
seasonal_adj: %{},
typical_range_km: 300,
extended_range_km: 600,
exceptional_range_km: 1000
},
3_456 => %{
label: "3456 MHz",
o2_db_km: 0.006,
h2o_coeff: 0.0,
humidity_effect: :beneficial,
humidity_penalty: 0.0,
rain_k: 0.002, rain_alpha: 1.20,
seasonal_base: %{1 => 38, 2 => 40, 3 => 22, 4 => 55, 5 => 68,
6 => 90, 7 => 95, 8 => 75, 9 => 78, 10 => 88,
11 => 78, 12 => 25},
seasonal_adj: %{},
typical_range_km: 250,
extended_range_km: 550,
exceptional_range_km: 900
},
5_760 => %{
label: "5760 MHz",
o2_db_km: 0.007,
h2o_coeff: 0.0,
humidity_effect: :beneficial,
humidity_penalty: 0.0,
rain_k: 0.005, rain_alpha: 1.25,
seasonal_base: %{1 => 38, 2 => 40, 3 => 22, 4 => 55, 5 => 68,
6 => 90, 7 => 95, 8 => 75, 9 => 78, 10 => 88,
11 => 78, 12 => 25},
seasonal_adj: %{},
typical_range_km: 220,
extended_range_km: 500,
exceptional_range_km: 1000
},
10_000 => %{ 10_000 => %{
label: "10 GHz", label: "10 GHz",
o2_db_km: 0.008, o2_db_km: 0.007,
h2o_coeff: 0.0005, h2o_coeff: 0.0,
humidity_effect: :beneficial, # More moisture = more refractivity = longer paths humidity_effect: :beneficial, # More moisture = more refractivity = longer paths
humidity_penalty: 0.0, humidity_penalty: 0.0,
rain_k: 0.010, rain_alpha: 1.28, rain_k: 0.010, rain_alpha: 1.28,
# Seasonal pattern INVERTED from 24+ GHz: ducting + humidity both help 10G # Seasonal pattern INVERTED from 24+ GHz: ducting + humidity both help 10G
# Ducting peaks Jun-Jul (69-77%), March worst (10.8%), Dec-Feb ~12-22% # Ducting peaks Jun-Jul (69-77%), March worst (10.8%), Dec-Feb ~12-22%
# Score tracks ducting probability scaled to 0-100 # Score tracks ducting probability scaled to 0-100
seasonal_base: %{1 => 38, 2 => 32, 3 => 22, 4 => 55, 5 => 68, seasonal_base: %{1 => 38, 2 => 40, 3 => 22, 4 => 55, 5 => 68,
6 => 90, 7 => 95, 8 => 75, 9 => 78, 10 => 82, 6 => 90, 7 => 95, 8 => 75, 9 => 78, 10 => 88,
11 => 78, 12 => 25}, 11 => 78, 12 => 25},
seasonal_adj: %{}, seasonal_adj: %{},
typical_range_km: 200, typical_range_km: 200,
@ -640,8 +725,8 @@ Binned distance analysis of 37,925 HRRR-matched contacts confirms and refines Pa
}, },
24_000 => %{ 24_000 => %{
label: "24 GHz", label: "24 GHz",
o2_db_km: 0.015, o2_db_km: 0.02,
h2o_coeff: 0.012, # Validated by commercial link data h2o_coeff: 0.002, # Validated by commercial link data
humidity_effect: :harmful, # 22.235 GHz H2O line shoulder humidity_effect: :harmful, # 22.235 GHz H2O line shoulder
humidity_penalty: 1.6, humidity_penalty: 1.6,
rain_k: 0.070, rain_alpha: 1.07, rain_k: 0.070, rain_alpha: 1.07,
@ -657,7 +742,7 @@ Binned distance analysis of 37,925 HRRR-matched contacts confirms and refines Pa
}, },
47_000 => %{ 47_000 => %{
label: "47 GHz", label: "47 GHz",
o2_db_km: 0.045, o2_db_km: 0.04,
h2o_coeff: 0.003, h2o_coeff: 0.003,
humidity_effect: :harmful, humidity_effect: :harmful,
humidity_penalty: 1.0, # Window band, moderate H2O sensitivity humidity_penalty: 1.0, # Window band, moderate H2O sensitivity
@ -962,10 +1047,11 @@ def score_pressure(current_mb, previous_mb) do
nil -> nil ->
# No trend — score on absolute value, low pressure = better # No trend — score on absolute value, low pressure = better
cond do cond do
current_mb < 1005 -> 80 # Low — frontal activity, boundary ducts current_mb < 980 -> 88 # Deep low — strong frontal dynamics
current_mb < 1010 -> 70 # Moderate low current_mb < 990 -> 82 # Low — frontal activity, boundary ducts
current_mb < 1015 -> 60 # Normal current_mb < 1000 -> 70 # Moderate low
current_mb < 1020 -> 45 # Mild high — stable, less ducting current_mb < 1010 -> 55 # Normal
current_mb < 1020 -> 40 # Mild high — stable, less ducting
true -> 30 # Strong ridge — inversions cap at wrong altitude true -> 30 # Strong ridge — inversions cap at wrong altitude
end end
@ -1489,6 +1575,41 @@ Duct coupling geometry becomes increasingly critical at higher frequencies due t
For surface ducts, the beam must arrive at <0.5° grazing incidence to be trapped. For elevated ducts (500-1500m AGL), the optimal elevation angle is path-distance dependent: slightly positive at close range, near-zero at the "sweet spot" distance, and slightly negative at extreme range due to Earth curvature. For surface ducts, the beam must arrive at <0.5° grazing incidence to be trapped. For elevated ducts (500-1500m AGL), the optimal elevation angle is path-distance dependent: slightly positive at close range, near-zero at the "sweet spot" distance, and slightly negative at extreme range due to Earth curvature.
### 902 MHz (33cm) — UHF Troposcatter/Ducting Band
**Primary mechanisms:** Tropospheric ducting, troposcatter, enhanced refraction
**Key variable:** Refractivity profile; gaseous absorption negligible (O₂ 0.006 dB/km, H₂O 0.0)
**Best conditions:** Same as 10 GHz — moderate-high humidity for refractivity, stable atmosphere, nocturnal inversions
**Unique:** Longest potential range of any configured band (typical 400 km, exceptional 1500 km). Rain attenuation is zero (k=0.000). Propagation behavior closely mirrors 10 GHz but with lower free-space path loss and wider antenna beamwidths, making duct coupling geometry less critical.
### 1296 MHz (23cm) — L-Band Ducting
**Primary mechanisms:** Tropospheric ducting, troposcatter
**Key variable:** Refractivity profile; absorption negligible (O₂ 0.006 dB/km, H₂O 0.0)
**Best conditions:** Same seasonal/diurnal profile as 902 MHz and 10 GHz
**Unique:** Typical 350 km, exceptional 1200 km. Zero rain attenuation. Slightly shorter range than 902 MHz due to increased FSPL.
### 2304 MHz (13cm) — S-Band
**Primary mechanisms:** Tropospheric ducting, enhanced refraction
**Key variable:** Refractivity profile; onset of measurable rain sensitivity (k=0.001, alpha=1.15)
**Best conditions:** Moderate-high humidity, stable inversions
**Unique:** Typical 300 km, exceptional 1000 km. First band where rain has any measurable effect, though still minimal.
### 3456 MHz (9cm) — S-Band Upper
**Primary mechanisms:** Tropospheric ducting, enhanced refraction
**Key variable:** Refractivity profile; mild rain sensitivity (k=0.002, alpha=1.20)
**Best conditions:** Same as lower beneficial bands
**Unique:** Typical 250 km, exceptional 900 km. Transitional band — still firmly in the "humidity beneficial" regime but approaching the range where free-space loss begins to limit practical paths.
### 5760 MHz (5cm) — C-Band
**Primary mechanisms:** Tropospheric ducting, enhanced refraction
**Key variable:** Refractivity profile; moderate rain sensitivity (k=0.005, alpha=1.25)
**Best conditions:** Moderate-high humidity, stable atmosphere
**Unique:** Typical 220 km, exceptional 1000 km. Last beneficial-humidity band before 10 GHz. O₂ absorption increases slightly to 0.007 dB/km (matching 10 GHz). Rain attenuation still modest but becoming relevant in heavy rain.
### 10 GHz (3cm) — Tropospheric Ducting Band ### 10 GHz (3cm) — Tropospheric Ducting Band
**Primary mechanisms:** Ducting, enhanced refraction **Primary mechanisms:** Ducting, enhanced refraction
@ -1619,6 +1740,248 @@ For each band:
--- ---
## Part 12: HRRR Native Hybrid-Sigma Levels
The standard HRRR product provides atmospheric profiles on 13 pressure levels (every 25 hPa from 1000-700 hPa), giving approximately 250m vertical spacing. This is insufficient for resolving the thin surface ducts (50-100m) that produce the strongest microwave propagation events. The native hybrid-sigma product provides dramatically better vertical resolution.
### Product Details
The native HRRR file (`wrfnatf00.grib2`) carries all variables on the 50 hybrid-sigma levels native to the HRRR model grid. File size is approximately 530 MB per hour for the essential variables.
**Extracted variables (7 per level, 350 messages total):**
| Variable | Description |
|----------|-------------|
| TMP | Temperature (K) |
| SPFH | Specific humidity (kg/kg) |
| HGT | Geopotential height (m) |
| UGRD | U-component wind (m/s) |
| VGRD | V-component wind (m/s) |
| TKE | Turbulent kinetic energy (m²/s²) |
| PRES | Pressure (Pa) |
**Vertical spacing:** ~10-50m near the surface vs ~250m for the pressure-level product. This resolves the thin boundary-layer structures (inversions, ducts, shear layers) that are invisible in the standard product.
### Extraction Method
Each native-level file is too large for per-point on-demand fetching. The worker fetches the file once per `(date, hour)` and extracts profiles for all points of interest in one pass using wgrib2 point extraction (`-lon`). This avoids creating a coast-to-coast grid (~476k cells x 350 messages) that would cause OOM. Profiles are bulk-inserted into the `hrrr_native_profiles` table.
The pure-function `build_native_profile/1` converts parsed GRIB2 output into arrays sorted by ascending height (level 1 = surface), with surface scalars cached separately for quick access.
### Derived Products
Native profiles feed into three analysis modules:
1. **Duct detection** (`Propagation.Duct`) — M-profile analysis, per-duct geometry and trapped frequency
2. **Inversion analysis** (`Propagation.Inversion`) — temperature inversion top, Bulk Richardson number, theta-e jump, wind shear
3. **Backtest features** — native_surface_refractivity, bulk_richardson, theta_e_jump, shear_at_top, duct_thickness, best_duct_freq
---
## Part 13: Duct Analysis
The scalar `min_refractivity_gradient` from the standard HRRR product captures whether ducting conditions exist but not the physical duct geometry. The `Propagation.Duct` module replaces this with explicit duct detection from native hybrid-sigma profiles, providing per-duct properties that enable frequency-dependent scoring.
### Modified Refractivity M-Profile
From the ITU-R P.453-14 refractivity N at each level:
```
N = 77.6 * P/T + 3.73e5 * e/T²
```
where water vapor pressure e is derived from specific humidity: `e = q*P / (0.622 + 0.378*q)`.
The modified refractivity M accounts for earth curvature:
```
M = N + 157 * h/1000
```
where h is height in meters. In a standard atmosphere, M always increases with height. A duct exists wherever M decreases with height (dM/dh < 0).
### Duct Detection
The algorithm walks the M-profile looking for contiguous regions where M decreases. Each duct is characterized by:
| Property | Description |
|----------|-------------|
| base_m | Height (m) of duct base — where M begins decreasing |
| top_m | Height (m) of duct top — where M resumes increasing |
| thickness_m | Duct thickness (top - base) |
| m_deficit | Total M decrease across the duct (M-units) — the strength of trapping |
Multiple ducts per profile are possible and independently reported (e.g., a surface duct at 50-200m and an elevated duct at 800-1200m).
### Minimum Trapped Frequency
The key improvement over the scalar gradient approach: a 50m duct can trap 24 GHz but not 3 GHz, and the scalar had no way to express this. The minimum trapped frequency uses the waveguide approximation from Bean & Dutton (1966):
```
λ_max = 2.5 * d * sqrt(ΔM * 1e-6) (meters)
f_min = c / λ_max (Hz)
```
where d is duct thickness in meters and ΔM is the M-deficit. Returns 999 GHz for degenerate ducts (d ≤ 0 or ΔM ≤ 0).
### Analysis Pipeline
The full pipeline — `Duct.analyze/1` — takes a native profile and returns:
- `ducts` — list of duct maps, each with base, top, thickness, M-deficit, and min_freq_ghz
- `best_duct_band_ghz` — the lowest min_freq_ghz across all detected ducts (nil if no ducts)
This enables per-band scoring: a duct is "usable" for a given band only if `min_freq_ghz <= band_frequency_ghz`.
---
## Part 14: Inversion Analysis
Temperature inversions — where temperature increases with height, violating the normal lapse rate — are the boundary layers that act as mirrors for RF propagation. The `Propagation.Inversion` module detects inversions from native HRRR profiles and computes the stability and turbulence properties at the inversion top that determine whether the layer is smooth enough to support ducting.
### Inversion Top Detection
The algorithm walks the native profile upward looking for the first (lowest) temperature inversion:
1. Find the **inversion base** — the level where dT/dz first turns positive (temperature begins increasing with height)
2. Find the **inversion top** — the level where dT/dz turns negative again (temperature resumes its normal decrease)
3. Report the inversion **strength_k** — total temperature increase from base to top (K)
### Stability Properties at Inversion Top
Three quantities characterize whether the inversion layer is stable enough for propagation:
**Bulk Richardson Number (Ri):**
```
Ri = (g / θ_ref) * Δθ * Δz / (ΔU² + ΔV²)
```
where Δθ is the potential temperature difference across the layer, ΔU/ΔV are wind component differences, and θ_ref is the mean potential temperature. Potential temperature: θ = T * (P₀/P)^0.286 where P₀ = 100000 Pa.
| Ri Range | Regime | Propagation Impact |
|----------|--------|-------------------|
| < 0.25 | Turbulent | Bad — mixing disrupts duct structure |
| 0.25-1.0 | Transition | Marginal — intermittent ducting |
| > 1.0 | Laminar | Good — stable, smooth reflective layer |
Ri is clamped to 100.0 for practical use when wind shear is near zero (effectively infinite stability).
**Equivalent Potential Temperature (θₑ) Jump:**
The change in θₑ across the inversion. A larger positive jump indicates stronger thermodynamic decoupling between the air masses above and below the inversion — a sharper boundary that reflects RF energy more efficiently.
**Wind Shear Magnitude:**
```
shear = sqrt((u_top - u_base)² + (v_top - v_base)²) (m/s)
```
Strong shear at the inversion top can mechanically disrupt the layer (reducing Ri below 0.25) or, in moderate amounts, help maintain the inversion through differential advection.
---
## Part 15: NEXRAD Radar Data
The `NexradClient` fetches IEM CONUS n0q composite reflectivity images as a proxy for boundary-layer turbulence and precipitation structure.
### Product Specification
| Parameter | Value |
|-----------|-------|
| Product | n0q composite reflectivity |
| Source | IEM archive (`mesonet.agron.iastate.edu`) |
| Format | Palettized 8-bit PNG |
| Dimensions | 12200 x 5400 pixels |
| Coverage | CONUS (-126W to -65W, 23N to 50N) |
| Resolution | 0.005 degrees/pixel |
| Cadence | Every 5 minutes |
| Pixel mapping | Value 0 = no echo; 1-255 maps linearly to -30 to +95 dBZ |
### Per-Point Processing
For each point of interest, a ~25 km box (~50x50 pixels) centered on the lat/lon is extracted. Summary statistics computed per box:
- **mean_reflectivity_dbz** — average dBZ of non-zero pixels
- **max_reflectivity_dbz** — peak reflectivity in the box
- **texture_variance** — sample variance of dBZ values within the box
- **pixel_count** — number of non-zero echo pixels
### Backtest Use
The `nexrad_texture` feature in the backtest framework uses texture_variance as a proxy for boundary-layer convective activity. Higher variance indicates more turbulence, which is generally worse for microwave propagation (disrupts stable ducting layers). The feature looks up the nearest NEXRAD observation within ±15 minutes and ±0.1 degrees of the target point.
---
## Part 16: Backtest Framework
The backtest framework (`Microwaveprop.Backtest`) evaluates whether a candidate feature function carries information about propagation quality by comparing its distribution during actual QSO events against a matched random baseline.
### Methodology
A feature function has the shape `(lat, lon, valid_time) -> float | nil`. The `evaluate/2` function:
1. Loads up to N QSOs (`:sample_size`, default 5000) with known positions, newest first
2. Evaluates the feature at each QSO's station1 location and timestamp
3. Generates a matched random baseline: for each baseline sample, picks a real QSO location and perturbs its timestamp by uniform ±30 days. This controls for seasonal and geographic distribution so the baseline is not trivially distinguishable.
4. Reports `Distribution` statistics (count, mean, stddev, p50, p90, min, max) for both the QSO and baseline samples
If a feature has discriminating power, the QSO distribution should differ systematically from the baseline distribution — e.g., stronger gradients or lower PWAT during actual contacts.
### Analysis Dimensions
- **`evaluate/2`** — QSO vs random baseline distributions
- **`lift_by_distance/2`** — Feature distribution binned by QSO distance (0-100 km, 100-250 km, 250-500 km, 500-1000 km, 1000+ km). A useful feature should show monotonically increasing values across distance bins.
- **`lift_by_band/2`** — Feature distribution grouped by band. Reveals band-dependent lift (e.g., duct geometry features should carry more information at 24+ GHz).
- **`consolidated_report/2`** — Runs all features against the same QSO sample, producing a single comparison table.
### Implemented Features
Features are grouped by data source and physical quantity:
**Standard HRRR profile (scalar):**
| Feature | Source | Description |
|---------|--------|-------------|
| naive_gradient | hrrr_profiles | Minimum refractivity gradient (N/km) — current scorer baseline |
| td_depression | hrrr_profiles | Surface T - Td (°C) — atmospheric stability proxy |
| pressure | hrrr_profiles | Surface pressure (hPa) — frontal activity proxy |
| time_of_day | timestamp | UTC hour as float [0, 24) — diurnal baseline |
**Native hybrid-sigma profile (duct/inversion):**
| Feature | Source | Description |
|---------|--------|-------------|
| native_surface_refractivity | hrrr_native_profiles | ITU-R P.453 N from native-level data — sanity check vs pressure-level |
| bulk_richardson | hrrr_native_profiles | Ri at inversion top — laminar (>1) vs turbulent (<0.25) |
| theta_e_jump | hrrr_native_profiles | θₑ jump (K) across inversion — decoupling strength |
| shear_at_top | hrrr_native_profiles | Wind shear (m/s) at inversion top |
| duct_thickness | hrrr_native_profiles | Max duct thickness (m) — larger ducts trap lower frequencies |
| best_duct_freq | hrrr_native_profiles | Lowest trapped frequency (GHz) — lower = stronger ducting |
| duct_usable_10ghz | hrrr_native_profiles | Binary: any duct traps 10 GHz? |
| duct_usable_24ghz | hrrr_native_profiles | Binary: any duct traps 24 GHz? |
| duct_usable_47ghz | hrrr_native_profiles | Binary: any duct traps 47 GHz? |
**Climatology and remote sensing:**
| Feature | Source | Description |
|---------|--------|-------------|
| temperature_anomaly | hrrr_profiles + hrrr_climatology | Surface T minus climatological mean for (grid cell, month, hour) — anomalously hot days produce enhanced ducting |
| nexrad_texture | nexrad_observations | Reflectivity texture variance — convective turbulence proxy |
**Placeholder (not yet implemented):**
| Feature | Source | Description |
|---------|--------|-------------|
| distance_to_front | — | Distance (km) to nearest detected front — awaiting frontal analysis pipeline |
| parallel_to_front | — | cos²(path-front angle) — awaiting frontal analysis + requires path bearing |
### HRRR Climatology
The `hrrr_climatology` table stores pre-computed mean and standard deviation of surface temperature from `hrrr_profiles`, keyed on `(lat, lon, month, hour)` at the HRRR grid resolution. This allows computing how anomalous the current surface temperature is relative to historical norms for the same location, season, and time of day. The meteorologist noted that extremely hot days (~10°F above normal in summer) produce enhanced ducting even in the afternoon when the time-of-day factor normally suppresses the score. The `temperature_anomaly` feature returns `surface_temp_c - climatological_mean` as a signed float.
---
## Constants Reference ## Constants Reference
| Constant | Value | Source | | Constant | Value | Source |
@ -1761,13 +2124,13 @@ The following ITU-R Recommendations provide the physics models underlying the sc
- Coverage: CONUS grid at 0.125° resolution - Coverage: CONUS grid at 0.125° resolution
- Update cadence: hourly (HRRR-based via `PropagationGridWorker`) + 10-minute ASOS adjustments (`AsosAdjustmentWorker`) - Update cadence: hourly (HRRR-based via `PropagationGridWorker`) + 10-minute ASOS adjustments (`AsosAdjustmentWorker`)
- Per grid point: composite score (0-100), 9 individual factor scores, valid_time - Per grid point: composite score (0-100), 9 individual factor scores, valid_time
- Bands scored: all configured bands (10, 24, 47, 68, 75, 122, 134, 142, 241 GHz) - Bands scored: all configured bands (902 MHz, 1296 MHz, 2304 MHz, 3456 MHz, 5760 MHz, 10, 24, 47, 68, 75, 122, 134, 241 GHz)
- Retention: 2 most recent valid_times kept, older data pruned automatically - Retention: 2 most recent valid_times kept, older data pruned automatically
## Known Data Quality Issues ## Known Data Quality Issues
- **EME contamination**: 4 QSOs >3,000 km remain in dataset (QRA64D/JT4F modes). Filter on `distance_km < 3000` for tropospheric analysis. - **EME contamination**: 4 QSOs >3,000 km remain in dataset (QRA64D/JT4F modes). Filter on `distance_km < 3000` for tropospheric analysis.
- **Unmodeled bands**: 142, 145, 288, 322, 403, 411 GHz have 1-4 QSOs each but no band_config entries. Too sparse for statistical analysis. - **Unmodeled bands**: 142, 145, 288, 322, 403, 411 GHz have 1-4 QSOs each but no band_config entries. Too sparse for statistical analysis. The 902 MHz through 5760 MHz bands are now implemented with beneficial humidity effect and shared seasonal tables matching 10 GHz.
- **Sounding data recency**: Latest soundings are from Sep 2024. Ingestion pipeline may need restart for live enrichment. - **Sounding data recency**: Latest soundings are from Sep 2024. Ingestion pipeline may need restart for live enrichment.
- **Surface obs density**: Historical obs are sparse (~1 per 4.7 days per station) because they were fetched per-QSO-window. Live polling is now active for continuous coverage. - **Surface obs density**: Historical obs are sparse (~1 per 4.7 days per station) because they were fetched per-QSO-window. Live polling is now active for continuous coverage.
- **Weather codes (wx_codes)**: Stored in surface_observations but unused by scoring. Direct fog/thunderstorm/freezing-rain detection could supplement indirect inference from Td depression and sky condition. - **Weather codes (wx_codes)**: Stored in surface_observations but unused by scoring. Direct fog/thunderstorm/freezing-rain detection could supplement indirect inference from Td depression and sky condition.