graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update algo.md with data-driven findings from 57K QSO analysis

Browse source
This commit is contained in:
Graham McIntire 2026-04-01 08:41:33 -05:00
parent b7fc195e82
commit 83a1511c84
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
1 changed files with 194 additions and 34 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

228
algo.md
View file

@ -494,6 +494,111 @@ The correct approach is to let the physics-based factors (humidity, refractivity
---
## Part 2b: Data-Driven Refinements (April 2026)
This section documents findings from a systematic correlation analysis matching 57,248 QSOs to HRRR atmospheric conditions at both endpoints. Each QSO was joined to the nearest HRRR grid point (0.125° snap) at both station positions, using the profile valid at the hour of the contact. Spearman rank correlation (rho) measures monotonic association between each atmospheric variable and achieved distance — a nonparametric measure robust to outliers and non-linear relationships.
### Correlation Rankings by Band
**10 GHz (n=52,846):**
| Variable | rho | n_valid |
|----------|-----|---------|
| Pressure (mb) | **-0.180** | 52,846 |
| Month | 0.105 | 52,846 |
| Dewpoint (°C) | -0.059 | 52,846 |
| HPBL (m) | 0.045 | 52,846 |
| PWAT (mm) | -0.039 | 52,846 |
| Refractivity Gradient | -0.034 | 52,347 |
| Temperature (°C) | 0.031 | 52,846 |
| Surface Refractivity | -0.024 | 52,347 |
| UTC Hour | **0.007** | 52,846 |
**24 GHz (n=3,621):**
| Variable | rho | n_valid |
|----------|-----|---------|
| Dewpoint (°C) | **-0.371** | 3,621 |
| PWAT (mm) | **-0.330** | 3,621 |
| Surface Refractivity | **-0.317** | 3,582 |
| Month | 0.272 | 3,621 |
| Temperature (°C) | -0.179 | 3,621 |
| Pressure (mb) | -0.172 | 3,621 |
| Refractivity Gradient | -0.075 | 3,582 |
| UTC Hour | 0.056 | 3,621 |
| HPBL (m) | -0.049 | 3,621 |
**47 GHz (n=680):**
| Variable | rho | n_valid |
|----------|-----|---------|
| Pressure (mb) | **-0.231** | 680 |
| PWAT (mm) | **-0.227** | 680 |
| Dewpoint (°C) | -0.181 | 680 |
| Refractivity Gradient | -0.139 | 678 |
| Month | 0.111 | 680 |
| Surface Refractivity | -0.109 | 678 |
| Temperature (°C) | 0.037 | 680 |
| UTC Hour | -0.024 | 680 |
| HPBL (m) | 0.004 | 680 |
**75 GHz (n=94):**
| Variable | rho | n_valid |
|----------|-----|---------|
| Dewpoint (°C) | **-0.703** | 94 |
| PWAT (mm) | **-0.608** | 94 |
| Temperature (°C) | **-0.589** | 94 |
| Pressure (mb) | **-0.570** | 94 |
| Surface Refractivity | **-0.526** | 94 |
| UTC Hour | -0.392 | 94 |
| HPBL (m) | 0.150 | 94 |
| Month | 0.144 | 94 |
| Refractivity Gradient | -0.082 | 94 |
### Key Insights
**1. Pressure was massively underweighted.** It is the #1 correlator at 10 GHz (rho=-0.180) but was assigned only 4% weight. The binned analysis is unambiguous:
| Pressure Bin | n | Median km | p25 | p75 |
|-------------|---|-----------|-----|-----|
| <1005 mb | 47,669 | **197.1** | 121.2 | 285.1 |
| 1005-1013 mb | 3,447 | 151.3 | 100.7 | 289.2 |
| 1013-1020 mb | 1,254 | 130.8 | 77.9 | 239.2 |
| >1020 mb | 476 | **103.4** | 76.8 | 191.3 |
Low pressure (<1005 mb) gives 197 km median vs 103 km for >1020 mb — a nearly 2x difference. The old scoring function scored high pressure higher, which was **completely backwards** for beyond-LOS propagation. Low pressure systems bring frontal boundaries, moisture gradients, and boundary-layer structures that create ducting conditions.
**2. Time of day was overweighted at 10 GHz.** UTC hour correlates at rho=0.007 — barely above zero. The binned data shows modest variation (181-210 km across 3-hour blocks) with no clear diurnal signal at 10 GHz. The 20% weight was far too high for this band. Time of day matters more at 24+ GHz (rho=0.056 at 24 GHz, -0.392 at 75 GHz), consistent with Finding 8 in Part 2, but the weight should scale with frequency rather than being uniform.
**3. Refractivity gradient is weaker than expected.** Correlation ranges from rho=-0.034 at 10 GHz to -0.139 at 47 GHz. The HRRR's 8-level vertical resolution is too coarse to resolve the thin ducting layers that produce the strongest gradients. The binned analysis at 10 GHz confirms: gradient <-300 gives 214 km median vs 192 km for >-100 — only an 11% improvement. Still useful but the 10% weight was generous given the data.
**4. PWAT is a strong independent predictor not captured by any existing factor.** Correlations range from rho=-0.039 at 10 GHz to -0.608 at 75 GHz. At 10 GHz, the relationship is non-monotonic with a sweet spot:
| PWAT Bin | n | Median km | p25 | p75 |
|---------|---|-----------|-----|-----|
| <10 mm | 1,834 | 160.6 | 96.8 | 265.0 |
| 10-20 mm | 15,041 | 193.8 | 125.1 | 280.7 |
| 20-30 mm | 17,788 | **218.8** | 129.9 | 295.6 |
| 30-40 mm | 13,999 | 173.9 | 106.5 | 289.6 |
| >40 mm | 4,184 | 155.2 | 84.5 | 256.1 |
At 24 GHz, the relationship is monotonic — lower is always better: <10 mm gives 126 km median vs 45 km for >40 mm. PWAT integrates the full moisture column and captures information beyond surface-level humidity and Td depression.
**5. Ducting detection is a non-discriminator at 10 GHz.** Ducting YES: n=7,979, median 189 km. Ducting NO: n=44,867, median 192 km. The non-ducting group actually achieves slightly longer median distances. Binary ducting detection from HRRR profiles is useless for scoring — consistent with Finding 5 in Part 2, now confirmed with 10x the sample size using HRRR data rather than soundings.
**6. Data is almost entirely Aug/Sep.** Of 52,846 QSOs at 10 GHz, 26,813 are August and 26,024 are September. Only 9 QSOs fall outside these months. This limits seasonal conclusions but does not invalidate the atmospheric correlations within those months — pressure, PWAT, and temperature-dewpoint vary substantially within Aug-Sep due to synoptic weather patterns.
### Interaction Effects (10 GHz)
The analysis tested whether atmospheric variables interact (i.e., does the effect of one variable depend on the value of another):
**Refractivity Gradient x Time of Day:** Strong gradients (avg < -100 N/km) improve night/morning distances by 20-30 km but have negligible or negative effect in the afternoon. At night, strong gradient gives 204 km vs 168 km for weak gradient. In the afternoon, the relationship inverts: weak gradient gives 198 km vs 190 km for strong. This suggests afternoon convective mixing disrupts duct structures regardless of gradient strength.
**HPBL x Season:** In summer, deeper BL correlates with longer distances (shallow 180 km, mid 207 km, deep 231 km). In fall, the relationship flattens (shallow 184 km, mid 194 km, deep 188 km). Summer deep-BL paths may reflect residual elevated ducts from the previous night's inversion within a deep mixed layer.
---
## Part 3: Band Configuration
```elixir
@ -817,31 +922,33 @@ def score_rain(rain_rate_mmhr, band_config) do
end
```
### 8. Pressure Score — Gradient-Focused
### 8. Pressure Score — Low Pressure Favors Beyond-LOS
Data contradicts the original high-pressure-is-good model. Frontal boundaries (changing pressure) create the strongest refractive gradients. Rising post-frontal pressure and slowly falling pre-frontal pressure both score well.
Data from 57,248 QSO-HRRR matches shows pressure is the **#1 correlator at 10 GHz** (rho=-0.180). The relationship is monotonic and strong: <1005 mb gives 197 km median vs 103 km for >1020 mb. Low pressure systems bring frontal boundaries, moisture gradients, and boundary-layer structures that create ducting conditions. The previous function scored high pressure higher — completely backwards for beyond-LOS propagation.
When trend data is available, falling pressure (approaching front) scores highest because pre-frontal dynamics create the strongest refractive gradients.
```elixir
def score_pressure(current_mb, previous_mb) do
case previous_mb do
nil ->
# No trend — mild scoring on absolute value
# No trend — score on absolute value, low pressure = better
cond do
current_mb > 1025 -> 55 # Strong ridge
current_mb > 1018 -> 65 # Mild high
current_mb > 1010 -> 60 # Normal
current_mb > 1005 -> 55 # Low
true -> 40 # Very low — active weather
current_mb < 1005 -> 80 # Low — frontal activity, boundary ducts
current_mb < 1010 -> 70 # Moderate low
current_mb < 1015 -> 60 # Normal
current_mb < 1020 -> 45 # Mild high — stable, less ducting
true -> 30 # Strong ridge — inversions cap at wrong altitude
end
prev ->
delta = current_mb - prev
cond do
delta > 2.5 -> 80 # Rising rapidly — post-frontal clearing
delta > 0.8 -> 70 # Rising — stabilizing
delta > 2.5 -> 65 # Rising rapidly — post-frontal clearing, some residual ducts
delta > 0.8 -> 55 # Rising — stabilizing
delta > -0.5 -> 60 # Steady — neutral
delta > -2.0 -> 65 # Falling slowly — approaching front, duct possible
true -> 45 # Falling rapidly — active weather
delta > -2.0 -> 75 # Falling slowly — approaching front, duct formation
true -> 70 # Falling rapidly — active frontal dynamics
end
end
end
@ -865,38 +972,85 @@ Thresholds calibrated for HRRR-derived gradients which are coarser than radioson
Shallow BL fallback: when gradient is unavailable but BL depth < 300m, score 82 (strong inversion cap).
### 10. PWAT Score — Precipitable Water (NEW)
PWAT (precipitable water, total column integrated moisture in mm) is a strong independent predictor that was not previously a separate scoring factor. Correlation with distance ranges from rho=-0.039 at 10 GHz to -0.608 at 75 GHz. Unlike surface humidity and Td depression which measure conditions at ground level, PWAT integrates the full moisture column and captures elevated moisture layers relevant to duct formation and path absorption.
At 10 GHz (beneficial humidity), the relationship is non-monotonic: 20-30 mm PWAT gives the best median distances (219 km), with both very dry (<10 mm, 161 km) and very wet (>40 mm, 155 km) conditions performing worse. Moderate PWAT indicates sufficient moisture for refractivity enhancement without the atmospheric instability that accompanies very high moisture content.
At 24+ GHz (harmful humidity), lower PWAT is universally better. At 24 GHz: <10 mm gives 126 km median, >40 mm gives 45 km — a 2.8x difference.
```elixir
def score_pwat(pwat_mm, band_config) do
case band_config.humidity_effect do
:beneficial ->
# 10 GHz: sweet spot at moderate PWAT (20-30 mm)
cond do
pwat_mm < 10 -> 55 # Very dry — poor refractivity
pwat_mm < 20 -> 75 # Moderate dry
pwat_mm < 30 -> 90 # Optimal — best median distances
pwat_mm < 40 -> 70 # High — beginning absorption penalty
true -> 50 # Very high — absorption dominates
end
:harmful ->
# 24+ GHz: lower is better, scales by frequency via humidity_penalty
cond do
pwat_mm < 10 -> 95 # Very dry — minimal absorption
pwat_mm < 20 -> 80 # Low — good conditions
pwat_mm < 30 -> 60 # Moderate — noticeable absorption
pwat_mm < 40 -> 35 # High — significant absorption
true -> 15 # Very high — severe absorption
end
end
end
```
---
## Part 5: Composite Score
### Weights
Revised from expanded dataset. Key changes: refractivity weight doubled (HRRR now provides hourly data, not just 12-hourly soundings), time-of-day weight increased (data shows it's the dominant variable above 24 GHz), pressure reduced (weak standalone predictor, 5% was still generous).
Revised April 2026 based on 57,248 QSO-HRRR correlation analysis. Key changes from previous weights:
| Factor | Weight | Rationale |
|--------|--------|-----------|
| Humidity | 20% | Dominant variable, but split role by frequency |
| Time of Day | 20% | First-order effect at 24+ GHz (28-360% enhancement at night) |
| Td Depression | 12% | Proxy for humidity aloft — strong signal |
| Refractivity | 10% | Best predictor; HRRR provides hourly coverage, sounding gradient is 3x discriminator |
| Sky Cover | 10% | Modest effect, mainly at higher frequencies |
| Season | 10% | Long-term baseline — now ducting-calibrated per band |
| Wind | 6% | Data shows minimal impact; penalty only for extremes |
| Rain | 8% | Critical for 24+ GHz paths |
| Pressure | 4% | Weak standalone predictor |
- **Pressure: 4% → 15%** — #1 correlator at 10 GHz (rho=-0.180); data shows low P (<1005 mb) gives 197 km median vs 103 km for >1020 mb. Previously scored backwards (high=good). Now the largest single-factor weight increase in algorithm history.
- **PWAT: 0% → 10%** — New factor. Strong independent predictor across all bands (rho=-0.039 to -0.608). Captures column-integrated moisture not represented by surface humidity or Td depression.
- **Time of Day: 20% → 10%** — rho=0.007 at 10 GHz; effect is real at 24+ GHz (rho=-0.392 at 75 GHz) but the 20% weight was unwarranted at the dominant 10 GHz band. Modest even at 24 GHz (rho=0.056).
- **Humidity: 20% → 18%** — Still the strongest signal at 24+ GHz (dewpoint rho=-0.371 at 24 GHz, -0.703 at 75 GHz) but mixed/weak at 10 GHz (rho=-0.059).
- **Td Depression: 12% → 10%** — Partially redundant with humidity and PWAT; retains value as a distinct atmospheric stability indicator.
- **Refractivity: 10% → 8%** — Weaker than expected (rho=-0.034 to -0.139). HRRR vertical resolution too coarse for thin duct detection. Still useful for strong events.
- **Sky Cover: 10% → 8%** — No direct measurement in this analysis; retained at reduced weight based on physical rationale (radiative cooling).
- **Season: 10% → 8%** — Aug/Sep bias in dataset limits validation. Physics-based seasonal curves remain valid but cannot be tuned from this data.
- **Rain: 8% → 8%** — Unchanged. ITU-R P.838-3 validated, critical for high-frequency paths.
- **Wind: 6% → 5%** — Minimal impact confirmed (Finding 2, Part 2).
| Factor | Old Weight | New Weight | Rationale |
|--------|-----------|-----------|-----------|
| Humidity | 20% | 18% | Strong at 24+ GHz (dewpoint rho=-0.37 to -0.70); mixed signal at 10 GHz |
| Pressure | 4% | **15%** | #1 correlator at 10 GHz (rho=-0.18); low P = longer paths confirmed |
| Time of Day | 20% | 10% | rho=0.007 at 10 GHz; modest even at 24 GHz |
| Td Depression | 12% | 10% | Partially redundant with humidity/PWAT |
| PWAT | — | **10%** | NEW: strong independent predictor all bands (rho=-0.04 to -0.61) |
| Refractivity | 10% | 8% | Weak correlation; HRRR too coarse for thin ducts |
| Sky Cover | 10% | 8% | No direct measurement in analysis; physics still valid |
| Season | 10% | 8% | Aug/Sep bias limits validation; physics-based curves retained |
| Rain | 8% | 8% | ITU-R validated, critical for high freq |
| Wind | 6% | 5% | Minimal impact confirmed |
```elixir
def composite_score(factors) do
round(
factors.humidity * 0.20 +
factors.time_of_day * 0.20 +
factors.td_depression * 0.12 +
factors.refractivity * 0.10 +
factors.sky * 0.10 +
factors.season * 0.10 +
factors.wind * 0.06 +
factors.humidity * 0.18 +
factors.pressure * 0.15 +
factors.time_of_day * 0.10 +
factors.td_depression * 0.10 +
factors.pwat * 0.10 +
factors.refractivity * 0.08 +
factors.sky * 0.08 +
factors.season * 0.08 +
factors.rain * 0.08 +
factors.pressure * 0.04
factors.wind * 0.05
)
end
```
@ -984,7 +1138,8 @@ def compute_score(conditions, band_config, path_type \\ :beyond_los) do
td_depression: score_td_depression(conditions.temp_f, conditions.dewpoint_f, band_config),
season: score_season(conditions.month, band_config),
pressure: score_pressure(conditions.slp, conditions.prev_slp),
rain: score_rain(conditions.rain_rate, band_config)
rain: score_rain(conditions.rain_rate, band_config),
pwat: score_pwat(conditions.pwat_mm, band_config)
}
factors = case path_type do
@ -1189,6 +1344,7 @@ def predict_scores(current_obs, obs_3hr_ago, forecast, band_config) do
season: score_season(month, band_config),
pressure: score_pressure(projected_slp, current_obs.slp),
rain: score_rain(projected_rain, band_config),
pwat: score_pwat(forecast_value(forecast, :pwat_mm, hours_ahead) || 50, band_config),
refractivity: 50 # Cannot predict from surface obs alone
}
@ -1384,7 +1540,8 @@ Surface Observations (ASOS, every 5-20 min)
HRRR Model (hourly, per grid point)
-> refractivity profile, dN/dh gradient, ducts, BL depth, PWAT, surface N
-> refractivity score component (10% weight — best discriminator)
-> refractivity score component (8% weight)
-> PWAT score component (10% weight — strong independent predictor, rho=-0.04 to -0.61)
-> 4,522 profiles in DB: 79% Enhanced, 15% Super, 5% Standard, 0.7% Ducting
-> Key thresholds: gradient < -300 = moderate ducting, < -500 = strong ducting
-> HPBL < 200m = strongest signal for enhanced propagation
@ -1455,6 +1612,9 @@ For each band:
| Ducting surface N threshold | 330 | Above this, ducting probability >50% |
| Signal prediction floor | +/- 2-3 dB | Measured from link data |
| CW bandwidth advantage | ~7 dB | 10*log10(2700/500); 24% range increase |
| Pressure correlation (10 GHz) | rho=-0.180 | 57,248 QSO-HRRR analysis, Apr 2026 |
| PWAT optimal range (10 GHz) | 20-30 mm | Best median distance (219 km) |
| PWAT correlation (75 GHz) | rho=-0.608 | 57,248 QSO-HRRR analysis, Apr 2026 |
## ITU-R Recommendations