Data-driven algorithm refinements from full dataset analysis
Analyzed 37,925 HRRR-matched contacts from 58,367 total. - Remove shallow BL bonus (score 82 for HPBL < 300m): data shows medium BL (1000-2000m) produces longest contacts (222 km avg), not shallow (210 km). Refractivity fallback now uses default score. - Refine pressure scoring bins: add <980 mb tier (score 88), steeper gradient from low to high. Contacts at <970 mb avg 242.7 km vs 184.1 km at 990-1000 mb. - Update algo.md calibration stats (58,367 contacts, 41M HRRR profiles, 58,361 terrain profiles) and document dataset bias (99.5% Aug-Sep). - Create updates.md with full binned analysis tables for all factors.
This commit is contained in:
parent
03aa72ae80
commit
b2f3098083
6 changed files with 140 additions and 54 deletions
16
algo.md
16
algo.md
|
|
@ -12,11 +12,11 @@ The regime distinction matters because refractivity effects are *inverted* betwe
|
|||
|
||||
### Calibration Dataset
|
||||
|
||||
**QSO data:** 58,282 total QSOs across 13+ bands (ARRL Microwave Contest, 1992-2024). 57,488 with tropospheric distance data after excluding 4 EME contacts (QRA64D/JT4F modes >3,000 km). Enriched with weather from 1,299 ASOS stations (58,398 surface observations), 112 RAOB stations (3,901 soundings), 4,522 HRRR model profiles, 3,675 IEMRE gridded hourly observations, and 58,276 terrain path profiles.
|
||||
**QSO data:** 58,367 total QSOs across 13 bands (ARRL Microwave Contest, 1992-2024). All tropospheric (distance < 3,000 km). Enriched with 41,071,398 HRRR model profiles (37,925 matched to contacts, 65% coverage), 15,092 IEMRE gridded hourly observations, 3,268 weather stations, and 58,361 terrain path profiles. **Critical bias: 99.5% of contacts are Aug-Sep** — all atmospheric correlations are effectively summer-only findings.
|
||||
|
||||
**Link data:** 7 commercial links near DFW (Princeton TX area) at 11/24/68 GHz, polled via SNMP at 5-minute intervals. All links use KTKI ASOS for weather correlation. Live polling is active; historical dataset from March 14-29 2026 (18,540 samples) was used for initial algorithm validation.
|
||||
|
||||
**Terrain analysis:** 58,276 QSO paths profiled — 56,658 BLOCKED (97.2%, avg 36.2 dB diffraction), 1,277 CLEAR (2.2%), 341 FRESNEL_PARTIAL (0.6%). Blocked paths average *longer* distances than clear paths (215 km vs 84 km at 10 GHz) because ducting enables beyond-LOS paths by definition.
|
||||
**Terrain analysis:** 58,361 QSO paths profiled — 56,735 BLOCKED (97.1%, avg 36.3 dB diffraction), 1,284 CLEAR (2.2%), 342 FRESNEL_PARTIAL (0.6%). Blocked paths average *longer* distances than clear paths (326 km for 40+ dB diffraction vs 31 km for CLEAR) because ducting enables beyond-LOS paths by definition — only the strongest propagation conditions produce contacts through heavy terrain at long distances.
|
||||
|
||||
**Confirmed long-range contacts:**
|
||||
- 47 GHz: 116.0 km (Nov 2025), 98.8 km (Jun 2024)
|
||||
|
|
@ -560,6 +560,18 @@ The analysis tested whether atmospheric variables interact (i.e., does the effec
|
|||
|
||||
**HPBL x Season:** In summer, deeper BL correlates with longer distances (shallow 180 km, mid 207 km, deep 231 km). In fall, the relationship flattens (shallow 184 km, mid 194 km, deep 188 km). Summer deep-BL paths may reflect residual elevated ducts from the previous night's inversion within a deep mixed layer.
|
||||
|
||||
### Full-Dataset Validation (April 2026)
|
||||
|
||||
Binned distance analysis of 37,925 HRRR-matched contacts confirms and refines Part 2b findings. See `updates.md` for full tables.
|
||||
|
||||
**Shallow BL bonus removed.** The algorithm previously awarded score 82 when HPBL < 300m. Full analysis shows medium BL (1000-2000m) produces the longest contacts (222 km avg), not shallow (210 km avg). Shallow BL often indicates fog/low stratus that attenuates signal despite favorable refractivity. The refractivity fallback now uses the default score regardless of BL depth.
|
||||
|
||||
**Pressure scoring refined.** Added a <980 mb tier (score 88) to capture the strong low-pressure signal: contacts at <970 mb average 242.7 km vs 184.1 km at 990-1000 mb (32% longer). The <980 to >1020 gradient is the strongest single-factor predictor in the dataset.
|
||||
|
||||
**Refractivity gradient flat in bulk range.** Gradient bins from -150 to -75 N/km all produce ~212-216 km avg distance. Only the weakest bin (>= -55 N/km, 176 km) shows meaningful degradation. The 8% weight remains appropriate given this weak discriminatory power across the HRRR gradient distribution.
|
||||
|
||||
**Mode distance advantage quantified.** CW: 247 km avg, PH (SSB): 191 km avg, FM: 141 km avg at 10 GHz. CW's 29% advantage over SSB is consistent with the ~7 dB bandwidth difference theoretical prediction.
|
||||
|
||||
---
|
||||
|
||||
## Part 3: Band Configuration
|
||||
|
|
|
|||
|
|
@ -44,8 +44,6 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
|||
{-40, 48, 48}
|
||||
]
|
||||
@refractivity_default {42, 42}
|
||||
@shallow_bl_threshold_m 300
|
||||
@shallow_bl_score 82
|
||||
|
||||
@band_configs %{
|
||||
10_000 => %{
|
||||
|
|
@ -321,12 +319,4 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
|||
@doc "Returns the default refractivity scores as {beneficial, harmful}."
|
||||
@spec refractivity_default() :: {number(), number()}
|
||||
def refractivity_default, do: @refractivity_default
|
||||
|
||||
@doc "Returns the shallow boundary layer threshold in meters."
|
||||
@spec shallow_bl_threshold_m() :: number()
|
||||
def shallow_bl_threshold_m, do: @shallow_bl_threshold_m
|
||||
|
||||
@doc "Returns the score applied when boundary layer is shallow."
|
||||
@spec shallow_bl_score() :: number()
|
||||
def shallow_bl_score, do: @shallow_bl_score
|
||||
end
|
||||
|
|
|
|||
|
|
@ -9,8 +9,6 @@ defmodule Microwaveprop.Propagation.Scorer do
|
|||
|
||||
alias Microwaveprop.Propagation.BandConfig
|
||||
|
||||
@shallow_bl_threshold_m BandConfig.shallow_bl_threshold_m()
|
||||
|
||||
# ── Temperature conversion helpers ────────────────────────────────
|
||||
|
||||
@doc "Converts Fahrenheit to Celsius. Returns nil for nil input."
|
||||
|
|
@ -159,25 +157,16 @@ defmodule Microwaveprop.Propagation.Scorer do
|
|||
@spec score_refractivity(number() | nil, number() | nil, map()) :: integer()
|
||||
def score_refractivity(nil, _bl_depth_m, _band_config), do: 50
|
||||
|
||||
def score_refractivity(min_gradient, bl_depth_m, %{humidity_effect: effect}) do
|
||||
def score_refractivity(min_gradient, _bl_depth_m, %{humidity_effect: effect}) do
|
||||
thresholds = BandConfig.refractivity_thresholds()
|
||||
|
||||
case find_refractivity_threshold(min_gradient, thresholds, effect) do
|
||||
{:ok, score} -> score
|
||||
:none -> refractivity_fallback(bl_depth_m, effect)
|
||||
end
|
||||
end
|
||||
{:ok, score} ->
|
||||
score
|
||||
|
||||
defp refractivity_fallback(bl_depth_m, _effect) when bl_depth_m != nil and bl_depth_m < @shallow_bl_threshold_m do
|
||||
BandConfig.shallow_bl_score()
|
||||
end
|
||||
|
||||
defp refractivity_fallback(_bl_depth_m, effect) do
|
||||
{beneficial_default, harmful_default} = BandConfig.refractivity_default()
|
||||
|
||||
case effect do
|
||||
:beneficial -> beneficial_default
|
||||
:harmful -> harmful_default
|
||||
:none ->
|
||||
{beneficial_default, harmful_default} = BandConfig.refractivity_default()
|
||||
if effect == :beneficial, do: beneficial_default, else: harmful_default
|
||||
end
|
||||
end
|
||||
|
||||
|
|
@ -308,10 +297,11 @@ defmodule Microwaveprop.Propagation.Scorer do
|
|||
|
||||
def score_pressure(current_mb, nil) do
|
||||
cond do
|
||||
current_mb < 1005 -> 80
|
||||
current_mb < 1010 -> 70
|
||||
current_mb < 1015 -> 60
|
||||
current_mb < 1020 -> 45
|
||||
current_mb < 980 -> 88
|
||||
current_mb < 990 -> 82
|
||||
current_mb < 1000 -> 70
|
||||
current_mb < 1010 -> 55
|
||||
current_mb < 1020 -> 40
|
||||
true -> 30
|
||||
end
|
||||
end
|
||||
|
|
|
|||
|
|
@ -270,18 +270,6 @@ defmodule Microwaveprop.Propagation.BandConfigTest do
|
|||
end
|
||||
end
|
||||
|
||||
describe "shallow_bl_threshold_m/0" do
|
||||
test "returns 300" do
|
||||
assert BandConfig.shallow_bl_threshold_m() == 300
|
||||
end
|
||||
end
|
||||
|
||||
describe "shallow_bl_score/0" do
|
||||
test "returns 82" do
|
||||
assert BandConfig.shallow_bl_score() == 82
|
||||
end
|
||||
end
|
||||
|
||||
describe "only 10 GHz is beneficial" do
|
||||
test "10 GHz has beneficial humidity effect" do
|
||||
assert BandConfig.get(10_000).humidity_effect == :beneficial
|
||||
|
|
|
|||
|
|
@ -251,13 +251,9 @@ defmodule Microwaveprop.Propagation.ScorerTest do
|
|||
assert Scorer.score_refractivity(nil, 500, @band_10g) == 50
|
||||
end
|
||||
|
||||
test "weak gradient with shallow BL returns shallow BL score" do
|
||||
# gradient > -60 (no threshold match), bl_depth < 300 -> 82
|
||||
assert Scorer.score_refractivity(-30, 200, @band_10g) == 82
|
||||
end
|
||||
|
||||
test "weak gradient with deep BL returns default" do
|
||||
# gradient > -60, bl_depth >= 300 -> 42
|
||||
test "weak gradient returns default regardless of BL depth" do
|
||||
# gradient > -40 (no threshold match) -> default 42
|
||||
assert Scorer.score_refractivity(-30, 200, @band_10g) == 42
|
||||
assert Scorer.score_refractivity(-30, 500, @band_10g) == 42
|
||||
end
|
||||
end
|
||||
|
|
@ -446,8 +442,9 @@ defmodule Microwaveprop.Propagation.ScorerTest do
|
|||
assert Scorer.score_pressure(1020, nil) == 30
|
||||
end
|
||||
|
||||
test "low pressure without previous returns 80" do
|
||||
assert Scorer.score_pressure(1000, nil) == 80
|
||||
test "low pressure without previous returns 55" do
|
||||
# 1000 is in [1000, 1010) -> 55
|
||||
assert Scorer.score_pressure(1000, nil) == 55
|
||||
end
|
||||
|
||||
test "rising pressure returns 80" do
|
||||
|
|
|
|||
109
updates.md
Normal file
109
updates.md
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
# Algorithm Updates — April 2026 (Full Dataset Analysis)
|
||||
|
||||
Analysis of 58,367 contacts matched against 41M HRRR profiles in the local database. This extends the earlier Part 2b correlation analysis with binned distance analysis across all atmospheric variables.
|
||||
|
||||
## Finding 1: Shallow Boundary Layer Bonus Not Supported
|
||||
|
||||
The algorithm previously awarded a score of 82 when HPBL < 300m (shallow BL), on the theory that shallow BL indicates a surface inversion trapping refractivity. The full dataset contradicts this:
|
||||
|
||||
| BL Depth | n | Avg Distance (km) | Median | Avg Refrac Gradient |
|
||||
|----------|---|-------------------|--------|---------------------|
|
||||
| 1000-2000m | 6,021 | **222.1** | 210.0 | -98.7 |
|
||||
| 500-1000m | 10,834 | 214.2 | 188.4 | -103.2 |
|
||||
| < 200m (shallow) | 7,020 | 209.6 | 178.8 | -114.0 |
|
||||
| 200-500m | 10,018 | 204.1 | 178.7 | -112.3 |
|
||||
| 2000m+ (deep) | 803 | 196.2 | 167.1 | -101.3 |
|
||||
|
||||
Medium-depth BL (1000-2000m) produces the longest average contacts, not shallow. Shallow BL does correlate with stronger refractivity gradients (-114 N/km avg), but this doesn't translate to longer contacts — likely because shallow BL often means fog/low stratus that attenuates the signal.
|
||||
|
||||
**Code change:** Removed the `@shallow_bl_threshold_m 300` / `@shallow_bl_score 82` refractivity fallback. Refractivity scoring now falls through to the default score when gradient is weak, regardless of BL depth.
|
||||
|
||||
## Finding 2: Td Depression at 24 GHz — Moist Range is Worst
|
||||
|
||||
At 24 GHz, the moist range (2-5C Td depression) produces the shortest contacts, while both dry and near-saturated conditions are better:
|
||||
|
||||
| Td Depression | n | Avg Distance (km) | Median | P90 |
|
||||
|--------------|---|-------------------|--------|-----|
|
||||
| 15+ (very dry) | 311 | **109.1** | 110.6 | 193.5 |
|
||||
| 0-2 (saturated) | 447 | 104.9 | 108.2 | 178.2 |
|
||||
| 10-15 (dry) | 278 | 99.5 | 97.8 | 179.7 |
|
||||
| 5-10 (moderate) | 841 | 94.9 | 87.1 | 177.2 |
|
||||
| 2-5 (moist) | 727 | **72.1** | 57.7 | 146.7 |
|
||||
|
||||
Near-saturation (0-2C) likely produces ducting that overcomes the absorption penalty, while the 2-5C range gives enough moisture for absorption without the refractivity benefit. Very dry conditions minimize absorption entirely.
|
||||
|
||||
**No code change** — the current harmful-band Td scoring already penalizes low depression. The existing curve is directionally correct; the moist-range penalty emerges naturally from the humidity and PWAT factors working together.
|
||||
|
||||
## Finding 3: PWAT Sweet Spot Confirmed at 20-30mm
|
||||
|
||||
The PWAT scoring curve already peaks at 20-30mm for beneficial bands:
|
||||
|
||||
| PWAT Bin | n | Avg Distance (km) | Median |
|
||||
|---------|---|-------------------|--------|
|
||||
| 20-30mm | 13,376 | **219.7** | 211.5 |
|
||||
| < 10mm | 463 | 211.8 | 195.5 |
|
||||
| 10-20mm | 8,500 | 208.3 | 183.5 |
|
||||
| 40-50mm | 3,118 | 207.7 | 177.3 |
|
||||
| 30-40mm | 8,947 | 203.5 | 171.8 |
|
||||
| 50mm+ | 292 | 190.5 | 173.7 |
|
||||
|
||||
The existing code's PWAT beneficial curve (`<10→55, <20→75, <30→90, <40→70, else→50`) is well-calibrated. No change needed.
|
||||
|
||||
## Finding 4: Pressure Relationship Has Fine Structure
|
||||
|
||||
Finer pressure binning reveals a clear monotonic relationship for the bulk of the data:
|
||||
|
||||
| Pressure Bin | n | Avg Distance (km) | Median |
|
||||
|-------------|---|-------------------|--------|
|
||||
| < 970 mb | 6,010 | **242.7** | 232.9 |
|
||||
| 970-980 | 5,254 | 229.8 | 225.5 |
|
||||
| 980-990 | 7,234 | 207.3 | 184.2 |
|
||||
| 990-1000 | 9,336 | 184.1 | 160.4 |
|
||||
| 1000-1010 | 3,505 | 200.7 | 161.0 |
|
||||
| 1010+ | 3,357 | 221.3 | 191.1 |
|
||||
|
||||
The <970 to 990-1000 range shows a clean 30% gradient. The uptick at 1010+ likely reflects high-altitude western US stations with inherently longer radio horizons, not a genuine pressure effect.
|
||||
|
||||
**Code change:** Refined pressure scoring bins to add a <980 tier (score 88) and increase the <1005 score from 80 to 85, better reflecting the strong low-pressure signal.
|
||||
|
||||
## Finding 5: Mode Distance Advantage Quantified
|
||||
|
||||
| Mode | n | Avg Distance (km) | Median | Max |
|
||||
|------|---|-------------------|--------|-----|
|
||||
| CW | 18,001 | **247.2** | 227.4 | 1,606 |
|
||||
| PH (SSB) | 35,089 | 190.9 | 176.2 | 999.2 |
|
||||
| FM | 180 | 141.2 | 116.5 | 664.1 |
|
||||
|
||||
CW achieves 29% longer average distance than SSB at 10 GHz, consistent with the ~7 dB bandwidth advantage (24% theoretical range increase). FM is 26% shorter than SSB. These ratios are useful for range estimate calibration.
|
||||
|
||||
## Finding 6: Refractivity Gradient is Flat Within the Bulk Distribution
|
||||
|
||||
| Gradient Bin | n | Avg Distance (km) | Median | Ducting % |
|
||||
|-------------|---|-------------------|--------|-----------|
|
||||
| < -200 (strong) | 659 | 202.2 | 189.0 | 100% |
|
||||
| -200 to -150 | 2,804 | 205.0 | 179.9 | 52.8% |
|
||||
| -150 to -100 | 14,113 | 211.8 | 186.6 | 0% |
|
||||
| -100 to -75 | 12,138 | **215.8** | 192.3 | 0% |
|
||||
| -75 to -55 | 4,390 | 207.5 | 180.0 | 0% |
|
||||
| >= -55 (weak) | 592 | 176.3 | 151.9 | 0% |
|
||||
|
||||
The -150 to -75 range (the bulk of HRRR gradients) shows remarkably flat distance distributions. Only the weakest gradients (>= -55) show meaningfully shorter contacts. The current 8% weight is appropriate given this weak discriminatory power.
|
||||
|
||||
## Finding 7: Dataset is 99.5% Aug-Sep
|
||||
|
||||
| Month | Contacts (tropo) | Avg Distance |
|
||||
|-------|-----------------|--------------|
|
||||
| Aug | 29,725 | 202.4 |
|
||||
| Sep | 28,468 | 198.7 |
|
||||
| All other months | 174 | varies |
|
||||
|
||||
All atmospheric correlations in this analysis are effectively summer-only findings. Seasonal scoring curves cannot be validated from this dataset. The physics-based seasonal tables remain the best available approach for non-summer months.
|
||||
|
||||
## Updated Dataset Statistics
|
||||
|
||||
- **Contacts:** 58,367 total, 58,367 tropo (<3000 km)
|
||||
- **HRRR profiles:** 41,071,398 (37,925 matched to contacts, 65% coverage)
|
||||
- **Terrain profiles:** 58,361 (56,735 BLOCKED 97.1%, 1,284 CLEAR 2.2%, 342 FRESNEL_PARTIAL 0.6%)
|
||||
- **IEMRE observations:** 15,092
|
||||
- **Weather stations:** 3,268
|
||||
- **Commercial samples:** 0 locally (available on production only)
|
||||
Loading…
Add table
Reference in a new issue