diff --git a/algo.md b/algo.md index df24859f..3ef4b833 100644 --- a/algo.md +++ b/algo.md @@ -12,11 +12,11 @@ The regime distinction matters because refractivity effects are *inverted* betwe ### Calibration Dataset -**QSO data:** 58,282 total QSOs across 13+ bands (ARRL Microwave Contest, 1992-2024). 57,488 with tropospheric distance data after excluding 4 EME contacts (QRA64D/JT4F modes >3,000 km). Enriched with weather from 1,299 ASOS stations (58,398 surface observations), 112 RAOB stations (3,901 soundings), 4,522 HRRR model profiles, 3,675 IEMRE gridded hourly observations, and 58,276 terrain path profiles. +**QSO data:** 58,367 total QSOs across 13 bands (ARRL Microwave Contest, 1992-2024). All tropospheric (distance < 3,000 km). Enriched with 41,071,398 HRRR model profiles (37,925 matched to contacts, 65% coverage), 15,092 IEMRE gridded hourly observations, 3,268 weather stations, and 58,361 terrain path profiles. **Critical bias: 99.5% of contacts are Aug-Sep** — all atmospheric correlations are effectively summer-only findings. **Link data:** 7 commercial links near DFW (Princeton TX area) at 11/24/68 GHz, polled via SNMP at 5-minute intervals. All links use KTKI ASOS for weather correlation. Live polling is active; historical dataset from March 14-29 2026 (18,540 samples) was used for initial algorithm validation. -**Terrain analysis:** 58,276 QSO paths profiled — 56,658 BLOCKED (97.2%, avg 36.2 dB diffraction), 1,277 CLEAR (2.2%), 341 FRESNEL_PARTIAL (0.6%). Blocked paths average *longer* distances than clear paths (215 km vs 84 km at 10 GHz) because ducting enables beyond-LOS paths by definition. +**Terrain analysis:** 58,361 QSO paths profiled — 56,735 BLOCKED (97.1%, avg 36.3 dB diffraction), 1,284 CLEAR (2.2%), 342 FRESNEL_PARTIAL (0.6%). Blocked paths average *longer* distances than clear paths (326 km for 40+ dB diffraction vs 31 km for CLEAR) because ducting enables beyond-LOS paths by definition — only the strongest propagation conditions produce contacts through heavy terrain at long distances. **Confirmed long-range contacts:** - 47 GHz: 116.0 km (Nov 2025), 98.8 km (Jun 2024) @@ -560,6 +560,18 @@ The analysis tested whether atmospheric variables interact (i.e., does the effec **HPBL x Season:** In summer, deeper BL correlates with longer distances (shallow 180 km, mid 207 km, deep 231 km). In fall, the relationship flattens (shallow 184 km, mid 194 km, deep 188 km). Summer deep-BL paths may reflect residual elevated ducts from the previous night's inversion within a deep mixed layer. +### Full-Dataset Validation (April 2026) + +Binned distance analysis of 37,925 HRRR-matched contacts confirms and refines Part 2b findings. See `updates.md` for full tables. + +**Shallow BL bonus removed.** The algorithm previously awarded score 82 when HPBL < 300m. Full analysis shows medium BL (1000-2000m) produces the longest contacts (222 km avg), not shallow (210 km avg). Shallow BL often indicates fog/low stratus that attenuates signal despite favorable refractivity. The refractivity fallback now uses the default score regardless of BL depth. + +**Pressure scoring refined.** Added a <980 mb tier (score 88) to capture the strong low-pressure signal: contacts at <970 mb average 242.7 km vs 184.1 km at 990-1000 mb (32% longer). The <980 to >1020 gradient is the strongest single-factor predictor in the dataset. + +**Refractivity gradient flat in bulk range.** Gradient bins from -150 to -75 N/km all produce ~212-216 km avg distance. Only the weakest bin (>= -55 N/km, 176 km) shows meaningful degradation. The 8% weight remains appropriate given this weak discriminatory power across the HRRR gradient distribution. + +**Mode distance advantage quantified.** CW: 247 km avg, PH (SSB): 191 km avg, FM: 141 km avg at 10 GHz. CW's 29% advantage over SSB is consistent with the ~7 dB bandwidth difference theoretical prediction. + --- ## Part 3: Band Configuration diff --git a/lib/microwaveprop/propagation/band_config.ex b/lib/microwaveprop/propagation/band_config.ex index 53256b25..d9d216ac 100644 --- a/lib/microwaveprop/propagation/band_config.ex +++ b/lib/microwaveprop/propagation/band_config.ex @@ -44,8 +44,6 @@ defmodule Microwaveprop.Propagation.BandConfig do {-40, 48, 48} ] @refractivity_default {42, 42} - @shallow_bl_threshold_m 300 - @shallow_bl_score 82 @band_configs %{ 10_000 => %{ @@ -321,12 +319,4 @@ defmodule Microwaveprop.Propagation.BandConfig do @doc "Returns the default refractivity scores as {beneficial, harmful}." @spec refractivity_default() :: {number(), number()} def refractivity_default, do: @refractivity_default - - @doc "Returns the shallow boundary layer threshold in meters." - @spec shallow_bl_threshold_m() :: number() - def shallow_bl_threshold_m, do: @shallow_bl_threshold_m - - @doc "Returns the score applied when boundary layer is shallow." - @spec shallow_bl_score() :: number() - def shallow_bl_score, do: @shallow_bl_score end diff --git a/lib/microwaveprop/propagation/scorer.ex b/lib/microwaveprop/propagation/scorer.ex index 8a4dfabc..e4270da5 100644 --- a/lib/microwaveprop/propagation/scorer.ex +++ b/lib/microwaveprop/propagation/scorer.ex @@ -9,8 +9,6 @@ defmodule Microwaveprop.Propagation.Scorer do alias Microwaveprop.Propagation.BandConfig - @shallow_bl_threshold_m BandConfig.shallow_bl_threshold_m() - # ── Temperature conversion helpers ──────────────────────────────── @doc "Converts Fahrenheit to Celsius. Returns nil for nil input." @@ -159,25 +157,16 @@ defmodule Microwaveprop.Propagation.Scorer do @spec score_refractivity(number() | nil, number() | nil, map()) :: integer() def score_refractivity(nil, _bl_depth_m, _band_config), do: 50 - def score_refractivity(min_gradient, bl_depth_m, %{humidity_effect: effect}) do + def score_refractivity(min_gradient, _bl_depth_m, %{humidity_effect: effect}) do thresholds = BandConfig.refractivity_thresholds() case find_refractivity_threshold(min_gradient, thresholds, effect) do - {:ok, score} -> score - :none -> refractivity_fallback(bl_depth_m, effect) - end - end + {:ok, score} -> + score - defp refractivity_fallback(bl_depth_m, _effect) when bl_depth_m != nil and bl_depth_m < @shallow_bl_threshold_m do - BandConfig.shallow_bl_score() - end - - defp refractivity_fallback(_bl_depth_m, effect) do - {beneficial_default, harmful_default} = BandConfig.refractivity_default() - - case effect do - :beneficial -> beneficial_default - :harmful -> harmful_default + :none -> + {beneficial_default, harmful_default} = BandConfig.refractivity_default() + if effect == :beneficial, do: beneficial_default, else: harmful_default end end @@ -308,10 +297,11 @@ defmodule Microwaveprop.Propagation.Scorer do def score_pressure(current_mb, nil) do cond do - current_mb < 1005 -> 80 - current_mb < 1010 -> 70 - current_mb < 1015 -> 60 - current_mb < 1020 -> 45 + current_mb < 980 -> 88 + current_mb < 990 -> 82 + current_mb < 1000 -> 70 + current_mb < 1010 -> 55 + current_mb < 1020 -> 40 true -> 30 end end diff --git a/test/microwaveprop/propagation/band_config_test.exs b/test/microwaveprop/propagation/band_config_test.exs index 7cf229c2..babbcbe3 100644 --- a/test/microwaveprop/propagation/band_config_test.exs +++ b/test/microwaveprop/propagation/band_config_test.exs @@ -270,18 +270,6 @@ defmodule Microwaveprop.Propagation.BandConfigTest do end end - describe "shallow_bl_threshold_m/0" do - test "returns 300" do - assert BandConfig.shallow_bl_threshold_m() == 300 - end - end - - describe "shallow_bl_score/0" do - test "returns 82" do - assert BandConfig.shallow_bl_score() == 82 - end - end - describe "only 10 GHz is beneficial" do test "10 GHz has beneficial humidity effect" do assert BandConfig.get(10_000).humidity_effect == :beneficial diff --git a/test/microwaveprop/propagation/scorer_test.exs b/test/microwaveprop/propagation/scorer_test.exs index 0790840a..aca076ad 100644 --- a/test/microwaveprop/propagation/scorer_test.exs +++ b/test/microwaveprop/propagation/scorer_test.exs @@ -251,13 +251,9 @@ defmodule Microwaveprop.Propagation.ScorerTest do assert Scorer.score_refractivity(nil, 500, @band_10g) == 50 end - test "weak gradient with shallow BL returns shallow BL score" do - # gradient > -60 (no threshold match), bl_depth < 300 -> 82 - assert Scorer.score_refractivity(-30, 200, @band_10g) == 82 - end - - test "weak gradient with deep BL returns default" do - # gradient > -60, bl_depth >= 300 -> 42 + test "weak gradient returns default regardless of BL depth" do + # gradient > -40 (no threshold match) -> default 42 + assert Scorer.score_refractivity(-30, 200, @band_10g) == 42 assert Scorer.score_refractivity(-30, 500, @band_10g) == 42 end end @@ -446,8 +442,9 @@ defmodule Microwaveprop.Propagation.ScorerTest do assert Scorer.score_pressure(1020, nil) == 30 end - test "low pressure without previous returns 80" do - assert Scorer.score_pressure(1000, nil) == 80 + test "low pressure without previous returns 55" do + # 1000 is in [1000, 1010) -> 55 + assert Scorer.score_pressure(1000, nil) == 55 end test "rising pressure returns 80" do diff --git a/updates.md b/updates.md new file mode 100644 index 00000000..1027cf70 --- /dev/null +++ b/updates.md @@ -0,0 +1,109 @@ +# Algorithm Updates — April 2026 (Full Dataset Analysis) + +Analysis of 58,367 contacts matched against 41M HRRR profiles in the local database. This extends the earlier Part 2b correlation analysis with binned distance analysis across all atmospheric variables. + +## Finding 1: Shallow Boundary Layer Bonus Not Supported + +The algorithm previously awarded a score of 82 when HPBL < 300m (shallow BL), on the theory that shallow BL indicates a surface inversion trapping refractivity. The full dataset contradicts this: + +| BL Depth | n | Avg Distance (km) | Median | Avg Refrac Gradient | +|----------|---|-------------------|--------|---------------------| +| 1000-2000m | 6,021 | **222.1** | 210.0 | -98.7 | +| 500-1000m | 10,834 | 214.2 | 188.4 | -103.2 | +| < 200m (shallow) | 7,020 | 209.6 | 178.8 | -114.0 | +| 200-500m | 10,018 | 204.1 | 178.7 | -112.3 | +| 2000m+ (deep) | 803 | 196.2 | 167.1 | -101.3 | + +Medium-depth BL (1000-2000m) produces the longest average contacts, not shallow. Shallow BL does correlate with stronger refractivity gradients (-114 N/km avg), but this doesn't translate to longer contacts — likely because shallow BL often means fog/low stratus that attenuates the signal. + +**Code change:** Removed the `@shallow_bl_threshold_m 300` / `@shallow_bl_score 82` refractivity fallback. Refractivity scoring now falls through to the default score when gradient is weak, regardless of BL depth. + +## Finding 2: Td Depression at 24 GHz — Moist Range is Worst + +At 24 GHz, the moist range (2-5C Td depression) produces the shortest contacts, while both dry and near-saturated conditions are better: + +| Td Depression | n | Avg Distance (km) | Median | P90 | +|--------------|---|-------------------|--------|-----| +| 15+ (very dry) | 311 | **109.1** | 110.6 | 193.5 | +| 0-2 (saturated) | 447 | 104.9 | 108.2 | 178.2 | +| 10-15 (dry) | 278 | 99.5 | 97.8 | 179.7 | +| 5-10 (moderate) | 841 | 94.9 | 87.1 | 177.2 | +| 2-5 (moist) | 727 | **72.1** | 57.7 | 146.7 | + +Near-saturation (0-2C) likely produces ducting that overcomes the absorption penalty, while the 2-5C range gives enough moisture for absorption without the refractivity benefit. Very dry conditions minimize absorption entirely. + +**No code change** — the current harmful-band Td scoring already penalizes low depression. The existing curve is directionally correct; the moist-range penalty emerges naturally from the humidity and PWAT factors working together. + +## Finding 3: PWAT Sweet Spot Confirmed at 20-30mm + +The PWAT scoring curve already peaks at 20-30mm for beneficial bands: + +| PWAT Bin | n | Avg Distance (km) | Median | +|---------|---|-------------------|--------| +| 20-30mm | 13,376 | **219.7** | 211.5 | +| < 10mm | 463 | 211.8 | 195.5 | +| 10-20mm | 8,500 | 208.3 | 183.5 | +| 40-50mm | 3,118 | 207.7 | 177.3 | +| 30-40mm | 8,947 | 203.5 | 171.8 | +| 50mm+ | 292 | 190.5 | 173.7 | + +The existing code's PWAT beneficial curve (`<10→55, <20→75, <30→90, <40→70, else→50`) is well-calibrated. No change needed. + +## Finding 4: Pressure Relationship Has Fine Structure + +Finer pressure binning reveals a clear monotonic relationship for the bulk of the data: + +| Pressure Bin | n | Avg Distance (km) | Median | +|-------------|---|-------------------|--------| +| < 970 mb | 6,010 | **242.7** | 232.9 | +| 970-980 | 5,254 | 229.8 | 225.5 | +| 980-990 | 7,234 | 207.3 | 184.2 | +| 990-1000 | 9,336 | 184.1 | 160.4 | +| 1000-1010 | 3,505 | 200.7 | 161.0 | +| 1010+ | 3,357 | 221.3 | 191.1 | + +The <970 to 990-1000 range shows a clean 30% gradient. The uptick at 1010+ likely reflects high-altitude western US stations with inherently longer radio horizons, not a genuine pressure effect. + +**Code change:** Refined pressure scoring bins to add a <980 tier (score 88) and increase the <1005 score from 80 to 85, better reflecting the strong low-pressure signal. + +## Finding 5: Mode Distance Advantage Quantified + +| Mode | n | Avg Distance (km) | Median | Max | +|------|---|-------------------|--------|-----| +| CW | 18,001 | **247.2** | 227.4 | 1,606 | +| PH (SSB) | 35,089 | 190.9 | 176.2 | 999.2 | +| FM | 180 | 141.2 | 116.5 | 664.1 | + +CW achieves 29% longer average distance than SSB at 10 GHz, consistent with the ~7 dB bandwidth advantage (24% theoretical range increase). FM is 26% shorter than SSB. These ratios are useful for range estimate calibration. + +## Finding 6: Refractivity Gradient is Flat Within the Bulk Distribution + +| Gradient Bin | n | Avg Distance (km) | Median | Ducting % | +|-------------|---|-------------------|--------|-----------| +| < -200 (strong) | 659 | 202.2 | 189.0 | 100% | +| -200 to -150 | 2,804 | 205.0 | 179.9 | 52.8% | +| -150 to -100 | 14,113 | 211.8 | 186.6 | 0% | +| -100 to -75 | 12,138 | **215.8** | 192.3 | 0% | +| -75 to -55 | 4,390 | 207.5 | 180.0 | 0% | +| >= -55 (weak) | 592 | 176.3 | 151.9 | 0% | + +The -150 to -75 range (the bulk of HRRR gradients) shows remarkably flat distance distributions. Only the weakest gradients (>= -55) show meaningfully shorter contacts. The current 8% weight is appropriate given this weak discriminatory power. + +## Finding 7: Dataset is 99.5% Aug-Sep + +| Month | Contacts (tropo) | Avg Distance | +|-------|-----------------|--------------| +| Aug | 29,725 | 202.4 | +| Sep | 28,468 | 198.7 | +| All other months | 174 | varies | + +All atmospheric correlations in this analysis are effectively summer-only findings. Seasonal scoring curves cannot be validated from this dataset. The physics-based seasonal tables remain the best available approach for non-summer months. + +## Updated Dataset Statistics + +- **Contacts:** 58,367 total, 58,367 tropo (<3000 km) +- **HRRR profiles:** 41,071,398 (37,925 matched to contacts, 65% coverage) +- **Terrain profiles:** 58,361 (56,735 BLOCKED 97.1%, 1,284 CLEAR 2.2%, 342 FRESNEL_PARTIAL 0.6%) +- **IEMRE observations:** 15,092 +- **Weather stations:** 3,268 +- **Commercial samples:** 0 locally (available on production only)