Graham McIntire
b2f3098083
Analyzed 37,925 HRRR-matched contacts from 58,367 total. - Remove shallow BL bonus (score 82 for HPBL < 300m): data shows medium BL (1000-2000m) produces longest contacts (222 km avg), not shallow (210 km). Refractivity fallback now uses default score. - Refine pressure scoring bins: add <980 mb tier (score 88), steeper gradient from low to high. Contacts at <970 mb avg 242.7 km vs 184.1 km at 990-1000 mb. - Update algo.md calibration stats (58,367 contacts, 41M HRRR profiles, 58,361 terrain profiles) and document dataset bias (99.5% Aug-Sep). - Create updates.md with full binned analysis tables for all factors.
109 lines
5.8 KiB
Markdown
109 lines
5.8 KiB
Markdown
# Algorithm Updates — April 2026 (Full Dataset Analysis)
|
|
|
|
Analysis of 58,367 contacts matched against 41M HRRR profiles in the local database. This extends the earlier Part 2b correlation analysis with binned distance analysis across all atmospheric variables.
|
|
|
|
## Finding 1: Shallow Boundary Layer Bonus Not Supported
|
|
|
|
The algorithm previously awarded a score of 82 when HPBL < 300m (shallow BL), on the theory that shallow BL indicates a surface inversion trapping refractivity. The full dataset contradicts this:
|
|
|
|
| BL Depth | n | Avg Distance (km) | Median | Avg Refrac Gradient |
|
|
|----------|---|-------------------|--------|---------------------|
|
|
| 1000-2000m | 6,021 | **222.1** | 210.0 | -98.7 |
|
|
| 500-1000m | 10,834 | 214.2 | 188.4 | -103.2 |
|
|
| < 200m (shallow) | 7,020 | 209.6 | 178.8 | -114.0 |
|
|
| 200-500m | 10,018 | 204.1 | 178.7 | -112.3 |
|
|
| 2000m+ (deep) | 803 | 196.2 | 167.1 | -101.3 |
|
|
|
|
Medium-depth BL (1000-2000m) produces the longest average contacts, not shallow. Shallow BL does correlate with stronger refractivity gradients (-114 N/km avg), but this doesn't translate to longer contacts — likely because shallow BL often means fog/low stratus that attenuates the signal.
|
|
|
|
**Code change:** Removed the `@shallow_bl_threshold_m 300` / `@shallow_bl_score 82` refractivity fallback. Refractivity scoring now falls through to the default score when gradient is weak, regardless of BL depth.
|
|
|
|
## Finding 2: Td Depression at 24 GHz — Moist Range is Worst
|
|
|
|
At 24 GHz, the moist range (2-5C Td depression) produces the shortest contacts, while both dry and near-saturated conditions are better:
|
|
|
|
| Td Depression | n | Avg Distance (km) | Median | P90 |
|
|
|--------------|---|-------------------|--------|-----|
|
|
| 15+ (very dry) | 311 | **109.1** | 110.6 | 193.5 |
|
|
| 0-2 (saturated) | 447 | 104.9 | 108.2 | 178.2 |
|
|
| 10-15 (dry) | 278 | 99.5 | 97.8 | 179.7 |
|
|
| 5-10 (moderate) | 841 | 94.9 | 87.1 | 177.2 |
|
|
| 2-5 (moist) | 727 | **72.1** | 57.7 | 146.7 |
|
|
|
|
Near-saturation (0-2C) likely produces ducting that overcomes the absorption penalty, while the 2-5C range gives enough moisture for absorption without the refractivity benefit. Very dry conditions minimize absorption entirely.
|
|
|
|
**No code change** — the current harmful-band Td scoring already penalizes low depression. The existing curve is directionally correct; the moist-range penalty emerges naturally from the humidity and PWAT factors working together.
|
|
|
|
## Finding 3: PWAT Sweet Spot Confirmed at 20-30mm
|
|
|
|
The PWAT scoring curve already peaks at 20-30mm for beneficial bands:
|
|
|
|
| PWAT Bin | n | Avg Distance (km) | Median |
|
|
|---------|---|-------------------|--------|
|
|
| 20-30mm | 13,376 | **219.7** | 211.5 |
|
|
| < 10mm | 463 | 211.8 | 195.5 |
|
|
| 10-20mm | 8,500 | 208.3 | 183.5 |
|
|
| 40-50mm | 3,118 | 207.7 | 177.3 |
|
|
| 30-40mm | 8,947 | 203.5 | 171.8 |
|
|
| 50mm+ | 292 | 190.5 | 173.7 |
|
|
|
|
The existing code's PWAT beneficial curve (`<10→55, <20→75, <30→90, <40→70, else→50`) is well-calibrated. No change needed.
|
|
|
|
## Finding 4: Pressure Relationship Has Fine Structure
|
|
|
|
Finer pressure binning reveals a clear monotonic relationship for the bulk of the data:
|
|
|
|
| Pressure Bin | n | Avg Distance (km) | Median |
|
|
|-------------|---|-------------------|--------|
|
|
| < 970 mb | 6,010 | **242.7** | 232.9 |
|
|
| 970-980 | 5,254 | 229.8 | 225.5 |
|
|
| 980-990 | 7,234 | 207.3 | 184.2 |
|
|
| 990-1000 | 9,336 | 184.1 | 160.4 |
|
|
| 1000-1010 | 3,505 | 200.7 | 161.0 |
|
|
| 1010+ | 3,357 | 221.3 | 191.1 |
|
|
|
|
The <970 to 990-1000 range shows a clean 30% gradient. The uptick at 1010+ likely reflects high-altitude western US stations with inherently longer radio horizons, not a genuine pressure effect.
|
|
|
|
**Code change:** Refined pressure scoring bins to add a <980 tier (score 88) and increase the <1005 score from 80 to 85, better reflecting the strong low-pressure signal.
|
|
|
|
## Finding 5: Mode Distance Advantage Quantified
|
|
|
|
| Mode | n | Avg Distance (km) | Median | Max |
|
|
|------|---|-------------------|--------|-----|
|
|
| CW | 18,001 | **247.2** | 227.4 | 1,606 |
|
|
| PH (SSB) | 35,089 | 190.9 | 176.2 | 999.2 |
|
|
| FM | 180 | 141.2 | 116.5 | 664.1 |
|
|
|
|
CW achieves 29% longer average distance than SSB at 10 GHz, consistent with the ~7 dB bandwidth advantage (24% theoretical range increase). FM is 26% shorter than SSB. These ratios are useful for range estimate calibration.
|
|
|
|
## Finding 6: Refractivity Gradient is Flat Within the Bulk Distribution
|
|
|
|
| Gradient Bin | n | Avg Distance (km) | Median | Ducting % |
|
|
|-------------|---|-------------------|--------|-----------|
|
|
| < -200 (strong) | 659 | 202.2 | 189.0 | 100% |
|
|
| -200 to -150 | 2,804 | 205.0 | 179.9 | 52.8% |
|
|
| -150 to -100 | 14,113 | 211.8 | 186.6 | 0% |
|
|
| -100 to -75 | 12,138 | **215.8** | 192.3 | 0% |
|
|
| -75 to -55 | 4,390 | 207.5 | 180.0 | 0% |
|
|
| >= -55 (weak) | 592 | 176.3 | 151.9 | 0% |
|
|
|
|
The -150 to -75 range (the bulk of HRRR gradients) shows remarkably flat distance distributions. Only the weakest gradients (>= -55) show meaningfully shorter contacts. The current 8% weight is appropriate given this weak discriminatory power.
|
|
|
|
## Finding 7: Dataset is 99.5% Aug-Sep
|
|
|
|
| Month | Contacts (tropo) | Avg Distance |
|
|
|-------|-----------------|--------------|
|
|
| Aug | 29,725 | 202.4 |
|
|
| Sep | 28,468 | 198.7 |
|
|
| All other months | 174 | varies |
|
|
|
|
All atmospheric correlations in this analysis are effectively summer-only findings. Seasonal scoring curves cannot be validated from this dataset. The physics-based seasonal tables remain the best available approach for non-summer months.
|
|
|
|
## Updated Dataset Statistics
|
|
|
|
- **Contacts:** 58,367 total, 58,367 tropo (<3000 km)
|
|
- **HRRR profiles:** 41,071,398 (37,925 matched to contacts, 65% coverage)
|
|
- **Terrain profiles:** 58,361 (56,735 BLOCKED 97.1%, 1,284 CLEAR 2.2%, 342 FRESNEL_PARTIAL 0.6%)
|
|
- **IEMRE observations:** 15,092
|
|
- **Weather stations:** 3,268
|
|
- **Commercial samples:** 0 locally (available on production only)
|