Graham McIntire
2eefe64c9b
Soundings (10,935): October has strongest gradients (-350 N/km) and 65.7% ducting, raised seasonal score 82→88. February ducting (20.2%) exceeds January (14.8%), raised 32→40. Commercial links (12,068 samples): 68 GHz shows 3.9 dB diurnal swing, 11 GHz shows 1.7 dB with inverted multipath pattern, 24 GHz remarkably stable at 0.9 dB. Diurnal sensitivity is non-monotonic with frequency. RAOB gradients avg -265 N/km vs HRRR -107 (2.5x stronger) — confirms HRRR misses thin surface ducts that soundings resolve.
200 lines
10 KiB
Markdown
200 lines
10 KiB
Markdown
# Algorithm Updates — April 2026 (Full Dataset Analysis)
|
|
|
|
Analysis of 58,367 contacts matched against 41M HRRR profiles in the local database. This extends the earlier Part 2b correlation analysis with binned distance analysis across all atmospheric variables.
|
|
|
|
## Finding 1: Shallow Boundary Layer Bonus Not Supported
|
|
|
|
The algorithm previously awarded a score of 82 when HPBL < 300m (shallow BL), on the theory that shallow BL indicates a surface inversion trapping refractivity. The full dataset contradicts this:
|
|
|
|
| BL Depth | n | Avg Distance (km) | Median | Avg Refrac Gradient |
|
|
|----------|---|-------------------|--------|---------------------|
|
|
| 1000-2000m | 6,021 | **222.1** | 210.0 | -98.7 |
|
|
| 500-1000m | 10,834 | 214.2 | 188.4 | -103.2 |
|
|
| < 200m (shallow) | 7,020 | 209.6 | 178.8 | -114.0 |
|
|
| 200-500m | 10,018 | 204.1 | 178.7 | -112.3 |
|
|
| 2000m+ (deep) | 803 | 196.2 | 167.1 | -101.3 |
|
|
|
|
Medium-depth BL (1000-2000m) produces the longest average contacts, not shallow. Shallow BL does correlate with stronger refractivity gradients (-114 N/km avg), but this doesn't translate to longer contacts — likely because shallow BL often means fog/low stratus that attenuates the signal.
|
|
|
|
**Code change:** Removed the `@shallow_bl_threshold_m 300` / `@shallow_bl_score 82` refractivity fallback. Refractivity scoring now falls through to the default score when gradient is weak, regardless of BL depth.
|
|
|
|
## Finding 2: Td Depression at 24 GHz — Moist Range is Worst
|
|
|
|
At 24 GHz, the moist range (2-5C Td depression) produces the shortest contacts, while both dry and near-saturated conditions are better:
|
|
|
|
| Td Depression | n | Avg Distance (km) | Median | P90 |
|
|
|--------------|---|-------------------|--------|-----|
|
|
| 15+ (very dry) | 311 | **109.1** | 110.6 | 193.5 |
|
|
| 0-2 (saturated) | 447 | 104.9 | 108.2 | 178.2 |
|
|
| 10-15 (dry) | 278 | 99.5 | 97.8 | 179.7 |
|
|
| 5-10 (moderate) | 841 | 94.9 | 87.1 | 177.2 |
|
|
| 2-5 (moist) | 727 | **72.1** | 57.7 | 146.7 |
|
|
|
|
Near-saturation (0-2C) likely produces ducting that overcomes the absorption penalty, while the 2-5C range gives enough moisture for absorption without the refractivity benefit. Very dry conditions minimize absorption entirely.
|
|
|
|
**No code change** — the current harmful-band Td scoring already penalizes low depression. The existing curve is directionally correct; the moist-range penalty emerges naturally from the humidity and PWAT factors working together.
|
|
|
|
## Finding 3: PWAT Sweet Spot Confirmed at 20-30mm
|
|
|
|
The PWAT scoring curve already peaks at 20-30mm for beneficial bands:
|
|
|
|
| PWAT Bin | n | Avg Distance (km) | Median |
|
|
|---------|---|-------------------|--------|
|
|
| 20-30mm | 13,376 | **219.7** | 211.5 |
|
|
| < 10mm | 463 | 211.8 | 195.5 |
|
|
| 10-20mm | 8,500 | 208.3 | 183.5 |
|
|
| 40-50mm | 3,118 | 207.7 | 177.3 |
|
|
| 30-40mm | 8,947 | 203.5 | 171.8 |
|
|
| 50mm+ | 292 | 190.5 | 173.7 |
|
|
|
|
The existing code's PWAT beneficial curve (`<10→55, <20→75, <30→90, <40→70, else→50`) is well-calibrated. No change needed.
|
|
|
|
## Finding 4: Pressure Relationship Has Fine Structure
|
|
|
|
Finer pressure binning reveals a clear monotonic relationship for the bulk of the data:
|
|
|
|
| Pressure Bin | n | Avg Distance (km) | Median |
|
|
|-------------|---|-------------------|--------|
|
|
| < 970 mb | 6,010 | **242.7** | 232.9 |
|
|
| 970-980 | 5,254 | 229.8 | 225.5 |
|
|
| 980-990 | 7,234 | 207.3 | 184.2 |
|
|
| 990-1000 | 9,336 | 184.1 | 160.4 |
|
|
| 1000-1010 | 3,505 | 200.7 | 161.0 |
|
|
| 1010+ | 3,357 | 221.3 | 191.1 |
|
|
|
|
The <970 to 990-1000 range shows a clean 30% gradient. The uptick at 1010+ likely reflects high-altitude western US stations with inherently longer radio horizons, not a genuine pressure effect.
|
|
|
|
**Code change:** Refined pressure scoring bins to add a <980 tier (score 88) and increase the <1005 score from 80 to 85, better reflecting the strong low-pressure signal.
|
|
|
|
## Finding 5: Mode Distance Advantage Quantified
|
|
|
|
| Mode | n | Avg Distance (km) | Median | Max |
|
|
|------|---|-------------------|--------|-----|
|
|
| CW | 18,001 | **247.2** | 227.4 | 1,606 |
|
|
| PH (SSB) | 35,089 | 190.9 | 176.2 | 999.2 |
|
|
| FM | 180 | 141.2 | 116.5 | 664.1 |
|
|
|
|
CW achieves 29% longer average distance than SSB at 10 GHz, consistent with the ~7 dB bandwidth advantage (24% theoretical range increase). FM is 26% shorter than SSB. These ratios are useful for range estimate calibration.
|
|
|
|
## Finding 6: Refractivity Gradient is Flat Within the Bulk Distribution
|
|
|
|
| Gradient Bin | n | Avg Distance (km) | Median | Ducting % |
|
|
|-------------|---|-------------------|--------|-----------|
|
|
| < -200 (strong) | 659 | 202.2 | 189.0 | 100% |
|
|
| -200 to -150 | 2,804 | 205.0 | 179.9 | 52.8% |
|
|
| -150 to -100 | 14,113 | 211.8 | 186.6 | 0% |
|
|
| -100 to -75 | 12,138 | **215.8** | 192.3 | 0% |
|
|
| -75 to -55 | 4,390 | 207.5 | 180.0 | 0% |
|
|
| >= -55 (weak) | 592 | 176.3 | 151.9 | 0% |
|
|
|
|
The -150 to -75 range (the bulk of HRRR gradients) shows remarkably flat distance distributions. Only the weakest gradients (>= -55) show meaningfully shorter contacts. The current 8% weight is appropriate given this weak discriminatory power.
|
|
|
|
## Finding 7: Dataset is 99.5% Aug-Sep
|
|
|
|
| Month | Contacts (tropo) | Avg Distance |
|
|
|-------|-----------------|--------------|
|
|
| Aug | 29,725 | 202.4 |
|
|
| Sep | 28,468 | 198.7 |
|
|
| All other months | 174 | varies |
|
|
|
|
All atmospheric correlations in this analysis are effectively summer-only findings. Seasonal scoring curves cannot be validated from this dataset. The physics-based seasonal tables remain the best available approach for non-summer months.
|
|
|
|
## Finding 8: Commercial Link Diurnal Patterns (12,068 samples, 7 links)
|
|
|
|
Data from March 30 - April 7, 2026. All links in Princeton TX area, LOS paths.
|
|
|
|
### 68 GHz (af60, 2.82 km) — Strongest diurnal signal
|
|
|
|
| UTC Hour | Avg RX (dBm) | Std Dev | Notes |
|
|
|----------|-------------|---------|-------|
|
|
| 09-10 | **-50.7** | 0.80 | Morning best |
|
|
| 00-08 | -51.2 to -52.2 | 0.6-1.2 | Night — gradual improvement |
|
|
| 11-12 | -52.1 to -52.3 | 4-5 | Weather events (high std) |
|
|
| 13-14 | **-54.3 to -54.6** | 5-6 | Afternoon worst |
|
|
| 15-23 | -52.8 to -53.4 | 0.8-3.7 | Evening recovery |
|
|
|
|
**3.9 dB diurnal swing.** Afternoon degradation at 68 GHz is dominated by gaseous absorption increase with daytime heating and humidity rise, plus convective scintillation. The high stddev at 11-15 UTC indicates individual weather events (rain, convection) can cause 10+ dB excursions.
|
|
|
|
### 11 GHz (core-new-hope, 5.66 km) — Inverted pattern
|
|
|
|
| UTC Hour | Avg RX (dBm) | Std Dev | Notes |
|
|
|----------|-------------|---------|-------|
|
|
| 08-12 | **-49.7 to -49.8** | 2.1-2.5 | Morning best, but variable |
|
|
| 01-07 | -49.8 to -50.1 | 1.7-2.0 | Night — slightly better mean, multipath |
|
|
| 14-21 | -50.8 to -51.4 | 1.2-1.5 | Afternoon/evening — worse mean, less variable |
|
|
|
|
**1.7 dB diurnal swing.** At 11 GHz, gaseous absorption is negligible. The diurnal pattern is driven by **multipath fading** from nocturnal inversions (higher stddev at night: 2.0-2.5 vs afternoon: 1.2-1.4). Mean signal is slightly better at night despite the multipath, because the refractivity enhancement aids the direct path.
|
|
|
|
### 24 GHz (climax-to-core, 4.36 km) — Remarkably stable
|
|
|
|
**Only 0.9 dB diurnal swing** (-63.4 to -64.3). The 22.235 GHz H₂O line creates a constant absorption floor that varies minimally with diurnal moisture changes on short paths. One anomaly at 12 UTC (stddev 2.51) is likely a single rain event.
|
|
|
|
### Key insight: Diurnal sensitivity by frequency on LOS paths
|
|
- 11 GHz: 1.7 dB (multipath-dominated)
|
|
- 24 GHz: 0.9 dB (absorption floor, stable)
|
|
- 68 GHz: 3.9 dB (gaseous absorption-dominated)
|
|
|
|
This is **non-monotonic** — 24 GHz is more stable than 11 GHz. The H₂O absorption at 24 GHz is a constant penalty that doesn't fluctuate much diurnally, while 11 GHz refractivity effects create multipath that varies with inversion development.
|
|
|
|
## Finding 9: RAOB Soundings Resolve Ducting HRRR Cannot (10,935 soundings)
|
|
|
|
### RAOB vs HRRR gradient distributions
|
|
|
|
| Statistic | RAOB | HRRR (at contacts) |
|
|
|-----------|------|-------------------|
|
|
| Average | -265.4 N/km | -107.2 N/km |
|
|
| P10 | -521.7 | ~-200 |
|
|
| P50 (median) | -199.5 | ~-75 |
|
|
| P90 | -83.3 | ~-50 |
|
|
| Min | -1969.9 | — |
|
|
|
|
RAOB gradients are **2.5x stronger** than HRRR because soundings at ~10m vertical resolution capture thin surface ducts (50-100m) that HRRR's 25 hPa pressure level spacing (~250m) cannot resolve.
|
|
|
|
### Sharp ducting threshold at -200 to -300 N/km
|
|
|
|
| Gradient Bin | n | Ducting % | Avg PWAT (mm) |
|
|
|-------------|---|-----------|---------------|
|
|
| < -500 | 622 | **100%** | 26.6 |
|
|
| -500 to -300 | 967 | **99.9%** | 26.9 |
|
|
| -300 to -200 | 1,168 | **91.2%** | 29.9 |
|
|
| -200 to -150 | 879 | 31.2% | 28.5 |
|
|
| -150 to -100 | 1,061 | 1.8% | 28.7 |
|
|
| -100 to -50 | 683 | 2.2% | 27.0 |
|
|
| >= -50 | 159 | 1.9% | 20.0 |
|
|
|
|
The transition from "always ducting" to "rarely ducting" occurs between -200 and -300 N/km. This is consistent with the theoretical ducting threshold of -157 N/km (where k goes negative), with the additional gradient needed to sustain a duct layer of practical depth.
|
|
|
|
### Seasonal ducting rates from RAOB
|
|
|
|
| Month | Total | Ducting % | Avg Gradient | Avg PWAT (mm) |
|
|
|-------|-------|-----------|-------------|---------------|
|
|
| Jun | 232 | **70.3%** | -339.9 | 28.5 |
|
|
| Jul | 247 | **71.3%** | -308.0 | 28.1 |
|
|
| Oct | 99 | **65.7%** | **-350.0** | 18.9 |
|
|
| Sep | 2,073 | 56.0% | -282.3 | 26.3 |
|
|
| Aug | 2,281 | 53.7% | -258.2 | 34.0 |
|
|
| Nov | 99 | 49.5% | -248.9 | 10.8 |
|
|
| May | 128 | 41.4% | -227.4 | 20.4 |
|
|
| Apr | 86 | 32.6% | -155.1 | 13.7 |
|
|
| Feb | 84 | 20.2% | -135.4 | 6.8 |
|
|
| Jan | 61 | 14.8% | -126.3 | 8.9 |
|
|
| Mar | 54 | 14.8% | -140.9 | 10.5 |
|
|
| Dec | 100 | 12.0% | -138.7 | 8.0 |
|
|
|
|
**October is the most underrated month.** It has the strongest average gradients of any month (-350 N/km), 65.7% ducting rate (3rd highest), but was scored only 82/100 in the 10 GHz seasonal table.
|
|
|
|
**February was scored below January despite higher ducting.** Feb: 20.2% ducting, was scored 32. Jan: 14.8% ducting, scored 38.
|
|
|
|
**Code changes:**
|
|
- 10 GHz February seasonal score: 32 → 40 (now exceeds January, matching its higher ducting rate)
|
|
- 10 GHz October seasonal score: 82 → 88 (3rd highest ducting, strongest gradients)
|
|
|
|
## Updated Dataset Statistics
|
|
|
|
- **Contacts:** 58,367 total, 58,367 tropo (<3000 km)
|
|
- **HRRR profiles:** 41,071,398 (37,925 matched to contacts, 65% coverage)
|
|
- **Terrain profiles:** 58,361 (56,735 BLOCKED 97.1%, 1,284 CLEAR 2.2%, 342 FRESNEL_PARTIAL 0.6%)
|
|
- **Soundings:** 10,935 (5,539 with gradient data, 2,964 with ducting detected)
|
|
- **Commercial samples:** 12,068 (7 links, March 30 - April 7, 2026)
|
|
- **IEMRE observations:** 15,092
|
|
- **Weather stations:** 3,268
|