From 880988b591f16c1e5b4e24cf76291216a5ab1e44 Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Sat, 11 Apr 2026 13:44:12 -0500 Subject: [PATCH] Update algo.md weights and data flow for native duct integration - Replace stale April 2026 manual weights with recalibrated values - Document native hybrid-sigma data in data flow section - Note refractivity factor now uses native 10-50m resolution - Add hourly grid integration section to Part 12 - Expose duct_info (count, freq, thickness) in scoring factors for UI --- algo.md | 78 ++++++++++++++++++++------------ lib/microwaveprop/propagation.ex | 12 ++++- 2 files changed, 61 insertions(+), 29 deletions(-) diff --git a/algo.md b/algo.md index 0728c6fb..4ac6e1bd 100644 --- a/algo.md +++ b/algo.md @@ -1126,31 +1126,29 @@ end ### Weights -Revised April 2026 based on 57,248 QSO-HRRR correlation analysis. Key changes from previous weights: +Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12, 72% improvement). Key changes from the April 2026 manual calibration: -- **Pressure: 4% → 15%** — #1 correlator at 10 GHz (rho=-0.180); data shows low P (<1005 mb) gives 197 km median vs 103 km for >1020 mb. Previously scored backwards (high=good). Now the largest single-factor weight increase in algorithm history. -- **PWAT: 0% → 10%** — New factor. Strong independent predictor across all bands (rho=-0.039 to -0.608). Captures column-integrated moisture not represented by surface humidity or Td depression. -- **Time of Day: 20% → 10%** — rho=0.007 at 10 GHz; effect is real at 24+ GHz (rho=-0.392 at 75 GHz) but the 20% weight was unwarranted at the dominant 10 GHz band. Modest even at 24 GHz (rho=0.056). -- **Humidity: 20% → 18%** — Still the strongest signal at 24+ GHz (dewpoint rho=-0.371 at 24 GHz, -0.703 at 75 GHz) but mixed/weak at 10 GHz (rho=-0.059). -- **Td Depression: 12% → 10%** — Partially redundant with humidity and PWAT; retains value as a distinct atmospheric stability indicator. -- **Refractivity: 10% → 8%** — Weaker than expected (rho=-0.034 to -0.139). HRRR vertical resolution too coarse for thin duct detection. Still useful for strong events. -- **Sky Cover: 10% → 8%** — No direct measurement in this analysis; retained at reduced weight based on physical rationale (radiative cooling). -- **Season: 10% → 8%** — Aug/Sep bias in dataset limits validation. Physics-based seasonal curves remain valid but cannot be tuned from this data. -- **Rain: 8% → 8%** — Unchanged. ITU-R P.838-3 validated, critical for high-frequency paths. -- **Wind: 6% → 5%** — Minimal impact confirmed (Finding 2, Part 2). +- **Rain: 8% → 13.6%** — Largest increase. Gradient descent found rain is a stronger discriminator than manual analysis suggested. +- **Season: 8% → 11.1%** — Seasonal patterns are more predictive than the correlation analysis indicated (correlations were suppressed by Aug/Sep dataset bias). +- **Refractivity: 8% → 10.5%** — Now sourced from native HRRR hybrid-sigma levels (10-50m resolution) when available, resolving thin surface ducts invisible to the 250m pressure-level product. Higher weight justified by higher-quality input data. +- **PWAT: 10% → 11.3%** — Confirmed as strong independent predictor. +- **Pressure: 15% → 10.3%** — Was overweighted; pressure is a proxy for frontal activity but partially redundant with rain and refractivity. +- **Humidity: 18% → 12.4%** — Partially captured by PWAT and Td depression. +- **Time of Day: 10% → 5.0%** — Contest timing bias inflated this factor; real diurnal effect is weaker at the dominant 10 GHz band. +- **Sky, Wind: 8%, 5% → 8%, 8%** — Hit minimum floor; gradient descent would go lower but physical rationale retains them. -| Factor | Old Weight | New Weight | Rationale | -|--------|-----------|-----------|-----------| -| Humidity | 20% | 18% | Strong at 24+ GHz (dewpoint rho=-0.37 to -0.70); mixed signal at 10 GHz | -| Pressure | 4% | **15%** | #1 correlator at 10 GHz (rho=-0.18); low P = longer paths confirmed | -| Time of Day | 20% | 10% | rho=0.007 at 10 GHz; modest even at 24 GHz | -| Td Depression | 12% | 10% | Partially redundant with humidity/PWAT | -| PWAT | — | **10%** | NEW: strong independent predictor all bands (rho=-0.04 to -0.61) | -| Refractivity | 10% | 8% | Weak correlation; HRRR too coarse for thin ducts | -| Sky Cover | 10% | 8% | No direct measurement in analysis; physics still valid | -| Season | 10% | 8% | Aug/Sep bias limits validation; physics-based curves retained | -| Rain | 8% | 8% | ITU-R validated, critical for high freq | -| Wind | 6% | 5% | Minimal impact confirmed | +| Factor | Weight | Source | +|--------|--------|--------| +| Rain | 13.6% | ITU-R P.838-3 specific attenuation | +| Humidity | 12.4% | Absolute humidity from surface T/Td | +| PWAT | 11.3% | HRRR precipitable water (column-integrated) | +| Season | 11.1% | Per-band monthly lookup tables | +| Refractivity | 10.5% | Native HRRR dM/dh (10-50m), fallback to pressure-level (250m) | +| Pressure | 10.3% | Surface pressure (frontal activity proxy) | +| Td Depression | 9.8% | Surface T minus Td (stability indicator) | +| Sky Cover | 8.0% | HRRR total cloud cover | +| Wind | 8.0% | HRRR 10m wind speed | +| Time of Day | 5.0% | Solar-time adjusted diurnal cycle | ```elixir # Recalibrated 2026-04-11 via gradient descent on 5000 QSOs (loss 0.42 → 0.12) @@ -1689,12 +1687,20 @@ Surface Observations (ASOS, every 5-20 min) -> 6-hour prediction timeline HRRR Model (hourly, per grid point) - -> refractivity profile, dN/dh gradient, ducts, BL depth, PWAT, surface N - -> refractivity score component (8% weight) - -> PWAT score component (10% weight — strong independent predictor, rho=-0.04 to -0.61) - -> 4,522 profiles in DB: 79% Enhanced, 15% Super, 5% Standard, 0.7% Ducting + Standard (surface + 13 pressure levels, ~25 MB/hour): + -> surface T/Td/P, HPBL, PWAT, wind, cloud, precip + -> pressure-level T/Td/HGT for refractivity profile (~250m spacing) + -> fallback refractivity gradient if native data unavailable + + Native hybrid-sigma (50 levels, ~300 MB/hour): + -> TMP, SPFH, HGT, PRES on 50 levels (10-50m near-surface spacing) + -> cell-by-cell M-profile, duct detection, trapped frequency + -> native_min_gradient replaces pressure-level gradient in scorer + -> best_duct_freq_ghz, max_duct_thickness_m, duct_count as metadata + + -> refractivity score (10.5% weight, native resolution when available) + -> PWAT score (11.3% weight) -> Key thresholds: gradient < -300 = moderate ducting, < -500 = strong ducting - -> HPBL < 200m = strongest signal for enhanced propagation Sounding Data (RAOB 00Z/12Z) -> 3,901 soundings from 112 stations @@ -1777,6 +1783,22 @@ Native profiles feed into three analysis modules: 2. **Inversion analysis** (`Propagation.Inversion`) — temperature inversion top, Bulk Richardson number, theta-e jump, wind shear 3. **Backtest features** — native_surface_refractivity, bulk_richardson, theta_e_jump, shear_at_top, duct_thickness, best_duct_freq +### Hourly Grid Integration + +The `PropagationGridWorker` fetches native duct metrics for every CONUS grid point alongside the standard surface and pressure products. For each forecast hour (f00-f18): + +1. **Download**: TMP, SPFH, HGT, PRES on all 50 hybrid levels (~300 MB of byte ranges per hour via `duct_byte_ranges/1`) +2. **Extract**: wgrib2 `-lola` interpolates to the CONUS 0.125° grid (~95k cells) +3. **Reduce**: Cell-by-cell reducer (`compute_duct_metrics/1`) computes M-profile, detects ducts, and collapses each cell to 4 scalars — peak memory ~86 MB instead of ~1.8 GB for the full grid map +4. **Merge**: Native duct metrics are merged into the standard HRRR grid profile before scoring +5. **Score**: The refractivity factor uses the native gradient (10-50m resolution) when available, falling back to the pressure-level gradient (~250m resolution) + +Per-cell output: `native_min_gradient` (dM/dh minimum from native levels), `best_duct_freq_ghz` (minimum trapped frequency), `max_duct_thickness_m`, `duct_count`. + +If the native fetch fails for any reason (data not yet available, network error), scoring continues with pressure-level data only — the native enhancement is purely additive. + +**Download cost**: ~300 MB/hour × 19 forecast hours = ~5.7 GB per hourly run (vs ~475 MB for surface + pressure only). Managed via sequential byte-range streaming to disk. + --- ## Part 13: Duct Analysis diff --git a/lib/microwaveprop/propagation.ex b/lib/microwaveprop/propagation.ex index f56964b2..aea545f4 100644 --- a/lib/microwaveprop/propagation.ex +++ b/lib/microwaveprop/propagation.ex @@ -91,9 +91,19 @@ defmodule Microwaveprop.Propagation do pwat_mm: hrrr_profile[:pwat_mm] } + duct_info = + if hrrr_profile[:duct_count] && hrrr_profile[:duct_count] > 0 do + %{ + duct_count: hrrr_profile[:duct_count], + best_duct_freq_ghz: hrrr_profile[:best_duct_freq_ghz], + max_duct_thickness_m: hrrr_profile[:max_duct_thickness_m] + } + end + Enum.map(BandConfig.all_bands(), fn band_config -> result = Scorer.composite_score(conditions, band_config) - Map.put(result, :band_mhz, band_config.freq_mhz) + result = Map.put(result, :band_mhz, band_config.freq_mhz) + if duct_info, do: put_in(result, [:factors, :duct_info], duct_info), else: result end) end