Update algo.md weights and data flow for native duct integration

- Replace stale April 2026 manual weights with recalibrated values
- Document native hybrid-sigma data in data flow section
- Note refractivity factor now uses native 10-50m resolution
- Add hourly grid integration section to Part 12
- Expose duct_info (count, freq, thickness) in scoring factors for UI

algo.md

@@ -1126,31 +1126,29 @@ end
 
 ### Weights
 
-Revised April 2026 based on 57,248 QSO-HRRR correlation analysis. Key changes from previous weights:
+Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12, 72% improvement). Key changes from the April 2026 manual calibration:
 
-- **Pressure: 4% → 15%** — #1 correlator at 10 GHz (rho=-0.180); data shows low P (<1005 mb) gives 197 km median vs 103 km for >1020 mb. Previously scored backwards (high=good). Now the largest single-factor weight increase in algorithm history.
-- **PWAT: 0% → 10%** — New factor. Strong independent predictor across all bands (rho=-0.039 to -0.608). Captures column-integrated moisture not represented by surface humidity or Td depression.
-- **Time of Day: 20% → 10%** — rho=0.007 at 10 GHz; effect is real at 24+ GHz (rho=-0.392 at 75 GHz) but the 20% weight was unwarranted at the dominant 10 GHz band. Modest even at 24 GHz (rho=0.056).
-- **Humidity: 20% → 18%** — Still the strongest signal at 24+ GHz (dewpoint rho=-0.371 at 24 GHz, -0.703 at 75 GHz) but mixed/weak at 10 GHz (rho=-0.059).
-- **Td Depression: 12% → 10%** — Partially redundant with humidity and PWAT; retains value as a distinct atmospheric stability indicator.
-- **Refractivity: 10% → 8%** — Weaker than expected (rho=-0.034 to -0.139). HRRR vertical resolution too coarse for thin duct detection. Still useful for strong events.
-- **Sky Cover: 10% → 8%** — No direct measurement in this analysis; retained at reduced weight based on physical rationale (radiative cooling).
-- **Season: 10% → 8%** — Aug/Sep bias in dataset limits validation. Physics-based seasonal curves remain valid but cannot be tuned from this data.
-- **Rain: 8% → 8%** — Unchanged. ITU-R P.838-3 validated, critical for high-frequency paths.
-- **Wind: 6% → 5%** — Minimal impact confirmed (Finding 2, Part 2).
+- **Rain: 8% → 13.6%** — Largest increase. Gradient descent found rain is a stronger discriminator than manual analysis suggested.
+- **Season: 8% → 11.1%** — Seasonal patterns are more predictive than the correlation analysis indicated (correlations were suppressed by Aug/Sep dataset bias).
+- **Refractivity: 8% → 10.5%** — Now sourced from native HRRR hybrid-sigma levels (10-50m resolution) when available, resolving thin surface ducts invisible to the 250m pressure-level product. Higher weight justified by higher-quality input data.
+- **PWAT: 10% → 11.3%** — Confirmed as strong independent predictor.
+- **Pressure: 15% → 10.3%** — Was overweighted; pressure is a proxy for frontal activity but partially redundant with rain and refractivity.
+- **Humidity: 18% → 12.4%** — Partially captured by PWAT and Td depression.
+- **Time of Day: 10% → 5.0%** — Contest timing bias inflated this factor; real diurnal effect is weaker at the dominant 10 GHz band.
+- **Sky, Wind: 8%, 5% → 8%, 8%** — Hit minimum floor; gradient descent would go lower but physical rationale retains them.
 
-| Factor | Old Weight | New Weight | Rationale |
-|--------|-----------|-----------|-----------|
-| Humidity | 20% | 18% | Strong at 24+ GHz (dewpoint rho=-0.37 to -0.70); mixed signal at 10 GHz |
-| Pressure | 4% | **15%** | #1 correlator at 10 GHz (rho=-0.18); low P = longer paths confirmed |
-| Time of Day | 20% | 10% | rho=0.007 at 10 GHz; modest even at 24 GHz |
-| Td Depression | 12% | 10% | Partially redundant with humidity/PWAT |
-| PWAT | — | **10%** | NEW: strong independent predictor all bands (rho=-0.04 to -0.61) |
-| Refractivity | 10% | 8% | Weak correlation; HRRR too coarse for thin ducts |
-| Sky Cover | 10% | 8% | No direct measurement in analysis; physics still valid |
-| Season | 10% | 8% | Aug/Sep bias limits validation; physics-based curves retained |
-| Rain | 8% | 8% | ITU-R validated, critical for high freq |
-| Wind | 6% | 5% | Minimal impact confirmed |
+| Factor | Weight | Source |
+|--------|--------|--------|
+| Rain | 13.6% | ITU-R P.838-3 specific attenuation |
+| Humidity | 12.4% | Absolute humidity from surface T/Td |
+| PWAT | 11.3% | HRRR precipitable water (column-integrated) |
+| Season | 11.1% | Per-band monthly lookup tables |
+| Refractivity | 10.5% | Native HRRR dM/dh (10-50m), fallback to pressure-level (250m) |
+| Pressure | 10.3% | Surface pressure (frontal activity proxy) |
+| Td Depression | 9.8% | Surface T minus Td (stability indicator) |
+| Sky Cover | 8.0% | HRRR total cloud cover |
+| Wind | 8.0% | HRRR 10m wind speed |
+| Time of Day | 5.0% | Solar-time adjusted diurnal cycle |
 
 ```elixir
 # Recalibrated 2026-04-11 via gradient descent on 5000 QSOs (loss 0.42 → 0.12)
@@ -1689,12 +1687,20 @@ Surface Observations (ASOS, every 5-20 min)
   -> 6-hour prediction timeline
 
 HRRR Model (hourly, per grid point)
-  -> refractivity profile, dN/dh gradient, ducts, BL depth, PWAT, surface N
-  -> refractivity score component (8% weight)
-  -> PWAT score component (10% weight — strong independent predictor, rho=-0.04 to -0.61)
-  -> 4,522 profiles in DB: 79% Enhanced, 15% Super, 5% Standard, 0.7% Ducting
+  Standard (surface + 13 pressure levels, ~25 MB/hour):
+    -> surface T/Td/P, HPBL, PWAT, wind, cloud, precip
+    -> pressure-level T/Td/HGT for refractivity profile (~250m spacing)
+    -> fallback refractivity gradient if native data unavailable
+  
+  Native hybrid-sigma (50 levels, ~300 MB/hour):
+    -> TMP, SPFH, HGT, PRES on 50 levels (10-50m near-surface spacing)
+    -> cell-by-cell M-profile, duct detection, trapped frequency
+    -> native_min_gradient replaces pressure-level gradient in scorer
+    -> best_duct_freq_ghz, max_duct_thickness_m, duct_count as metadata
+  
+  -> refractivity score (10.5% weight, native resolution when available)
+  -> PWAT score (11.3% weight)
   -> Key thresholds: gradient < -300 = moderate ducting, < -500 = strong ducting
-  -> HPBL < 200m = strongest signal for enhanced propagation
 
 Sounding Data (RAOB 00Z/12Z)
   -> 3,901 soundings from 112 stations
@@ -1777,6 +1783,22 @@ Native profiles feed into three analysis modules:
 2. **Inversion analysis** (`Propagation.Inversion`) — temperature inversion top, Bulk Richardson number, theta-e jump, wind shear
 3. **Backtest features** — native_surface_refractivity, bulk_richardson, theta_e_jump, shear_at_top, duct_thickness, best_duct_freq
 
+### Hourly Grid Integration
+
+The `PropagationGridWorker` fetches native duct metrics for every CONUS grid point alongside the standard surface and pressure products. For each forecast hour (f00-f18):
+
+1. **Download**: TMP, SPFH, HGT, PRES on all 50 hybrid levels (~300 MB of byte ranges per hour via `duct_byte_ranges/1`)
+2. **Extract**: wgrib2 `-lola` interpolates to the CONUS 0.125° grid (~95k cells)
+3. **Reduce**: Cell-by-cell reducer (`compute_duct_metrics/1`) computes M-profile, detects ducts, and collapses each cell to 4 scalars — peak memory ~86 MB instead of ~1.8 GB for the full grid map
+4. **Merge**: Native duct metrics are merged into the standard HRRR grid profile before scoring
+5. **Score**: The refractivity factor uses the native gradient (10-50m resolution) when available, falling back to the pressure-level gradient (~250m resolution)
+
+Per-cell output: `native_min_gradient` (dM/dh minimum from native levels), `best_duct_freq_ghz` (minimum trapped frequency), `max_duct_thickness_m`, `duct_count`.
+
+If the native fetch fails for any reason (data not yet available, network error), scoring continues with pressure-level data only — the native enhancement is purely additive.
+
+**Download cost**: ~300 MB/hour × 19 forecast hours = ~5.7 GB per hourly run (vs ~475 MB for surface + pressure only). Managed via sequential byte-range streaming to disk.
+
 ---
 
 ## Part 13: Duct Analysis

lib/microwaveprop/propagation.ex

@@ -91,9 +91,19 @@ defmodule Microwaveprop.Propagation do
       pwat_mm: hrrr_profile[:pwat_mm]
     }
 
+    duct_info =
+      if hrrr_profile[:duct_count] && hrrr_profile[:duct_count] > 0 do
+        %{
+          duct_count: hrrr_profile[:duct_count],
+          best_duct_freq_ghz: hrrr_profile[:best_duct_freq_ghz],
+          max_duct_thickness_m: hrrr_profile[:max_duct_thickness_m]
+        }
+      end
+
     Enum.map(BandConfig.all_bands(), fn band_config ->
       result = Scorer.composite_score(conditions, band_config)
-      Map.put(result, :band_mhz, band_config.freq_mhz)
+      result = Map.put(result, :band_mhz, band_config.freq_mhz)
+      if duct_info, do: put_in(result, [:factors, :duct_info], duct_info), else: result
     end)
   end