From 572c2851b30778014ceaeb102eb2837ced4f3b31 Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Tue, 14 Apr 2026 17:30:31 -0500 Subject: [PATCH] Draft upper-air factors section in algo.md pending backfill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With the native hybrid-sigma profile now storing levels all the way up to ~19 km, the scorer can finally see the mid-to-upper troposphere. The five factors that would plausibly add signal — 500 mb dewpoint depression, 300 mb wind speed, 850→500 mb dθ/dz, tropopause height, and 500 mb height anomaly — get rationale, derivation notes, and placeholder weights. Actual weights have to wait for the native backfill to catch up enough QSO hours for a representative training sample, at which point the existing gradient-descent recalibration pass refits the full factor set. --- algo.md | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-) diff --git a/algo.md b/algo.md index c7976a34..c4855665 100644 --- a/algo.md +++ b/algo.md @@ -1302,13 +1302,75 @@ def score_pwat(pwat_mm, band_config) do end ``` +### Upper-Air Factors (Pending Native-Profile Backfill) + +The 10 factors above are all surface or column-integrated quantities. None of them see the mid-to-upper troposphere, because the legacy HRRR ingestion capped at 700 mb (~3 km). With the native hybrid-sigma profile (Part 12) now storing all 50 levels up to ~19 km, the scorer can consume synoptic-scale signals that discriminate ridge-vs-trough regimes — the single strongest predictor of tropo propagation at microwave frequencies. + +**Status**: feature plumbing is in place (`hrrr_native_profiles` schema already stores the full-atmosphere arrays). Calibration is blocked on the backfill finishing — the top-N hours by contact count must be ingested before gradient descent can assign weights. Once backfill completes, re-run the calibration pipeline (`scripts/recalibrate_algo.py`) with these features included. + +The five proposed factors below are derived from the native profile at the QSO's path midpoint, not the endpoints, since the synoptic pattern is spatially smooth over a 300 km path. + +**1. 500 mb dewpoint depression — mid-level dryness** + +Dry air at 500 mb above moist lower levels is the textbook signature of synoptic subsidence: a ridge aloft pushes dry stratospheric air down, warming the mid-troposphere and capping the boundary layer. High depression (> 30 °C) correlates with anticyclonic regimes and subsidence-driven trapping; small depression (< 5 °C) indicates deep convective moisture, mixing, and poor tropo. + +Derived from native profile: interpolate T and Td to 500 mb, return `T500_c - Td500_c`. Td comes from SPFH via the Magnus inverse (same path as `HrrrNativeProfile.to_skew_t_profile/1`). + +**2. 300 mb wind speed — jet-level flow** + +The 300 mb wind is the standard proxy for jet-stream position and intensity. Strong jet (> 50 m/s) means active storm track, frontal passage, vertical wind shear, and convective mixing — all bad for tropo. Weak jet (< 15 m/s) indicates zonal/blocked flow aloft, which is a necessary (not sufficient) condition for stable ducting patterns to persist beyond a single diurnal cycle. + +Derived from native profile: interpolate `sqrt(u² + v²)` to 300 mb. + +**3. 850→500 mb potential-temperature gradient — deep subsidence metric** + +The mid-troposphere lapse rate, converted to potential temperature so it's mixing-invariant: + +``` +dθ/dz = (θ_500 - θ_850) / (z_500 - z_850) +``` + +where `θ = T * (1000/p)^0.2854`. A strongly positive gradient (> 4 K/km) means the column is stably stratified through a deep layer — subsidence is warming the mid-troposphere faster than the surface cools, creating the deep capping inversion that supports elevated ducts and keeps the boundary-layer moisture trapped. Near-zero or negative means the column is mixing through its full depth (cumulus convection, post-frontal), which destroys tropo. + +This factor is expected to correlate more strongly with long-path 10 GHz distances than any existing factor except refractivity gradient itself, because it captures the synoptic *reason* the gradient is there. + +**4. Tropopause height — airmass proxy** + +A high tropopause (> 13 km) means warm, deep troposphere — subtropical airmass under a ridge, the classic beyond-LOS regime. A low tropopause (< 10 km) means cold polar airmass, active frontal zone, dynamic weather. This is a slowly varying but very clean indicator of the regime at the timescale of a contact. + +Derived from native profile: walk levels upward from the surface, find the first level where `dT/dz` transitions from negative (troposphere) to ≥ -2 K/km sustained for > 2 km (WMO definition). Height of that transition is the tropopause. + +**5. 500 mb geopotential-height anomaly — synoptic regime indicator** + +The single best synoptic-scale discriminator between ridging (beneficial) and troughing (harmful). Requires a climatology baseline — monthly/daily 500 mb height normals per grid point. Ridge anomaly (> +60 m above climo) is beyond-LOS-favorable; trough (< -60 m) is harmful. + +**Dependency**: 500 mb climatology must be computed from the same native backfill, as a by-product (or from ERA5 — cheaper and already at `era5_profiles`). This is the one upper-air factor that needs infrastructure beyond the native profile schema. Hold on implementing until the climatology path is clear. + +### Weight placeholders + +``` +| Factor | Weight | Source | +| 500 mb dewpoint depression | TBD | Native HRRR profile, interpolated to 500 mb | +| 300 mb wind speed | TBD | Native HRRR profile, interpolated to 300 mb | +| 850-500 mb dθ/dz (subsidence) | TBD | Native HRRR profile, potential-temp gradient | +| Tropopause height | TBD | Native HRRR profile, WMO lapse-rate definition | +| 500 mb height anomaly | TBD | Native HRRR or ERA5, climatology-baseline (dependency) | +``` + +Calibration plan once backfill is complete: + +1. Extract the five features for every QSO with a matching native profile (~20-40k expected coverage). +2. Compute per-band correlations with both `distance_km` and `composite_score` residuals. +3. Feed features into the existing gradient-descent recalibration alongside the current 10 factors; current weights are re-fit simultaneously to avoid overclaiming the upper-air contribution. +4. Redistribute weight out of whichever current factors are partially redundant with upper-air signals (early suspects: pressure, season, refractivity — all indirect proxies for the same synoptic regime). + --- ## Part 5: Composite Score ### Weights -Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12, 72% improvement). Key changes from the April 2026 manual calibration: +Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12, 72% improvement). Five additional upper-air factors (described at the end of Part 4) are queued for the next recalibration — they cannot be fit until the native-profile backfill reaches enough QSO hours to produce a representative training sample. Key changes from the April 2026 manual calibration: - **Rain: 8% → 13.6%** — Largest increase. Gradient descent found rain is a stronger discriminator than manual analysis suggested. - **Season: 8% → 11.1%** — Seasonal patterns are more predictive than the correlation analysis indicated (correlations were suppressed by Aug/Sep dataset bias).