Draft upper-air factors section in algo.md pending backfill

With the native hybrid-sigma profile now storing levels all the way up
to ~19 km, the scorer can finally see the mid-to-upper troposphere.
The five factors that would plausibly add signal — 500 mb dewpoint
depression, 300 mb wind speed, 850→500 mb dθ/dz, tropopause height,
and 500 mb height anomaly — get rationale, derivation notes, and
placeholder weights. Actual weights have to wait for the native
backfill to catch up enough QSO hours for a representative training
sample, at which point the existing gradient-descent recalibration
pass refits the full factor set.
This commit is contained in:
Graham McIntire 2026-04-14 17:30:31 -05:00
parent f57ea7b0c9
commit 572c2851b3
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59

64
algo.md
View file

@ -1302,13 +1302,75 @@ def score_pwat(pwat_mm, band_config) do
end
```
### Upper-Air Factors (Pending Native-Profile Backfill)
The 10 factors above are all surface or column-integrated quantities. None of them see the mid-to-upper troposphere, because the legacy HRRR ingestion capped at 700 mb (~3 km). With the native hybrid-sigma profile (Part 12) now storing all 50 levels up to ~19 km, the scorer can consume synoptic-scale signals that discriminate ridge-vs-trough regimes — the single strongest predictor of tropo propagation at microwave frequencies.
**Status**: feature plumbing is in place (`hrrr_native_profiles` schema already stores the full-atmosphere arrays). Calibration is blocked on the backfill finishing — the top-N hours by contact count must be ingested before gradient descent can assign weights. Once backfill completes, re-run the calibration pipeline (`scripts/recalibrate_algo.py`) with these features included.
The five proposed factors below are derived from the native profile at the QSO's path midpoint, not the endpoints, since the synoptic pattern is spatially smooth over a 300 km path.
**1. 500 mb dewpoint depression — mid-level dryness**
Dry air at 500 mb above moist lower levels is the textbook signature of synoptic subsidence: a ridge aloft pushes dry stratospheric air down, warming the mid-troposphere and capping the boundary layer. High depression (> 30 °C) correlates with anticyclonic regimes and subsidence-driven trapping; small depression (< 5 °C) indicates deep convective moisture, mixing, and poor tropo.
Derived from native profile: interpolate T and Td to 500 mb, return `T500_c - Td500_c`. Td comes from SPFH via the Magnus inverse (same path as `HrrrNativeProfile.to_skew_t_profile/1`).
**2. 300 mb wind speed — jet-level flow**
The 300 mb wind is the standard proxy for jet-stream position and intensity. Strong jet (> 50 m/s) means active storm track, frontal passage, vertical wind shear, and convective mixing — all bad for tropo. Weak jet (< 15 m/s) indicates zonal/blocked flow aloft, which is a necessary (not sufficient) condition for stable ducting patterns to persist beyond a single diurnal cycle.
Derived from native profile: interpolate `sqrt(u² + v²)` to 300 mb.
**3. 850→500 mb potential-temperature gradient — deep subsidence metric**
The mid-troposphere lapse rate, converted to potential temperature so it's mixing-invariant:
```
dθ/dz = (θ_500 - θ_850) / (z_500 - z_850)
```
where `θ = T * (1000/p)^0.2854`. A strongly positive gradient (> 4 K/km) means the column is stably stratified through a deep layer — subsidence is warming the mid-troposphere faster than the surface cools, creating the deep capping inversion that supports elevated ducts and keeps the boundary-layer moisture trapped. Near-zero or negative means the column is mixing through its full depth (cumulus convection, post-frontal), which destroys tropo.
This factor is expected to correlate more strongly with long-path 10 GHz distances than any existing factor except refractivity gradient itself, because it captures the synoptic *reason* the gradient is there.
**4. Tropopause height — airmass proxy**
A high tropopause (> 13 km) means warm, deep troposphere — subtropical airmass under a ridge, the classic beyond-LOS regime. A low tropopause (< 10 km) means cold polar airmass, active frontal zone, dynamic weather. This is a slowly varying but very clean indicator of the regime at the timescale of a contact.
Derived from native profile: walk levels upward from the surface, find the first level where `dT/dz` transitions from negative (troposphere) to ≥ -2 K/km sustained for > 2 km (WMO definition). Height of that transition is the tropopause.
**5. 500 mb geopotential-height anomaly — synoptic regime indicator**
The single best synoptic-scale discriminator between ridging (beneficial) and troughing (harmful). Requires a climatology baseline — monthly/daily 500 mb height normals per grid point. Ridge anomaly (> +60 m above climo) is beyond-LOS-favorable; trough (< -60 m) is harmful.
**Dependency**: 500 mb climatology must be computed from the same native backfill, as a by-product (or from ERA5 — cheaper and already at `era5_profiles`). This is the one upper-air factor that needs infrastructure beyond the native profile schema. Hold on implementing until the climatology path is clear.
### Weight placeholders
```
| Factor | Weight | Source |
| 500 mb dewpoint depression | TBD | Native HRRR profile, interpolated to 500 mb |
| 300 mb wind speed | TBD | Native HRRR profile, interpolated to 300 mb |
| 850-500 mb dθ/dz (subsidence) | TBD | Native HRRR profile, potential-temp gradient |
| Tropopause height | TBD | Native HRRR profile, WMO lapse-rate definition |
| 500 mb height anomaly | TBD | Native HRRR or ERA5, climatology-baseline (dependency) |
```
Calibration plan once backfill is complete:
1. Extract the five features for every QSO with a matching native profile (~20-40k expected coverage).
2. Compute per-band correlations with both `distance_km` and `composite_score` residuals.
3. Feed features into the existing gradient-descent recalibration alongside the current 10 factors; current weights are re-fit simultaneously to avoid overclaiming the upper-air contribution.
4. Redistribute weight out of whichever current factors are partially redundant with upper-air signals (early suspects: pressure, season, refractivity — all indirect proxies for the same synoptic regime).
---
## Part 5: Composite Score
### Weights
Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12, 72% improvement). Key changes from the April 2026 manual calibration:
Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12, 72% improvement). Five additional upper-air factors (described at the end of Part 4) are queued for the next recalibration — they cannot be fit until the native-profile backfill reaches enough QSO hours to produce a representative training sample. Key changes from the April 2026 manual calibration:
- **Rain: 8% → 13.6%** — Largest increase. Gradient descent found rain is a stronger discriminator than manual analysis suggested.
- **Season: 8% → 11.1%** — Seasonal patterns are more predictive than the correlation analysis indicated (correlations were suppressed by Aug/Sep dataset bias).