Draft upper-air factors section in algo.md pending backfill
With the native hybrid-sigma profile now storing levels all the way up to ~19 km, the scorer can finally see the mid-to-upper troposphere. The five factors that would plausibly add signal — 500 mb dewpoint depression, 300 mb wind speed, 850→500 mb dθ/dz, tropopause height, and 500 mb height anomaly — get rationale, derivation notes, and placeholder weights. Actual weights have to wait for the native backfill to catch up enough QSO hours for a representative training sample, at which point the existing gradient-descent recalibration pass refits the full factor set.
This commit is contained in:
parent
f57ea7b0c9
commit
572c2851b3
1 changed files with 63 additions and 1 deletions
64
algo.md
64
algo.md
|
|
@ -1302,13 +1302,75 @@ def score_pwat(pwat_mm, band_config) do
|
|||
end
|
||||
```
|
||||
|
||||
### Upper-Air Factors (Pending Native-Profile Backfill)
|
||||
|
||||
The 10 factors above are all surface or column-integrated quantities. None of them see the mid-to-upper troposphere, because the legacy HRRR ingestion capped at 700 mb (~3 km). With the native hybrid-sigma profile (Part 12) now storing all 50 levels up to ~19 km, the scorer can consume synoptic-scale signals that discriminate ridge-vs-trough regimes — the single strongest predictor of tropo propagation at microwave frequencies.
|
||||
|
||||
**Status**: feature plumbing is in place (`hrrr_native_profiles` schema already stores the full-atmosphere arrays). Calibration is blocked on the backfill finishing — the top-N hours by contact count must be ingested before gradient descent can assign weights. Once backfill completes, re-run the calibration pipeline (`scripts/recalibrate_algo.py`) with these features included.
|
||||
|
||||
The five proposed factors below are derived from the native profile at the QSO's path midpoint, not the endpoints, since the synoptic pattern is spatially smooth over a 300 km path.
|
||||
|
||||
**1. 500 mb dewpoint depression — mid-level dryness**
|
||||
|
||||
Dry air at 500 mb above moist lower levels is the textbook signature of synoptic subsidence: a ridge aloft pushes dry stratospheric air down, warming the mid-troposphere and capping the boundary layer. High depression (> 30 °C) correlates with anticyclonic regimes and subsidence-driven trapping; small depression (< 5 °C) indicates deep convective moisture, mixing, and poor tropo.
|
||||
|
||||
Derived from native profile: interpolate T and Td to 500 mb, return `T500_c - Td500_c`. Td comes from SPFH via the Magnus inverse (same path as `HrrrNativeProfile.to_skew_t_profile/1`).
|
||||
|
||||
**2. 300 mb wind speed — jet-level flow**
|
||||
|
||||
The 300 mb wind is the standard proxy for jet-stream position and intensity. Strong jet (> 50 m/s) means active storm track, frontal passage, vertical wind shear, and convective mixing — all bad for tropo. Weak jet (< 15 m/s) indicates zonal/blocked flow aloft, which is a necessary (not sufficient) condition for stable ducting patterns to persist beyond a single diurnal cycle.
|
||||
|
||||
Derived from native profile: interpolate `sqrt(u² + v²)` to 300 mb.
|
||||
|
||||
**3. 850→500 mb potential-temperature gradient — deep subsidence metric**
|
||||
|
||||
The mid-troposphere lapse rate, converted to potential temperature so it's mixing-invariant:
|
||||
|
||||
```
|
||||
dθ/dz = (θ_500 - θ_850) / (z_500 - z_850)
|
||||
```
|
||||
|
||||
where `θ = T * (1000/p)^0.2854`. A strongly positive gradient (> 4 K/km) means the column is stably stratified through a deep layer — subsidence is warming the mid-troposphere faster than the surface cools, creating the deep capping inversion that supports elevated ducts and keeps the boundary-layer moisture trapped. Near-zero or negative means the column is mixing through its full depth (cumulus convection, post-frontal), which destroys tropo.
|
||||
|
||||
This factor is expected to correlate more strongly with long-path 10 GHz distances than any existing factor except refractivity gradient itself, because it captures the synoptic *reason* the gradient is there.
|
||||
|
||||
**4. Tropopause height — airmass proxy**
|
||||
|
||||
A high tropopause (> 13 km) means warm, deep troposphere — subtropical airmass under a ridge, the classic beyond-LOS regime. A low tropopause (< 10 km) means cold polar airmass, active frontal zone, dynamic weather. This is a slowly varying but very clean indicator of the regime at the timescale of a contact.
|
||||
|
||||
Derived from native profile: walk levels upward from the surface, find the first level where `dT/dz` transitions from negative (troposphere) to ≥ -2 K/km sustained for > 2 km (WMO definition). Height of that transition is the tropopause.
|
||||
|
||||
**5. 500 mb geopotential-height anomaly — synoptic regime indicator**
|
||||
|
||||
The single best synoptic-scale discriminator between ridging (beneficial) and troughing (harmful). Requires a climatology baseline — monthly/daily 500 mb height normals per grid point. Ridge anomaly (> +60 m above climo) is beyond-LOS-favorable; trough (< -60 m) is harmful.
|
||||
|
||||
**Dependency**: 500 mb climatology must be computed from the same native backfill, as a by-product (or from ERA5 — cheaper and already at `era5_profiles`). This is the one upper-air factor that needs infrastructure beyond the native profile schema. Hold on implementing until the climatology path is clear.
|
||||
|
||||
### Weight placeholders
|
||||
|
||||
```
|
||||
| Factor | Weight | Source |
|
||||
| 500 mb dewpoint depression | TBD | Native HRRR profile, interpolated to 500 mb |
|
||||
| 300 mb wind speed | TBD | Native HRRR profile, interpolated to 300 mb |
|
||||
| 850-500 mb dθ/dz (subsidence) | TBD | Native HRRR profile, potential-temp gradient |
|
||||
| Tropopause height | TBD | Native HRRR profile, WMO lapse-rate definition |
|
||||
| 500 mb height anomaly | TBD | Native HRRR or ERA5, climatology-baseline (dependency) |
|
||||
```
|
||||
|
||||
Calibration plan once backfill is complete:
|
||||
|
||||
1. Extract the five features for every QSO with a matching native profile (~20-40k expected coverage).
|
||||
2. Compute per-band correlations with both `distance_km` and `composite_score` residuals.
|
||||
3. Feed features into the existing gradient-descent recalibration alongside the current 10 factors; current weights are re-fit simultaneously to avoid overclaiming the upper-air contribution.
|
||||
4. Redistribute weight out of whichever current factors are partially redundant with upper-air signals (early suspects: pressure, season, refractivity — all indirect proxies for the same synoptic regime).
|
||||
|
||||
---
|
||||
|
||||
## Part 5: Composite Score
|
||||
|
||||
### Weights
|
||||
|
||||
Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12, 72% improvement). Key changes from the April 2026 manual calibration:
|
||||
Recalibrated 2026-04-11 via gradient descent on 5,000 QSOs (loss 0.42 → 0.12, 72% improvement). Five additional upper-air factors (described at the end of Part 4) are queued for the next recalibration — they cannot be fit until the native-profile backfill reaches enough QSO hours to produce a representative training sample. Key changes from the April 2026 manual calibration:
|
||||
|
||||
- **Rain: 8% → 13.6%** — Largest increase. Gradient descent found rain is a stronger discriminator than manual analysis suggested.
|
||||
- **Season: 8% → 11.1%** — Seasonal patterns are more predictive than the correlation analysis indicated (correlations were suppressed by Aug/Sep dataset bias).
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue