Graham McIntire
900685aa06
- Spike docs at docs/research/hrrr_native_levels.md confirming files
are on AWS S3 for 5+ years, 50 hybrid levels, and include TKE and
SPFH needed for Phase 2 turbulence features. Architectural finding:
per-point on-demand fetching is impractical (~530 MB/file), so
the ingestion worker batches per (date, hour) instead.
- hrrr_native_profiles schema: arrays per level plus cached surface
scalars and placeholder columns for Phase 2/4 derived fields.
Strictly additive — the existing hrrr_profiles table is untouched.
- HrrrNativeClient: pure URL/message-list helpers, build_native_profile/1
that turns a parsed wgrib2 map into the schema shape (TDD'd).
- Exposed HrrrClient.download_grib_ranges/2 so the native client
reuses the existing parallel byte-range download + disk cache.
- HrrrNativeGridWorker: Oban worker keyed on {year, month, day, hour},
unique at :infinity, pulls distinct (lat, lon) points from contacts
in the ±30 min window, downloads the native grib2, extracts per
point, bulk-upserts.
- mix hrrr_native_backfill --limit N enqueues the top-N hours by
contact count.
Phase 1 gate still pending Task 1.6 (sanity-check backtest after
live data lands).
131 lines
7.2 KiB
Markdown
131 lines
7.2 KiB
Markdown
# HRRR Hybrid-Sigma ("Native") Levels — Spike Findings
|
||
|
||
**Date:** 2026-04-09
|
||
**Purpose:** Task 1.1 of `docs/plans/2026-04-09-propagation-modeling-improvements.md`
|
||
**Question:** Can we practically ingest HRRR native hybrid-sigma profiles to replace the 25 hPa pressure-level product for boundary-layer propagation features?
|
||
|
||
## TL;DR
|
||
|
||
**Yes — with one architecture change.** The files are available on AWS S3 going back at least 5 years and expose all the variables needed for Richardson number, θₑ, shear, and TKE computations on 50 hybrid-sigma levels. But the per-file download is too large to run on-demand per QSO point. Phase 1 should ingest native profiles in batch mode per HRRR run hour, not per QSO.
|
||
|
||
## File location
|
||
|
||
Same S3 bucket as the existing surface/pressure products:
|
||
|
||
```
|
||
https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.<YYYYMMDD>/conus/hrrr.t<HH>z.wrfnatf00.grib2
|
||
https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.<YYYYMMDD>/conus/hrrr.t<HH>z.wrfnatf00.grib2.idx
|
||
```
|
||
|
||
`wrfnatf00` = native levels, 0-hour forecast. Forecast hours `wrfnatf01` through `wrfnatf18` also exist.
|
||
|
||
## Historical availability
|
||
|
||
Spot-checked on 2026-04-09 by HEAD-ing the `.idx` URL:
|
||
|
||
| Date | HTTP status |
|
||
|---|---|
|
||
| 2026-03-09 (30d) | 200 |
|
||
| 2026-02-09 (60d) | 200 |
|
||
| 2026-01-09 (~90d) | 200 |
|
||
| 2025-12-09 (~120d) | 200 |
|
||
| 2024-04-09 (~1y) | 200 |
|
||
| 2023-04-09 (~2y) | 200 |
|
||
| 2022-04-09 (~3y) | 200 |
|
||
| 2021-04-09 (~5y) | 200 |
|
||
|
||
No rotation problem. We can backtest against years of historical native data if we want to, which unblocks Phase 6 (climatology) as well.
|
||
|
||
## What the file contains
|
||
|
||
The `.idx` file for one HRRR run has **1133 GRIB2 messages**. **1000 of them are on hybrid levels 1-50**, plus 133 surface/2m/10m/composite entries.
|
||
|
||
Each of the 50 hybrid levels carries these 20 variables:
|
||
|
||
```
|
||
PRES pressure (Pa)
|
||
HGT geopotential height (m)
|
||
TMP temperature (K)
|
||
SPFH specific humidity (kg/kg) ← derive Td, θ, θₑ from SPFH + TMP + PRES
|
||
UGRD u wind (m/s)
|
||
VGRD v wind (m/s)
|
||
VVEL vertical velocity (Pa/s)
|
||
TKE turbulent kinetic energy (m²/s²) ← **exactly the turbulence signal Phase 2 needs**
|
||
CLMR cloud water mixing ratio
|
||
CIMIXR cloud ice mixing ratio
|
||
RWMR rain water mixing ratio
|
||
SNMR snow mixing ratio
|
||
GRLE graupel mixing ratio
|
||
NCONCD number concentration of cloud droplets
|
||
NCCICE number concentration of cloud ice
|
||
SPNCR number concentration of rain
|
||
PMTF ???
|
||
PMTC ???
|
||
FRACCC convective cloud fraction
|
||
MASSDEN particulate mass density
|
||
```
|
||
|
||
The seven we care about for Phase 2–4 are **TMP, SPFH, HGT, UGRD, VGRD, TKE, PRES** × 50 levels = 350 messages out of the 1133.
|
||
|
||
Note: **there is no DPT on hybrid levels.** The pressure-level product has dewpoint directly; the native product only has specific humidity. We will compute `Td` from `(T, SPFH, P)` using the Bolton (1980) or Magnus-Tetens formulation. This is standard and cheap.
|
||
|
||
## Vertical spacing
|
||
|
||
Hybrid levels are defined by the HRRR vertical coordinate function; level 1 is closest to the surface. The actual geometric height of each level varies by grid point (because hybrid coordinates conform to terrain). Typical spacing for a plains grid point:
|
||
|
||
- Level 1: ~8 m AGL
|
||
- Level 2: ~25 m AGL
|
||
- Level 3: ~50 m AGL
|
||
- Level 5: ~130 m AGL
|
||
- Level 10: ~460 m AGL
|
||
- Level 20: ~1.7 km AGL
|
||
- Level 30: ~3.8 km AGL
|
||
- Level 50: ~19 km AGL
|
||
|
||
**The first ~5 hybrid levels alone span the range where most surface-based ducts live.** That's the whole point: 25 hPa pressure levels (~250 m spacing at the surface) completely miss anything thinner than ~200 m, and the native product gives us 30-50 m resolution in the same band.
|
||
|
||
(Actual per-level heights must be read from the HGT message for each grid point, not from a table. We store `heights_m` as `{:array, :float}` on `hrrr_native_profiles` for exactly this reason.)
|
||
|
||
## Download size — the architectural gotcha
|
||
|
||
Downloaded one full `wrfnatf00.grib2.idx` and walked the offsets for the seven essential variables on all 50 levels:
|
||
|
||
- 350 GRIB2 messages
|
||
- Total byte span of the essential messages: **~530 MB**
|
||
- Full native file: ~566 MB (so essentials are almost the whole file)
|
||
|
||
That's **per HRRR run hour**, for the full CONUS grid.
|
||
|
||
This matters because the plan's draft of Task 1.4 has a per-QSO `HrrrNativeFetchWorker`. 530 MB × 58k QSOs = 29 TB of re-downloads, which is absurd. **Byte-range requests don't save us** — GRIB2 messages can only be decoded whole, so we'd have to pull ~350 × 1.5 MB per point, and AWS doesn't support server-side subsetting on this bucket.
|
||
|
||
### What to do instead
|
||
|
||
Two workable architectures:
|
||
|
||
1. **Batch per run hour.** A single worker downloads one full `wrfnatf00.grib2` (~566 MB), extracts native profiles at *all* points of interest for that hour in one pass (the grid of contacts + whatever points the live scorer wants), and bulk-upserts them into `hrrr_native_profiles`. This is how the existing `PropagationGridWorker` pipeline works for the pressure-level file; we mirror it for native levels.
|
||
|
||
2. **Per-day local cache.** Download one file per `(date, hour)`, stash it on local disk for the duration of a backfill job, extract all points of interest for that hour, delete the file. This is probably only worth the bandwidth for dense backfills and can be folded into option 1.
|
||
|
||
**Plan correction:** the `HrrrNativeFetchWorker` in Task 1.4 should be renamed `HrrrNativeGridWorker` and take `{date, hour}` as its unique key, not `{lat, lon, valid_time}`. The backfill mix task in Task 1.5 iterates over historical (date, hour) pairs where we have contacts and enqueues one grid worker per hour.
|
||
|
||
## Bandwidth budget
|
||
|
||
If we backfill 1 year of hourly native grids for every hour where we have at least one QSO, and we have ~58k QSOs spread over 5+ years, conservatively ~15k distinct (date, hour) hours. 15k × 566 MB ≈ **8.5 TB**. That's a lot. We don't need it all — Phase 2 only needs enough data to backtest. Realistic first backfill:
|
||
|
||
- Top 500 distinct hours by QSO count → ~280 GB (one-time, finishes overnight)
|
||
- Extract only the lat/lon points where we have QSOs in that hour plus the propagation grid points
|
||
- Store derived profiles only (< 5 KB per profile), not the raw GRIB2
|
||
|
||
Storage for derived profiles: 500 hours × ~300 points/hour × 5 KB ≈ 750 MB in the DB. Fine.
|
||
|
||
## Proposed Phase 1 plan adjustments
|
||
|
||
1. Rename worker: `HrrrNativeGridWorker` instead of `HrrrNativeFetchWorker`; unique keys `{date, hour}`; downloads the full file once per run hour and bulk-processes points.
|
||
2. Keep `hrrr_native_profiles` schema as planned (per-point, per-valid_time), just populated in batch.
|
||
3. Task 1.5 backfill mix task now enqueues grid workers for the top-N (date, hour) hours by QSO density.
|
||
4. Feature functions in later phases look up `hrrr_native_profiles` the same way the existing scorer looks up `hrrr_profiles` — no API change on the consuming side.
|
||
5. Computing Td from SPFH is a small Phase 2 helper; budget ~half a day for it.
|
||
|
||
## Gate decision
|
||
|
||
**Gate pass.** Files are available historically, all required variables are present, vertical resolution is 5-30× finer than the pressure-level product near the surface, and the ingestion architecture is a known pattern (matches the existing grid worker). Proceed to Task 1.2 (schema) with the batch-mode correction.
|