prop/docs/research/hrrr_native_levels.md

HRRR Hybrid-Sigma ("Native") Levels — Spike Findings

Date: 2026-04-09 Purpose: Task 1.1 of docs/plans/2026-04-09-propagation-modeling-improvements.md Question: Can we practically ingest HRRR native hybrid-sigma profiles to replace the 25 hPa pressure-level product for boundary-layer propagation features?

TL;DR

Yes — with one architecture change. The files are available on AWS S3 going back at least 5 years and expose all the variables needed for Richardson number, θₑ, shear, and TKE computations on 50 hybrid-sigma levels. But the per-file download is too large to run on-demand per QSO point. Phase 1 should ingest native profiles in batch mode per HRRR run hour, not per QSO.

File location

Same S3 bucket as the existing surface/pressure products:

https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.<YYYYMMDD>/conus/hrrr.t<HH>z.wrfnatf00.grib2
https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.<YYYYMMDD>/conus/hrrr.t<HH>z.wrfnatf00.grib2.idx

wrfnatf00 = native levels, 0-hour forecast. Forecast hours wrfnatf01 through wrfnatf18 also exist.

Historical availability

Spot-checked on 2026-04-09 by HEAD-ing the .idx URL:

Date	HTTP status
2026-03-09 (30d)	200
2026-02-09 (60d)	200
2026-01-09 (~90d)	200
2025-12-09 (~120d)	200
2024-04-09 (~1y)	200
2023-04-09 (~2y)	200
2022-04-09 (~3y)	200
2021-04-09 (~5y)	200

No rotation problem. We can backtest against years of historical native data if we want to, which unblocks Phase 6 (climatology) as well.

What the file contains

The .idx file for one HRRR run has 1133 GRIB2 messages. 1000 of them are on hybrid levels 1-50, plus 133 surface/2m/10m/composite entries.

Each of the 50 hybrid levels carries these 20 variables:

PRES      pressure (Pa)
HGT       geopotential height (m)
TMP       temperature (K)
SPFH      specific humidity (kg/kg)   ← derive Td, θ, θₑ from SPFH + TMP + PRES
UGRD      u wind (m/s)
VGRD      v wind (m/s)
VVEL      vertical velocity (Pa/s)
TKE       turbulent kinetic energy (m²/s²)   ← **exactly the turbulence signal Phase 2 needs**
CLMR      cloud water mixing ratio
CIMIXR    cloud ice mixing ratio
RWMR      rain water mixing ratio
SNMR      snow mixing ratio
GRLE      graupel mixing ratio
NCONCD    number concentration of cloud droplets
NCCICE    number concentration of cloud ice
SPNCR     number concentration of rain
PMTF      ???
PMTC      ???
FRACCC    convective cloud fraction
MASSDEN   particulate mass density

The seven we care about for Phase 2–4 are TMP, SPFH, HGT, UGRD, VGRD, TKE, PRES × 50 levels = 350 messages out of the 1133.

Note: there is no DPT on hybrid levels. The pressure-level product has dewpoint directly; the native product only has specific humidity. We will compute Td from (T, SPFH, P) using the Bolton (1980) or Magnus-Tetens formulation. This is standard and cheap.

Vertical spacing

Hybrid levels are defined by the HRRR vertical coordinate function; level 1 is closest to the surface. The actual geometric height of each level varies by grid point (because hybrid coordinates conform to terrain). Typical spacing for a plains grid point:

• Level 1: ~8 m AGL
• Level 2: ~25 m AGL
• Level 3: ~50 m AGL
• Level 5: ~130 m AGL
• Level 10: ~460 m AGL
• Level 20: ~1.7 km AGL
• Level 30: ~3.8 km AGL
• Level 50: ~19 km AGL

The first ~5 hybrid levels alone span the range where most surface-based ducts live. That's the whole point: 25 hPa pressure levels (~250 m spacing at the surface) completely miss anything thinner than ~200 m, and the native product gives us 30-50 m resolution in the same band.

(Actual per-level heights must be read from the HGT message for each grid point, not from a table. We store heights_m as {:array, :float} on hrrr_native_profiles for exactly this reason.)

Download size — the architectural gotcha

Downloaded one full wrfnatf00.grib2.idx and walked the offsets for the seven essential variables on all 50 levels:

• 350 GRIB2 messages
• Total byte span of the essential messages: ~530 MB
• Full native file: ~566 MB (so essentials are almost the whole file)

That's per HRRR run hour, for the full CONUS grid.

This matters because the plan's draft of Task 1.4 has a per-QSO HrrrNativeFetchWorker. 530 MB × 58k QSOs = 29 TB of re-downloads, which is absurd. Byte-range requests don't save us — GRIB2 messages can only be decoded whole, so we'd have to pull ~350 × 1.5 MB per point, and AWS doesn't support server-side subsetting on this bucket.

What to do instead

Two workable architectures:

1. Batch per run hour. A single worker downloads one full wrfnatf00.grib2 (~566 MB), extracts native profiles at all points of interest for that hour in one pass (the grid of contacts + whatever points the live scorer wants), and bulk-upserts them into hrrr_native_profiles. This is how the existing PropagationGridWorker pipeline works for the pressure-level file; we mirror it for native levels.

2. Per-day local cache. Download one file per (date, hour), stash it on local disk for the duration of a backfill job, extract all points of interest for that hour, delete the file. This is probably only worth the bandwidth for dense backfills and can be folded into option 1.

Plan correction: the HrrrNativeFetchWorker in Task 1.4 should be renamed HrrrNativeGridWorker and take {date, hour} as its unique key, not {lat, lon, valid_time}. The backfill mix task in Task 1.5 iterates over historical (date, hour) pairs where we have contacts and enqueues one grid worker per hour.

Bandwidth budget

If we backfill 1 year of hourly native grids for every hour where we have at least one QSO, and we have ~58k QSOs spread over 5+ years, conservatively ~15k distinct (date, hour) hours. 15k × 566 MB ≈ 8.5 TB. That's a lot. We don't need it all — Phase 2 only needs enough data to backtest. Realistic first backfill:

• Top 500 distinct hours by QSO count → ~280 GB (one-time, finishes overnight)
• Extract only the lat/lon points where we have QSOs in that hour plus the propagation grid points
• Store derived profiles only (< 5 KB per profile), not the raw GRIB2

Storage for derived profiles: 500 hours × ~300 points/hour × 5 KB ≈ 750 MB in the DB. Fine.

Proposed Phase 1 plan adjustments

1. Rename worker: HrrrNativeGridWorker instead of HrrrNativeFetchWorker; unique keys {date, hour}; downloads the full file once per run hour and bulk-processes points.
2. Keep hrrr_native_profiles schema as planned (per-point, per-valid_time), just populated in batch.
3. Task 1.5 backfill mix task now enqueues grid workers for the top-N (date, hour) hours by QSO density.
4. Feature functions in later phases look up hrrr_native_profiles the same way the existing scorer looks up hrrr_profiles — no API change on the consuming side.
5. Computing Td from SPFH is a small Phase 2 helper; budget ~half a day for it.

Gate decision

Gate pass. Files are available historically, all required variables are present, vertical resolution is 5-30× finer than the pressure-level product near the surface, and the ingestion architecture is a known pattern (matches the existing grid worker). Proceed to Task 1.2 (schema) with the batch-mode correction.