prop/docs/plans
Graham McIntire 13180d572e
NARR backfill plan + spike-captured inventory fixture
Replaces the broken ERA5/CDS path with NCEP NARR fetched anonymously
from NCEI. NARR is GRIB1, anonymous HTTPS, byte-range-fetchable, with
1979-01-01 → 2014-10-02 coverage that lines up perfectly against the
HRRR archive start. Variables match the existing era5_profiles schema
1:1 (DPT is a direct surface dewpoint record, no SPFH derivation
needed at the surface).

Phase 1 spike done by hand: byte-range fetched 5 surface records and
3 pressure-level records for DFW 2010-06-15 12Z. wgrib2 turned out to
NOT read GRIB1 messages (rd_grib2_seq_msg, grib1 message ignored, use
wgrib), so the plan now uses cdo for both GRIB1 decoding and Lambert
conformal nearest-neighbor remap. cdo -merge is the right operator
for combining records of different variables — cdo -cat is for time
concat and silently drops mismatched levels, shell cat produces an
invalid composite. cdo emits param numbers as varNNN; mapping table
is in the plan's Architecture section.

Spike values for DFW 2010-06-15 12Z (all physically sane):
  TMP_2m=296.94 K (23.8°C)   DPT_2m=294.49 K (21.3°C)
  HPBL=703 m                  PRES_sfc=99383 Pa
  PWAT=45.1 mm                TMP_850mb=291.52 K
  HGT_850mb=1547.8 m          SPFH_850mb=10.4 g/kg

Plan file kept at the original 2026-04-15-merra2-historical-backfill
filename for git history continuity even though the implementation
is NARR not MERRA-2 (history block at the top documents the pivot).
2026-04-15 17:10:46 -05:00
..
2026-03-30-commercial-link-monitoring.md Add commercial link monitoring via SNMP polling 2026-03-30 13:02:59 -05:00
2026-03-30-conus-propagation-map.md Refactor HRRR fetch to batch points per hour 2026-03-30 17:21:47 -05:00
2026-04-01-algorithm-refinement-ml-training-design.md Normalize ML features to prevent NaN gradient explosion 2026-04-01 09:08:33 -05:00
2026-04-01-algorithm-refinement-ml-training.md Normalize ML features to prevent NaN gradient explosion 2026-04-01 09:08:33 -05:00
2026-04-07-multi-source-atmospheric-data.md Path-integrated HRRR scoring across pos1/mid/pos2 2026-04-07 12:04:16 -05:00
2026-04-09-propagation-modeling-improvements.md Plan: propagation modeling improvements from April 2026 review 2026-04-09 16:10:54 -05:00
2026-04-11-contact-edit-approval-design.md Add contact edit approval system with admin review queue 2026-04-11 16:15:49 -05:00
2026-04-13-hrdps-canadian-prop-grid.md Ingest Canadian radiosondes via UWYO + plans for RDPS/HRDPS 2026-04-13 09:14:34 -05:00
2026-04-13-rdps-vertical-profiles.md Ingest Canadian radiosondes via UWYO + plans for RDPS/HRDPS 2026-04-13 09:14:34 -05:00
2026-04-14-duckdb-scores-storage.md Move NFS share from skippy to node3 at 10.0.15.103:/data 2026-04-14 14:25:42 -05:00
2026-04-15-merra2-historical-backfill.md NARR backfill plan + spike-captured inventory fixture 2026-04-15 17:10:46 -05:00