prop-pipeline: .sgrid dense scalars, HRDPS rotated-pole decode-once, memory trim
Item 1 — .sgrid dense binary scalar format replacing chunked gzip+msgpack
- New rust/prop_grid_rs/src/sgrid.rs: magic SGRD, same header layout as .pgrid,
20-field cell-major f32 body, write_atomic (tmp+rename, NFS-safe)
- New lib/microwaveprop/weather/sgrid.ex: Elixir reader modeled on pgrid.ex
(pread single-cell reads, bounds-filtered viewport reads, NaN→nil sentinel)
- ScalarFile updated to prefer .sgrid reads, chunked .mp.gz as fallback
- Pipeline writes .sgrid alongside existing chunked format (all three paths)
Item 2 — HRDPS decode-once + rotated-pole index, restore 0.125° resolution
- New rust/prop_grid_rs/src/rotated_pole.rs: CF-convention geographic→rotated
transform, OnceLock-cached GDS params parsed from wgrib2 -grid, precomputed
Vec<u32> lookup table mapping target cells to native grid indices
- Native decode path in decoder.rs: wgrib2 -no_header -order we:sn -bin
(raw f32 dump, ~0.32 s/message) + indexing via lookup table
- HRDPS_STEP: 0.5° → 0.125° (4× finer, ~57k Canadian cells vs ~3.5k)
Item 3 — k8s memory limits: 3Gi → 1.5Gi
(per-task grid footprint: ~200-400 MB HashMap → ~18 MB dense planes)
Item 4 — CLAUDE.md and profiles_file.ex documentation drift fixed:
.pgrid primary, .mp.gz legacy, .sgrid added, write_atomic protocol doc,
cleanup gaps reorganized, 'Only f00 is persisted' corrected to f00..f48
Graham McIntire
db57b3a1c7