Reproduced in production pod 2026-04-30: wgrib2 3.6.0 crashes with
`free(): invalid size` when -lon point extraction is run against any
HRDPS rotated lat/lon GRIB2 file, regardless of batch size (failed at
N=200, N=1000, N=2000). Output before the abort is denormal f32 garbage
(val=4.93944e-41) — wgrib2's internal record buffer is being corrupted.
Same files extract cleanly with wgrib2 3.8.0 locally. Bumping the build
arg in both Dockerfile (Elixir image, used by hrrr-point-rs/per-QSO
HRDPS lookups) and rust/prop_grid_rs/Dockerfile (Rust grid worker) so
both consumers get the fix.
Also bumped POINT_BATCH 200 → 1000 in decoder.rs. With wgrib2 3.8.0
stable at 1000+ point batches, this drops the chain-step invocation
count from 286 → 57 per step. Comment captures the reproduction
context so a future bisection finds it quickly.
Production observation 2026-04-29: HRDPS chain steps using
decoder::extract_grid (wgrib2 -lola) ran at 1000m CPU for 10+ minutes
per task without producing any output. The -lola full-grid
interpolation against HRDPS's rotated lat/lon source grid is far
slower than against HRRR's Lambert source — wgrib2 appears to rebuild
the projection table per record.
The probe (lib/mix/tasks/hrdps_probe.ex, deleted in 1da78a80) had
already proved that wgrib2 -lon at five Canadian cities completes in
under a second. Add decoder::extract_points that uses the same -lon
flag for the HRDPS-only point set (~57k cells), batched at 2000 points
per wgrib2 invocation to stay under Linux ARG_MAX.
run_chain_step_hrdps now calls extract_points instead of extract_grid.
The post-extract HashSet filter to hrdps_only_points is gone —
extraction is already restricted to those cells. Per-cell wall time
should drop from "never completes" to ~30-90 s for 57k cells.
extract_grid stays unchanged — it works correctly for HRRR's CONUS
Lambert source.
Tests: parse_lon_output exercises the wgrib2 -s -lon output shape and
proves multi-record / multi-point parsing. parse_lon_segment unit
tests cover the undefined-value sentinel and the snap-back rejection
that ignores cells wgrib2 returned that don't correspond to anything
in the requested batch.
The /skewt page was empty in prod because Phase 2 of the Rust cutover
moved f01..f18 grid scoring to Rust but left profile-file writing only
on the analysis (f00) path. Result: /data/scores/profiles/ accumulated
one file per chain run instead of nineteen, so SkewtLive only ever saw
the analysis hour — and that hour falls outside the 1 h past cutoff
within an hour of the chain finishing, leaving the page empty most of
the time.
Two surgical fixes:
* `pipeline::run_chain_step` now builds `profile_entries` alongside
`prepared` from the merged grid in a single iter pass, then spawns
`profiles_file::write_atomic` on the blocking pool to overlap the
NFS write with the score-band scoring/write fan-out — same pattern
`run_analysis_step` already uses. `ChainStepStats` gains
`profile_cells_written` so the worker log line is symmetric with
the analysis step's existing field and a future failure mode that
drops the profile would show up as a divergence.
* `SkewtLive.available_valid_times/0` widens the past cutoff from
1 h to 3 h. The chain's analysis hour can be ~70 min old by the
time the next chain finishes (and ~2 h on a missed cycle), so the
1 h cutoff was leaving the page blank during normal transients.
18 h forward window unchanged.
Cosmetic: cargo fmt picked up trivial rewrites in decoder/fetcher/
native_duct/hrrr_point_worker that had drifted; rolled in.
Verification: cargo build --release, cargo clippy --lib --bins -D
warnings, cargo test --lib (134/134), mix test
test/microwaveprop_web/live/skewt_live_test.exs (3/3) all green.
The decoder's `parse_lola_binary` inserts the same "VAR:LEVEL"
string into ~92k cell maps for every one of ~60 GRIB2 messages in
a chain step — previously a String::clone per cell, yielding
~5.5M heap allocations and frees per hourly run. Switching
`CellValues` from `HashMap<String, f32>` to
`HashMap<Arc<str>, f32>` makes each insertion a single atomic
refcount increment: the string is allocated once per message, and
`Arc::clone` is cheap from there.
Callers were mostly unchanged — `cell.get("...")` still works via
`Arc<str>: Borrow<str>`. Two touch-ups needed: native_duct.rs had
a couple of `cell.get(&format!(...))` lookups that were passing
`&String` (triggering the wrong Borrow impl) — switched to
.as_str(), and the test fixtures' `.insert(format!(...), _)` now
need a `.into()` to coerce into `Arc<str>`. Zero behavior change.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
