Reworks the post-fetch half of the propagation pipeline. Fetch and GRIB2
decode were already cheap — measured against a live HRRR cycle, all 39
pressure messages decode via `wgrib2 -lola` in 0.29 s and the 31 MB
byte-range fetch takes ~3 s — so nothing here touches the decoder. All the
cost was downstream.
Also fixes a broken NOTIFY that made every chain step run up to 5 times.
pg_notify
`NOTIFY propagation_ready, $1` is a Postgres syntax error: NOTIFY is a
utility statement whose payload must be a literal, so a bind raises
42601. It shared a transaction with the `status='done'` UPDATE, so every
successful step rolled back, stayed 'running', and was requeued by
reclaim_stale_running up to @max_reclaim_attempts times. Elixir's
NotifyListener never fired either, so ScoreCache warm and the
"propagation:updated" fan-out were dead.
FieldGrid
A decoded grid was HashMap<(i32,i32), HashMap<Arc<str>, f32>> — a dense
rectangular grid stored as ~95k nested hash maps, costing ~4.6M inserts
on decode, ~3.7M on merge and ~14M lookups across three derivation
passes. wgrib2 -lola already emits one dense row-major f32 block per
message, so keep it: dense per-message planes, names hashed once per
grid into plane ids, NaN as the missing sentinel. This is what forced
PROP_GRID_RS_PARALLELISM=1 under a 3Gi limit.
Fused pass
Three 95k-cell derivation passes plus 23 band-major scoring passes over
a staged Vec<(f64,f64,Conditions,BandInvariants)> (~19MB re-streamed 23
times) collapse into one pass: levels extracted once per cell, all 23
bands scored while the cell is hot, scores accumulated cell-major so
rayon chunks own disjoint slices. Scores land straight in the dense
score-file body — no ScorePoint scatter.
.pgrid
The profile artifact was an rmpv tree plus gzip -9, written 30x an hour,
and ProfilesFile.read_point/3 gunzipped and unpacked the entire 95k-cell
file to return one cell on every map click and Skew-T load. Replaced
with a dense cell-major f32 record array carrying a self-describing
field table. Elixir reads it via :file.pread; .mp.gz and .etf.gz remain
readable so files written before this drain out of the 48h window.
Measured on a full CONUS grid (95,073 cells x 48 planes x 23 bands):
derive + score + build artifacts 0.022 s
profile write 3.957 s -> 0.006 s (22.0 MB -> 22.4 MB on disk)
single-cell read whole-file decode -> 0.5 us
23 score files 0.003 s
Also
- hrrr_points: batched UNNEST upsert replacing one awaited INSERT per
point. Keeps ON CONFLICT DO UPDATE — the PSKR sampler's two-pass loop
depends on it.
- fetcher: real semaphore capping in-flight ranges at
MAX_PARALLEL_RANGES, which the comment claimed but the code did not do
(it spawned all 27 while the connection pool was sized for 8).
- metrics: per-stage histogram. Only chain-step and decode durations
were instrumented, which is why the write cost stayed invisible.
- profiles_file: parse_valid_time anchors on the known extension set, so
sibling-suffixed names like <iso>.hrdps.prop no longer parse as
<iso>.hrdps and vanish from prune and list operations.
- PROP_GRID_RS_PARALLELISM 1 -> 3. Memory limit held at 3Gi until RSS is
observed at the new parallelism.
- cargo fmt over the crate; worker.rs, hrdps_fetcher.rs and nexrad.rs
were already unformatted at HEAD and the pre-commit hook gates on it.
HRDPS still runs at 0.5 degrees. wgrib2 -lola scales linearly in output
points on rotated lat/lon (12.5 s wall, 202 s CPU for one message at
0.125 degrees) because it has no inverse projection for those grids; a raw
native dump is 0.32 s. The fix is decode-once plus a closed-form
rotated-pole index, left for a follow-up.
Pre-existing CI failure (`tests/scorer_golden.rs`) — the Rust scorer
was still on the pre-recalibration weight vector, while the Elixir
side picked up new gradient-descent-fit weights in f98ee153.
- Update `DEFAULT_WEIGHTS` in band_config.rs to match the new Elixir
values (humidity 0.1262, time_of_day 0.0380, td_depression 0.1010,
refractivity 0.0986, sky 0.0841, season 0.1134, wind 0.0841,
rain 0.1431, pressure 0.0967, pwat 0.1147).
- Loosen the sum-to-one tolerance in `default_weights_sum_to_one`
from 1e-9 to 1e-3 to match Elixir's `assert_in_delta total, 1.0,
0.001` — gradient-descent fits land near 1.0 but not exact.
- Regenerate `tests/scores.bincode` from `mix rust.golden` so the
golden fixture reflects the recalibrated scorer outputs.
`cargo clippy --all-targets -- -D warnings` clean.
139 Rust unit tests + golden fixture pass.
3072 Elixir tests pass.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
