Replaces the serial claim→process loop with a JoinSet of N worker tasks,
each running the same claim→fetch→decode→score→complete flow against
grid_tasks. FOR UPDATE SKIP LOCKED already prevents contention between
workers so no additional coordination is needed.
talos5 is 4c/4t (i5-7260U); PROP_GRID_RS_PARALLELISM=4 turns the full
f01..f18 chain from ~18 min serial into roughly ~5 min. Per-task peak
memory after the streamed-bands refactor is ~250 Mi, so 4 concurrent is
~1 Gi — well under the 2 Gi container limit.
Postgres pool sized to parallelism + 2 so NOTIFY and an opportunistic
claim retry share the pool without blocking a worker transaction.
Introduces TaskKind { Forecast, Analysis } and threads it through the
claim flow. Forecast-only filter on the claim query keeps the claim
lane clean while Elixir still owns f00 — a half-deployed cluster won't
deadlock on an unfulfillable analysis row.
Prep for Stream A of Phase 3: once the Rust-side native-duct + NEXRAD +
commercial pipeline lands, a sibling claim_next_analysis function pulls
kind='analysis' rows and dispatches to the new pipeline.
Previous pipeline held three overlapping large structures in RAM:
1. `merged: HashMap<Key, HashMap<String, f32>>` — 92k outer × ~60 String
keys inner = ~200 MB of fragmented heap
2. `per_band: HashMap<u32, Vec<ScorePoint>>` — all 23 bands × 92k
ScorePoints before the first write
3. A `scores.clone()` per band during the write loop (doubled that
band's footprint transiently)
Consolidate into:
- Consume `merged` into a compact `Vec<(lat, lon, Conditions,
BandInvariants)>`; the huge String-keyed HashMaps drop as soon as
collection finishes.
- Loop over bands: score → write → vector drops at end of iteration.
One band's ~1.8 MB score vector lives at a time.
Peak heap goes from ~1.5 Gi to ~250 Mi on the chain step. 2 Gi limit
stays, but we now have 8× headroom instead of tight-fit.
Elixir's Ecto :utc_datetime maps to Postgres TIMESTAMP WITHOUT TIME
ZONE — Ecto's contract is that the value is UTC, but the column
itself is naive. sqlx's DateTime<Utc> binds/reads as TIMESTAMPTZ, so
the read failed with "mismatched types ... TIMESTAMPTZ is not
compatible with SQL type TIMESTAMP".
Read as NaiveDateTime and reattach UTC at the decode boundary, keeping
the downstream DateTime<Utc> signature. Test bind goes the other
direction — bind .naive_utc() against the TIMESTAMP column.
Previous Dockerfile compiled a 'fn main(){}' placeholder to warm the
dep cache, then was supposed to rebuild with real source — but cargo's
fingerprint wasn't picking up the re-added source after rm -rf, so the
placeholder artefact (~430 KB) was what got COPY'd into runtime.
Replace the stub-then-real dance with a single cache-mounted build.
BuildKit persists /usr/local/cargo/registry + git + /src/target across
runs, so dep changes are the only full-rebuild trigger. Copy the real
binary into /usr/local/bin/worker before the COPY --from= in runtime
because COPY --from= can't read from a cache mount.
wgrib2 isn't in Debian apt on trixie — the main prop Dockerfile builds
it from source in a dedicated stage. Rather than duplicate that 5-min
compile in the Rust image's CI path, copy the already-built
/usr/local/bin/wgrib2 + /usr/local/lib/libg2c.so* out of the published
Elixir image.
Also add the libgfortran5/libaec0/zlib1g/libpng16-16t64/libopenjp2-7
runtime deps that wgrib2 links dynamically against — same set the
Elixir runtime installs.
The previous approach (`docker run -v $(pwd)/rust/prop_grid_rs:/src`)
fails under act_runner because the runner container's pwd isn't a path
the host docker daemon can see — clippy installed, then cargo couldn't
find Cargo.toml.
Move the lint + test gate into the Dockerfile builder stage so the
build context ships over the socket the normal way. If clippy warns or
a test regresses, the image build fails and nothing is pushed.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
