The prior Docker build re-compiled all 546 transitive Rust deps from
scratch on every push because target/ is intentionally not cache-mounted
(Stream C disk-pressure failures). Two changes drop the cold-build cost
substantially:
* cargo-chef splits dep compilation into a separate Docker layer keyed
on recipe.json (derived from Cargo.toml/Cargo.lock). Source-only
pushes — the common case — now reuse the cooked deps as a normal
layer instead of re-running cargo for every crate.
* clippy now runs in --release profile so it shares artifacts with
`cargo test --release` and `cargo build --release`. Previously clippy
ran in dev profile, forcing a full second compile of every dep before
the release build could start.
Verified `cargo clippy --release --all-targets -- -D warnings` is clean
locally — no new lints surface under the release profile.
wgrib2 3.8.0's CMakeLists.txt:93 calls find_package(g2c 2.3.0), so the
previous 2.1.0 pin makes both Dockerfiles error out with "Could not find
a configuration file for package g2c that is compatible with requested
version 2.3.0". Bumping to 2.3.0 across both the Elixir and Rust images
since their wgrib2-builder stages stay in lockstep.
Production wgrib2 3.8.0 still hit `*** FATAL ERROR: packing type 40
not supported ***` on HRDPS files. ldd confirmed wgrib2 was linking
only libc/libm — no OpenJPEG, no Jasper, no g2c.
wgrib2 3.8.0 cmake doesn't have USE_OPENJPEG or USE_JASPER options.
The legacy flag names we passed were silently ignored. The actual
knob is USE_G2CLIB_LOW, which links wgrib2 to the external g2c lib
(already built in the previous stage with OpenJPEG support). Dropped
the bogus USE_PNG / USE_OPENJPEG flags and added USE_G2CLIB_LOW=ON
in both Dockerfiles.
Verified by reading wgrib2 v3.8.0's CMakeLists.txt: the documented
JPEG2000 path is "USE_G2CLIB_LOW: Use g2c low-level decoders
(png,jpeg2000)?".
Reproduced in production pod 2026-04-30: wgrib2 3.6.0 crashes with
`free(): invalid size` when -lon point extraction is run against any
HRDPS rotated lat/lon GRIB2 file, regardless of batch size (failed at
N=200, N=1000, N=2000). Output before the abort is denormal f32 garbage
(val=4.93944e-41) — wgrib2's internal record buffer is being corrupted.
Same files extract cleanly with wgrib2 3.8.0 locally. Bumping the build
arg in both Dockerfile (Elixir image, used by hrrr-point-rs/per-QSO
HRDPS lookups) and rust/prop_grid_rs/Dockerfile (Rust grid worker) so
both consumers get the fix.
Also bumped POINT_BATCH 200 → 1000 in decoder.rs. With wgrib2 3.8.0
stable at 1000+ point batches, this drops the chain-step invocation
count from 286 → 57 per step. Comment captures the reproduction
context so a future bisection finds it quickly.
The FROM ${WGRIB2_IMAGE} AS wgrib2-src pattern required docker buildx
to pull a private-registry image mid-build. That pull started failing
consistently after Stream C landed — docker login on the runner
succeeds but buildx's implicit FROM pull can't always reach the same
credentials (buildx instance context vs the host daemon).
Rebuild wgrib2 + NCEPLIBS-g2c from source in a dedicated builder stage,
mirroring the Elixir Dockerfile's self-contained wgrib2-builder. Adds
~5 min per CI run — worth the end of a 3-run CI-failure streak.
The docker buildx build step has failed twice since Stream C added the
hrrr_point_worker binary. Dropping the target/ BuildKit cache mount
eliminates the most likely cause (disk pressure inside a persistent
cache shared between the two bin builds). Full rebuild from scratch
adds ~3 min to the build but the failure mode is worth more than the
cache.
Also dropped the empty unit test scaffolding in hrrr_points.rs; the
module is thin glue over fetcher+decoder which already carry coverage.
Integration tests for process_batch/upsert_profile are gated on a live
DB + HRRR mirror and live in tests/ (when added).
Two bugs surfacing 13 h after Stream A cutover:
1. prop-grid-rs pods OOMKilled 16–18× overnight on the analysis step.
f00's native-level GRIB2 (~530 MB + wgrib2 working set) runs on top
of 2 concurrent forecast tasks, briefly reaching ~2 Gi. 2 Gi limit
was too tight — 3 Gi plus parallelism 3→2 gives the analysis step
room without eliminating forecast-lane headroom.
2. hrrr-point-rs pod is CrashLoopBackOff because its container is
running image main-1776640915-65f7963 (pre-Stream-C) which doesn't
contain the hrrr_point_worker binary. The grid-rs CI run for
commit 4fefb81 failed and no newer image got published. Touch the
Dockerfile to re-trigger the workflow so flux picks up a new tag
with both binaries. No functional Dockerfile change, just a
docstring update so the path-filter kicks.
Forecast + analysis both claim per FOR UPDATE SKIP LOCKED, so
reducing per-pod parallelism doesn't break the chain — the two
replicas cover 4 slots cluster-wide, which still drains f01..f18 in
~5 min.
Phase 3 Stream C Rust side. Completes the HrrrFetchWorker port.
Pipeline:
- db::claim_next_hrrr_task — FOR UPDATE SKIP LOCKED on hrrr_fetch_tasks,
newest valid_time first. Accepts the points JSONB directly.
- hrrr_points::process_batch — fetch surface + pressure GRIB2 once
per task (tokio::try_join), decode via the existing wgrib2 plumbing,
then for each requested point pull the cell and UPSERT INTO
hrrr_profiles (conflict on lat/lon/valid_time).
- db::complete_hrrr_task / fail_hrrr_task — status transitions; Elixir
backfill re-enqueues failed rows on next /30-min scan.
Shipping pieces:
- new bin src/bin/hrrr_point_worker.rs
- new module src/hrrr_points.rs (process_batch, upsert_profile)
- new Cargo [[bin]] entry; Dockerfile builds both binaries in one stage
and ships them in the runtime image so a single CI pipeline covers
the whole cluster
- k8s/deployment-hrrr-point-rs.yaml (1 replica, 1 Gi limit, anti-affinity
against prop-grid-rs so chain + point work don't fight for wgrib2
slots). Uses the same image; command: override picks the right binary.
- kustomization.yaml: include the new deployment so flux applies it
- deployment-grid-rs.yaml: bump readiness initialDelaySeconds 3→15 +
failureThreshold 3→6 so a slow DB connect during startup can't race
the first probe
119 Rust tests green.
Previous Dockerfile compiled a 'fn main(){}' placeholder to warm the
dep cache, then was supposed to rebuild with real source — but cargo's
fingerprint wasn't picking up the re-added source after rm -rf, so the
placeholder artefact (~430 KB) was what got COPY'd into runtime.
Replace the stub-then-real dance with a single cache-mounted build.
BuildKit persists /usr/local/cargo/registry + git + /src/target across
runs, so dep changes are the only full-rebuild trigger. Copy the real
binary into /usr/local/bin/worker before the COPY --from= in runtime
because COPY --from= can't read from a cache mount.
wgrib2 isn't in Debian apt on trixie — the main prop Dockerfile builds
it from source in a dedicated stage. Rather than duplicate that 5-min
compile in the Rust image's CI path, copy the already-built
/usr/local/bin/wgrib2 + /usr/local/lib/libg2c.so* out of the published
Elixir image.
Also add the libgfortran5/libaec0/zlib1g/libpng16-16t64/libopenjp2-7
runtime deps that wgrib2 links dynamically against — same set the
Elixir runtime installs.
The previous approach (`docker run -v $(pwd)/rust/prop_grid_rs:/src`)
fails under act_runner because the runner container's pwd isn't a path
the host docker daemon can see — clippy installed, then cargo couldn't
find Cargo.toml.
Move the lint + test gate into the Dockerfile builder stage so the
build context ships over the socket the normal way. If clippy warns or
a test regresses, the image build fails and nothing is pushed.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
