Graham McIntire
b80878056d
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
33 lines
1.1 KiB
Docker
33 lines
1.1 KiB
Docker
# syntax=docker/dockerfile:1.7
|
|
#
|
|
# Multi-arch build for `prop-grid-rs`. wgrib2 is installed from Debian
|
|
# Trixie (same version the Elixir image uses). The binary runs as a
|
|
# non-root uid that matches the fsGroup on the NFS scores mount.
|
|
FROM rust:1.94-trixie AS builder
|
|
WORKDIR /src
|
|
|
|
# Cache deps: copy manifests first so dependency changes alone don't
|
|
# invalidate the source-layer cache.
|
|
COPY Cargo.toml Cargo.lock* ./
|
|
RUN mkdir -p src src/bin \
|
|
&& echo 'fn main() {}' > src/bin/worker.rs \
|
|
&& echo '' > src/lib.rs \
|
|
&& cargo build --release --bin worker \
|
|
&& rm -rf src
|
|
|
|
COPY src ./src
|
|
RUN cargo build --release --bin worker
|
|
|
|
# Runtime image: slim debian plus wgrib2. libexpat/libjasper pulled in
|
|
# transitively by wgrib2 for PNG/JPEG2000 GRIB2 decompression.
|
|
FROM debian:trixie-slim AS runtime
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
wgrib2 \
|
|
ca-certificates \
|
|
tini \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
COPY --from=builder /src/target/release/worker /usr/local/bin/prop-grid-rs
|
|
|
|
USER 65534:65534
|
|
ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/prop-grid-rs"]
|