run_analysis_step chains the Rust-side f00 work end-to-end:
- fetch surface + pressure GRIB2 + native duct in parallel
(tokio::try_join!) with per-leg error-lift to PipelineError
- merge pressure into surface, fold native duct metrics per cell
- NEXRAD composite reflectivity overlay via the IEM n0q PNG path
- commercial-link degradation via the Postgres per-link baseline
- score all 23 bands in parallel (rayon) then write band files
via parallel spawn_blocking (try_join_all)
- write the ProfilesFile as MessagePack alongside the score files
worker.rs main loop now claims kind='analysis' first, falls back to
kind='forecast'. Analysis tasks dispatch to run_analysis_step;
forecast tasks still go through run_chain_step unchanged.
ChainOutcome enum tags log events (kind=analysis|forecast) and
carries analysis-specific counters (profile_cells_written, duct_cells,
nexrad_cells_with_echo, commercial_cells_boosted) so Phase 3 cutover
observability is readable from kubectl logs alone. 118 Rust tests
green. Elixir seeder flip + f00 code deletion lands next.
Rust-side writer for the f00 ProfilesFile. Replaces the Elixir ETF
format (:erlang.term_to_binary) with MessagePack + gzip so Rust can
produce files without porting the Erlang serializer.
Wire layout: top-level map with 'v' (format version), 'valid_time'
(ISO string), and 'cells' (array of {lat, lon, profile}). Profile
values are an rmpv::Value so the writer stays agnostic about what
fields the f00 pipeline decides to include — native-duct metrics,
NEXRAD reflectivity, commercial-link degradation, and the sounding
profile list all serialize through the same path.
All keys are strings — MessagePack has no atom type. Elixir's reader
(follow-up commit) converts known keys to atoms via a whitelist.
Atomic writes via rename(2) mirror ScoresFile/ProfilesFile pattern:
write to tmp sibling, rename into place on the shared NFS mount.
4 unit tests cover round-trip, atomic rename, path layout, and
lat/lon snap.
native_duct.rs is the Rust side of HrrrNativeClient.fetch_native_duct_grid:
fetch native-level HRRR idx → select duct byte ranges (TMP/SPFH/HGT/PRES
× 50 hybrid levels = 200 messages, ~300 MB subset) → wgrib2 decode via
the existing decoder::extract_grid plumbing → per-cell build_native_profile
→ duct::analyze → HashMap<(lat, lon), DuctMetrics>.
build_native_profile keeps the Elixir semantics intact: drop cells that
don't reach 3 levels, sort by ascending height, tolerate missing SPFH/PRES
as zero (the duct math handles zeros gracefully). merge_duct_grid gives
the pipeline.rs caller a drop-in replacement for Elixir's apply_duct_grid.
claim_next_analysis is the analysis-lane companion to claim_next: selects
kind='analysis' rows only, same SKIP LOCKED + newest-first ordering. Kept
on a separate claim function so the forecast lane can keep running
unaffected during cutover; the analysis pipeline wiring lands next.
9 pure unit tests cover message generation, URL builder, profile-sort,
min-level filter, and merge-into-base behaviour. Network-live golden
against a real native GRIB2 deferred to the cutover commit.
Port of the subset of Microwaveprop.Weather.NexradClient used by
f00's chain step: 5-min-rounded IEM n0q PNG fetch, palette-indexed
decode via the png crate, and per-POI max_reflectivity_dbz extraction
from a 25-pixel half-box (matches Elixir's DEFAULT_BOX_HALF).
The palette↔dBZ math and (lat, lon)↔pixel projection are bit-for-bit
copies of the Elixir formulas. 6 pure-math unit tests cover pixel
mapping, box clipping, and timestamp rounding. Network-live golden
fixture deferred until the full Stream A wiring lands.
Port of Microwaveprop.Commercial.build_link_lookup/2 +
link_degradation_from_lookup/3. The Elixir context stays intact
(LiveView sensor + SNMP poller); this Rust side is the read-only
consumer used by the f00 chain step (Phase 3 Stream A).
Three pure-math unit tests cover: haversine distance (DFW→Austin and
same-point zero), degradation_at aggregation behaviour, and
two-decimal rounding convention that Elixir's Float.round(_, 2)
imposes. SQL queries mirror the Elixir Ecto queries one-for-one —
baseline window, current point-in-time read, and MIN_BASELINE_SAMPLES
quality gate.
Pure port of lib/microwaveprop/propagation/duct.ex — the ITU-R P.834
duct detector that computes the modified refractivity M-profile and
finds regions where dM/dh < 0, then uses Bean & Dutton to derive the
minimum trapped frequency per duct.
Part of Stream A (f00 port): compute_duct_metrics and the per-cell
native-level duct merge both build on analyze(). Landing it now with
standalone unit tests lets later tasks (fetch_native_duct_grid, the
wgrib2 subprocess for native levels) treat it as a known-good
dependency instead of a concurrent port.
Three independent improvements in one commit:
1. Parallel band scoring (pipeline.rs):
- rayon par_iter across 23 bands replaces the serial for loop.
- All band files now write via try_join_all over spawn_blocking
instead of serializing one-at-a-time on NFS. Chain per-step
drops from ~60s to ~15-25s on a 4-core box.
2. Prometheus metrics (new metrics.rs):
- axum server on METRICS_ADDR (default :9100) exposes /metrics and
/health. Scraped by existing Prometheus via annotation.
- Histograms: chain step duration (by outcome), decode duration.
- Gauge: tasks in flight (RAII guarded so panics don't leak).
- Counter: chain steps by outcome.
- worker.rs records step duration and wraps each run in an
InFlightGuard.
3. HA + ordering fixes:
- deployment-grid-rs.yaml: replicas 1→2, required podAntiAffinity
spreads them across hosts, nodeAffinity prefers talos5 for one.
PROP_GRID_RS_PARALLELISM 4→3 per pod (6 concurrent across
replicas; smaller secondary-node footprint).
- Readiness probe on /health so rollouts wait for the runtime to
be alive.
- claim_next ordering: run_time ASC → DESC so the newest hourly
run drains before any stragglers from a broken prior run. Within
a run, forecast_hour ASC keeps nearest-first.
- grid_task_enqueuer sets kind="forecast" explicitly and updates
conflict_target to the new 3-column unique index introduced by
migration 20260419222624.
Replaces the serial claim→process loop with a JoinSet of N worker tasks,
each running the same claim→fetch→decode→score→complete flow against
grid_tasks. FOR UPDATE SKIP LOCKED already prevents contention between
workers so no additional coordination is needed.
talos5 is 4c/4t (i5-7260U); PROP_GRID_RS_PARALLELISM=4 turns the full
f01..f18 chain from ~18 min serial into roughly ~5 min. Per-task peak
memory after the streamed-bands refactor is ~250 Mi, so 4 concurrent is
~1 Gi — well under the 2 Gi container limit.
Postgres pool sized to parallelism + 2 so NOTIFY and an opportunistic
claim retry share the pool without blocking a worker transaction.
Introduces TaskKind { Forecast, Analysis } and threads it through the
claim flow. Forecast-only filter on the claim query keeps the claim
lane clean while Elixir still owns f00 — a half-deployed cluster won't
deadlock on an unfulfillable analysis row.
Prep for Stream A of Phase 3: once the Rust-side native-duct + NEXRAD +
commercial pipeline lands, a sibling claim_next_analysis function pulls
kind='analysis' rows and dispatches to the new pipeline.
Previous pipeline held three overlapping large structures in RAM:
1. `merged: HashMap<Key, HashMap<String, f32>>` — 92k outer × ~60 String
keys inner = ~200 MB of fragmented heap
2. `per_band: HashMap<u32, Vec<ScorePoint>>` — all 23 bands × 92k
ScorePoints before the first write
3. A `scores.clone()` per band during the write loop (doubled that
band's footprint transiently)
Consolidate into:
- Consume `merged` into a compact `Vec<(lat, lon, Conditions,
BandInvariants)>`; the huge String-keyed HashMaps drop as soon as
collection finishes.
- Loop over bands: score → write → vector drops at end of iteration.
One band's ~1.8 MB score vector lives at a time.
Peak heap goes from ~1.5 Gi to ~250 Mi on the chain step. 2 Gi limit
stays, but we now have 8× headroom instead of tight-fit.
Elixir's Ecto :utc_datetime maps to Postgres TIMESTAMP WITHOUT TIME
ZONE — Ecto's contract is that the value is UTC, but the column
itself is naive. sqlx's DateTime<Utc> binds/reads as TIMESTAMPTZ, so
the read failed with "mismatched types ... TIMESTAMPTZ is not
compatible with SQL type TIMESTAMP".
Read as NaiveDateTime and reattach UTC at the decode boundary, keeping
the downstream DateTime<Utc> signature. Test bind goes the other
direction — bind .naive_utc() against the TIMESTAMP column.
Previous Dockerfile compiled a 'fn main(){}' placeholder to warm the
dep cache, then was supposed to rebuild with real source — but cargo's
fingerprint wasn't picking up the re-added source after rm -rf, so the
placeholder artefact (~430 KB) was what got COPY'd into runtime.
Replace the stub-then-real dance with a single cache-mounted build.
BuildKit persists /usr/local/cargo/registry + git + /src/target across
runs, so dep changes are the only full-rebuild trigger. Copy the real
binary into /usr/local/bin/worker before the COPY --from= in runtime
because COPY --from= can't read from a cache mount.
wgrib2 isn't in Debian apt on trixie — the main prop Dockerfile builds
it from source in a dedicated stage. Rather than duplicate that 5-min
compile in the Rust image's CI path, copy the already-built
/usr/local/bin/wgrib2 + /usr/local/lib/libg2c.so* out of the published
Elixir image.
Also add the libgfortran5/libaec0/zlib1g/libpng16-16t64/libopenjp2-7
runtime deps that wgrib2 links dynamically against — same set the
Elixir runtime installs.
The previous approach (`docker run -v $(pwd)/rust/prop_grid_rs:/src`)
fails under act_runner because the runner container's pwd isn't a path
the host docker daemon can see — clippy installed, then cargo couldn't
find Cargo.toml.
Move the lint + test gate into the Dockerfile builder stage so the
build context ships over the socket the normal way. If clippy warns or
a test regresses, the image build fails and nothing is pushed.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
