Stages 4-5 of the HRDPS plan. Lands the Rust forecast-hour pipeline
for source='hrdps' grid_tasks rows.
What ships:
- `hrdps_fetcher.rs` — sibling of fetcher.rs for ECCC's MSC Datamart
shape: date-prefixed URLs, one variable per GRIB2 file, concurrent
fetch + byte-concat into a single multi-record blob. wgrib2 reads
multi-record files natively so the downstream decoder path is
unchanged. Mirrors lib/microwaveprop/weather/hrdps_client.ex.
- `grid::hrdps_only_points()` + `grid::hrdps_grid_spec()` — Canadian
cells (49-60°N, -141 to -52°W) minus the HRRR overlap. Disjoint
from conus_points by construction.
- `db::TaskSource` enum + `source` column on ClaimedTask. claim_next
surfaces source for forecast lane dispatch; claim_next_analysis
filters to source='hrrr' only (HRDPS analysis path is not yet
ported — would need native-duct + NEXRAD + commercial merges that
Rust doesn't have for HRDPS).
- `pipeline::run_chain_step_hrdps` — fetch combined blob, decode with
HRDPS grid spec, drop HRRR-overlap cells via a HashSet of
hrdps_only_points, back-fill DPT from DEPR (HRDPS publishes DEPR
as primary), score every band, write `<band>/<iso>.hrdps.prop`.
Skips per-valid_time profile + scalar artifacts since those would
clobber HRRR's at the same valid_time — surfacing HRDPS on /weather
is a separate stage.
- worker.rs dispatches forecast lane on (kind, source) — Hrrr →
run_chain_step, Hrdps → run_chain_step_hrdps. Analysis stays
HRRR-only. Worker constructs an HrdpsClient alongside HrrrClient.
- `scores_file::write_atomic_hrdps` writes to `.hrdps.prop` with the
Canadian grid spec via the new encode_with_spec. Coexists with the
HRRR `.prop` file for the same (band, valid_time).
163 unit tests pass; backfill_dpt_from_depr handles surface and
pressure-level levels with no double-fill when DPT exists.
Phase 3 Stream C Rust side. Completes the HrrrFetchWorker port.
Pipeline:
- db::claim_next_hrrr_task — FOR UPDATE SKIP LOCKED on hrrr_fetch_tasks,
newest valid_time first. Accepts the points JSONB directly.
- hrrr_points::process_batch — fetch surface + pressure GRIB2 once
per task (tokio::try_join), decode via the existing wgrib2 plumbing,
then for each requested point pull the cell and UPSERT INTO
hrrr_profiles (conflict on lat/lon/valid_time).
- db::complete_hrrr_task / fail_hrrr_task — status transitions; Elixir
backfill re-enqueues failed rows on next /30-min scan.
Shipping pieces:
- new bin src/bin/hrrr_point_worker.rs
- new module src/hrrr_points.rs (process_batch, upsert_profile)
- new Cargo [[bin]] entry; Dockerfile builds both binaries in one stage
and ships them in the runtime image so a single CI pipeline covers
the whole cluster
- k8s/deployment-hrrr-point-rs.yaml (1 replica, 1 Gi limit, anti-affinity
against prop-grid-rs so chain + point work don't fight for wgrib2
slots). Uses the same image; command: override picks the right binary.
- kustomization.yaml: include the new deployment so flux applies it
- deployment-grid-rs.yaml: bump readiness initialDelaySeconds 3→15 +
failureThreshold 3→6 so a slow DB connect during startup can't race
the first probe
119 Rust tests green.
native_duct.rs is the Rust side of HrrrNativeClient.fetch_native_duct_grid:
fetch native-level HRRR idx → select duct byte ranges (TMP/SPFH/HGT/PRES
× 50 hybrid levels = 200 messages, ~300 MB subset) → wgrib2 decode via
the existing decoder::extract_grid plumbing → per-cell build_native_profile
→ duct::analyze → HashMap<(lat, lon), DuctMetrics>.
build_native_profile keeps the Elixir semantics intact: drop cells that
don't reach 3 levels, sort by ascending height, tolerate missing SPFH/PRES
as zero (the duct math handles zeros gracefully). merge_duct_grid gives
the pipeline.rs caller a drop-in replacement for Elixir's apply_duct_grid.
claim_next_analysis is the analysis-lane companion to claim_next: selects
kind='analysis' rows only, same SKIP LOCKED + newest-first ordering. Kept
on a separate claim function so the forecast lane can keep running
unaffected during cutover; the analysis pipeline wiring lands next.
9 pure unit tests cover message generation, URL builder, profile-sort,
min-level filter, and merge-into-base behaviour. Network-live golden
against a real native GRIB2 deferred to the cutover commit.
Three independent improvements in one commit:
1. Parallel band scoring (pipeline.rs):
- rayon par_iter across 23 bands replaces the serial for loop.
- All band files now write via try_join_all over spawn_blocking
instead of serializing one-at-a-time on NFS. Chain per-step
drops from ~60s to ~15-25s on a 4-core box.
2. Prometheus metrics (new metrics.rs):
- axum server on METRICS_ADDR (default :9100) exposes /metrics and
/health. Scraped by existing Prometheus via annotation.
- Histograms: chain step duration (by outcome), decode duration.
- Gauge: tasks in flight (RAII guarded so panics don't leak).
- Counter: chain steps by outcome.
- worker.rs records step duration and wraps each run in an
InFlightGuard.
3. HA + ordering fixes:
- deployment-grid-rs.yaml: replicas 1→2, required podAntiAffinity
spreads them across hosts, nodeAffinity prefers talos5 for one.
PROP_GRID_RS_PARALLELISM 4→3 per pod (6 concurrent across
replicas; smaller secondary-node footprint).
- Readiness probe on /health so rollouts wait for the runtime to
be alive.
- claim_next ordering: run_time ASC → DESC so the newest hourly
run drains before any stragglers from a broken prior run. Within
a run, forecast_hour ASC keeps nearest-first.
- grid_task_enqueuer sets kind="forecast" explicitly and updates
conflict_target to the new 3-column unique index introduced by
migration 20260419222624.
Introduces TaskKind { Forecast, Analysis } and threads it through the
claim flow. Forecast-only filter on the claim query keeps the claim
lane clean while Elixir still owns f00 — a half-deployed cluster won't
deadlock on an unfulfillable analysis row.
Prep for Stream A of Phase 3: once the Rust-side native-duct + NEXRAD +
commercial pipeline lands, a sibling claim_next_analysis function pulls
kind='analysis' rows and dispatches to the new pipeline.
Elixir's Ecto :utc_datetime maps to Postgres TIMESTAMP WITHOUT TIME
ZONE — Ecto's contract is that the value is UTC, but the column
itself is naive. sqlx's DateTime<Utc> binds/reads as TIMESTAMPTZ, so
the read failed with "mismatched types ... TIMESTAMPTZ is not
compatible with SQL type TIMESTAMP".
Read as NaiveDateTime and reattach UTC at the decode boundary, keeping
the downstream DateTime<Utc> signature. Test bind goes the other
direction — bind .naive_utc() against the TIMESTAMP column.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
