The forgejo-runner was offline when 989d310..410a137 landed, so the
build-grid-rs workflow never dispatched for those pushes. Empty CI
retrigger commits don't match the path filter. Touching the crate
root forces build-grid-rs.yaml to fire and pick up the HRRR S3
fallback + scores_file .prop rename + point lookup fix.
Also corrects the now-stale "f01..f18 forecast hours only" note — as
of Phase 3 Stream A, Rust owns the f00 analysis step too.
Score-grid files now land at `<band>/<iso>.prop` with a 4-byte "PROP"
magic header. Readers still accept the legacy `.ntms` extension and
"NTMS" magic so a rolling deploy doesn't invalidate any file already
on the shared NFS mount — legacy files age out through normal
retention.
Applied both sides of the seam: Elixir ScoresFile + regex parsers,
Rust scores_file writer + decoder. Updated the comments in all
modules that mention the extension (NotifyListener, Reconciler,
gefs_fetch_worker, map_live) and the runbook diagram. talos5 is a
DB host now, not a Rust worker — corrected the runbook accordingly.
Root cause of the blank /map analysis breakdown: the skippy.w5isp.com
caching proxy was unreachable from the cluster, so every f00 analysis
step failed at the idx fetch and Rust never wrote a ProfilesFile. The
/map point-detail panel silently degraded to empty factors because
no profile ever lands on `/data/scores/profiles/`.
HrrrClient now takes an optional fallback base. When the primary base
exhausts its retries (5× exponential backoff), fetches retry once
against the fallback — defaulted to the NOAA S3 public bucket. Forecast,
analysis, and native-duct paths all go through the same
fetch_blob_with_fallback helper so a proxy outage degrades to direct
S3 instead of stalling.
Moved the native-URL builder out of native_duct into fetcher so the
single helper can build both primary and fallback URLs for the native
hybrid-sigma product.
Shared NFS idx cache at /data/hrrr_idx lets all prop-grid-rs and
hrrr-point-rs replicas deduplicate redundant NOAA S3 idx fetches.
Opt-in via HRRR_IDX_CACHE_DIR; falls back to in-memory-only when unset.
Also scales prop-grid-rs to 3 replicas (no talos5 pinning),
hrrr-point-rs to 2 replicas, and drops hot pod replicas 4→3 so the
physical-host anti-affinity still fits with talos5 cordoned off for
Postgres.
New telemetry module wires tracing-subscriber to both a local JSON
fmt layer (keeps kubectl-logs output identical) and an
OpenTelemetry OTLP/gRPC exporter, activated only when
OTEL_EXPORTER_OTLP_ENDPOINT is set. The returned TelemetryGuard
holds the SdkTracerProvider until process shutdown so queued spans
flush before exit.
Both bin targets (worker, hrrr_point_worker) now call
telemetry::init(service_name) at startup; service_name becomes the
OTel service.name attribute so Tempo groups spans per binary.
Tracing instrumentation on the three main work units:
- pipeline::run_chain_step (forecast f01..f18)
- pipeline::run_analysis_step (analysis f00)
- hrrr_points::process_batch (per-QSO point drain)
k8s manifests set OTEL_EXPORTER_OTLP_ENDPOINT to the cluster
collector at otel-collector.observability.svc.cluster.local:4317.
Backend wiring lives in the vntx-infra repo.
The commercial schema uses uuid primary keys everywhere in this project
(all schemas: `@primary_key {:id, :binary_id, autogenerate: true}`), but
the Rust port bound SELECTs against commercial_links.id and
commercial_samples.link_id as i64. The f00 analysis step failed on first
contact with:
error occurred while decoding column 0: mismatched types; Rust type
`i64` (as SQL type `INT8`) is not compatible with SQL type `UUID`
Switched LinkLookupEntry.link_id and fetch_per_link_degradation's
parameter to uuid::Uuid, updated the sqlx::query_as tuple type, and
patched the test helper to build distinct UUIDs from a byte tag so the
no-range / aggregation / rounding tests still exercise the same
identity semantics without needing a real DB column.
Three fixes after hrrr-point-rs restarts on the first live drain:
1. Type mismatch (hard error): the `profile` column on `hrrr_profiles`
is `jsonb[]` (one element per pressure level), but the Rust worker
was binding a single jsonb array-of-objects cast as `::jsonb`. Every
insert failed with `column "profile" is of type jsonb[] but
expression is of type jsonb`. Switched to
`Vec<sqlx::types::Json<Value>>`, dropped the explicit `::jsonb`
cast, and enabled sqlx's `json` feature so the encoder maps the
array into `_jsonb` correctly.
2. Silent zero-inserts: four consecutive 2019-09-22 batches completed
with `profiles_inserted: 0` and no diagnostic. Most likely the
upstream archive doesn't keep cycles that old, but without a log
it looks identical to a snap-mismatch bug. Added a WARN when both
surface and pressure grids come back empty so fetch-miss vs.
projection-miss is distinguishable.
3. OOMKilled: the pod took four OOM restarts in ten minutes at 1 Gi.
A single CONUS decode holds the ~40 MB blob, the ~200 MB wgrib2
working set, AND the full 92k-cell merged map until all requested
points drain. 1 Gi has no headroom. Bumped to 2 Gi, matching the
rest of the per-container budgets in this namespace.
The FROM ${WGRIB2_IMAGE} AS wgrib2-src pattern required docker buildx
to pull a private-registry image mid-build. That pull started failing
consistently after Stream C landed — docker login on the runner
succeeds but buildx's implicit FROM pull can't always reach the same
credentials (buildx instance context vs the host daemon).
Rebuild wgrib2 + NCEPLIBS-g2c from source in a dedicated builder stage,
mirroring the Elixir Dockerfile's self-contained wgrib2-builder. Adds
~5 min per CI run — worth the end of a 3-run CI-failure streak.
The docker buildx build step has failed twice since Stream C added the
hrrr_point_worker binary. Dropping the target/ BuildKit cache mount
eliminates the most likely cause (disk pressure inside a persistent
cache shared between the two bin builds). Full rebuild from scratch
adds ~3 min to the build but the failure mode is worth more than the
cache.
Also dropped the empty unit test scaffolding in hrrr_points.rs; the
module is thin glue over fetcher+decoder which already carry coverage.
Integration tests for process_batch/upsert_profile are gated on a live
DB + HRRR mirror and live in tests/ (when added).
Two bugs surfacing 13 h after Stream A cutover:
1. prop-grid-rs pods OOMKilled 16–18× overnight on the analysis step.
f00's native-level GRIB2 (~530 MB + wgrib2 working set) runs on top
of 2 concurrent forecast tasks, briefly reaching ~2 Gi. 2 Gi limit
was too tight — 3 Gi plus parallelism 3→2 gives the analysis step
room without eliminating forecast-lane headroom.
2. hrrr-point-rs pod is CrashLoopBackOff because its container is
running image main-1776640915-65f7963 (pre-Stream-C) which doesn't
contain the hrrr_point_worker binary. The grid-rs CI run for
commit 4fefb81 failed and no newer image got published. Touch the
Dockerfile to re-trigger the workflow so flux picks up a new tag
with both binaries. No functional Dockerfile change, just a
docstring update so the path-filter kicks.
Forecast + analysis both claim per FOR UPDATE SKIP LOCKED, so
reducing per-pod parallelism doesn't break the chain — the two
replicas cover 4 slots cluster-wide, which still drains f01..f18 in
~5 min.
Phase 3 Stream C Rust side. Completes the HrrrFetchWorker port.
Pipeline:
- db::claim_next_hrrr_task — FOR UPDATE SKIP LOCKED on hrrr_fetch_tasks,
newest valid_time first. Accepts the points JSONB directly.
- hrrr_points::process_batch — fetch surface + pressure GRIB2 once
per task (tokio::try_join), decode via the existing wgrib2 plumbing,
then for each requested point pull the cell and UPSERT INTO
hrrr_profiles (conflict on lat/lon/valid_time).
- db::complete_hrrr_task / fail_hrrr_task — status transitions; Elixir
backfill re-enqueues failed rows on next /30-min scan.
Shipping pieces:
- new bin src/bin/hrrr_point_worker.rs
- new module src/hrrr_points.rs (process_batch, upsert_profile)
- new Cargo [[bin]] entry; Dockerfile builds both binaries in one stage
and ships them in the runtime image so a single CI pipeline covers
the whole cluster
- k8s/deployment-hrrr-point-rs.yaml (1 replica, 1 Gi limit, anti-affinity
against prop-grid-rs so chain + point work don't fight for wgrib2
slots). Uses the same image; command: override picks the right binary.
- kustomization.yaml: include the new deployment so flux applies it
- deployment-grid-rs.yaml: bump readiness initialDelaySeconds 3→15 +
failureThreshold 3→6 so a slow DB connect during startup can't race
the first probe
119 Rust tests green.
run_analysis_step chains the Rust-side f00 work end-to-end:
- fetch surface + pressure GRIB2 + native duct in parallel
(tokio::try_join!) with per-leg error-lift to PipelineError
- merge pressure into surface, fold native duct metrics per cell
- NEXRAD composite reflectivity overlay via the IEM n0q PNG path
- commercial-link degradation via the Postgres per-link baseline
- score all 23 bands in parallel (rayon) then write band files
via parallel spawn_blocking (try_join_all)
- write the ProfilesFile as MessagePack alongside the score files
worker.rs main loop now claims kind='analysis' first, falls back to
kind='forecast'. Analysis tasks dispatch to run_analysis_step;
forecast tasks still go through run_chain_step unchanged.
ChainOutcome enum tags log events (kind=analysis|forecast) and
carries analysis-specific counters (profile_cells_written, duct_cells,
nexrad_cells_with_echo, commercial_cells_boosted) so Phase 3 cutover
observability is readable from kubectl logs alone. 118 Rust tests
green. Elixir seeder flip + f00 code deletion lands next.
Rust-side writer for the f00 ProfilesFile. Replaces the Elixir ETF
format (:erlang.term_to_binary) with MessagePack + gzip so Rust can
produce files without porting the Erlang serializer.
Wire layout: top-level map with 'v' (format version), 'valid_time'
(ISO string), and 'cells' (array of {lat, lon, profile}). Profile
values are an rmpv::Value so the writer stays agnostic about what
fields the f00 pipeline decides to include — native-duct metrics,
NEXRAD reflectivity, commercial-link degradation, and the sounding
profile list all serialize through the same path.
All keys are strings — MessagePack has no atom type. Elixir's reader
(follow-up commit) converts known keys to atoms via a whitelist.
Atomic writes via rename(2) mirror ScoresFile/ProfilesFile pattern:
write to tmp sibling, rename into place on the shared NFS mount.
4 unit tests cover round-trip, atomic rename, path layout, and
lat/lon snap.
native_duct.rs is the Rust side of HrrrNativeClient.fetch_native_duct_grid:
fetch native-level HRRR idx → select duct byte ranges (TMP/SPFH/HGT/PRES
× 50 hybrid levels = 200 messages, ~300 MB subset) → wgrib2 decode via
the existing decoder::extract_grid plumbing → per-cell build_native_profile
→ duct::analyze → HashMap<(lat, lon), DuctMetrics>.
build_native_profile keeps the Elixir semantics intact: drop cells that
don't reach 3 levels, sort by ascending height, tolerate missing SPFH/PRES
as zero (the duct math handles zeros gracefully). merge_duct_grid gives
the pipeline.rs caller a drop-in replacement for Elixir's apply_duct_grid.
claim_next_analysis is the analysis-lane companion to claim_next: selects
kind='analysis' rows only, same SKIP LOCKED + newest-first ordering. Kept
on a separate claim function so the forecast lane can keep running
unaffected during cutover; the analysis pipeline wiring lands next.
9 pure unit tests cover message generation, URL builder, profile-sort,
min-level filter, and merge-into-base behaviour. Network-live golden
against a real native GRIB2 deferred to the cutover commit.
Port of the subset of Microwaveprop.Weather.NexradClient used by
f00's chain step: 5-min-rounded IEM n0q PNG fetch, palette-indexed
decode via the png crate, and per-POI max_reflectivity_dbz extraction
from a 25-pixel half-box (matches Elixir's DEFAULT_BOX_HALF).
The palette↔dBZ math and (lat, lon)↔pixel projection are bit-for-bit
copies of the Elixir formulas. 6 pure-math unit tests cover pixel
mapping, box clipping, and timestamp rounding. Network-live golden
fixture deferred until the full Stream A wiring lands.
Port of Microwaveprop.Commercial.build_link_lookup/2 +
link_degradation_from_lookup/3. The Elixir context stays intact
(LiveView sensor + SNMP poller); this Rust side is the read-only
consumer used by the f00 chain step (Phase 3 Stream A).
Three pure-math unit tests cover: haversine distance (DFW→Austin and
same-point zero), degradation_at aggregation behaviour, and
two-decimal rounding convention that Elixir's Float.round(_, 2)
imposes. SQL queries mirror the Elixir Ecto queries one-for-one —
baseline window, current point-in-time read, and MIN_BASELINE_SAMPLES
quality gate.
Pure port of lib/microwaveprop/propagation/duct.ex — the ITU-R P.834
duct detector that computes the modified refractivity M-profile and
finds regions where dM/dh < 0, then uses Bean & Dutton to derive the
minimum trapped frequency per duct.
Part of Stream A (f00 port): compute_duct_metrics and the per-cell
native-level duct merge both build on analyze(). Landing it now with
standalone unit tests lets later tasks (fetch_native_duct_grid, the
wgrib2 subprocess for native levels) treat it as a known-good
dependency instead of a concurrent port.
Three independent improvements in one commit:
1. Parallel band scoring (pipeline.rs):
- rayon par_iter across 23 bands replaces the serial for loop.
- All band files now write via try_join_all over spawn_blocking
instead of serializing one-at-a-time on NFS. Chain per-step
drops from ~60s to ~15-25s on a 4-core box.
2. Prometheus metrics (new metrics.rs):
- axum server on METRICS_ADDR (default :9100) exposes /metrics and
/health. Scraped by existing Prometheus via annotation.
- Histograms: chain step duration (by outcome), decode duration.
- Gauge: tasks in flight (RAII guarded so panics don't leak).
- Counter: chain steps by outcome.
- worker.rs records step duration and wraps each run in an
InFlightGuard.
3. HA + ordering fixes:
- deployment-grid-rs.yaml: replicas 1→2, required podAntiAffinity
spreads them across hosts, nodeAffinity prefers talos5 for one.
PROP_GRID_RS_PARALLELISM 4→3 per pod (6 concurrent across
replicas; smaller secondary-node footprint).
- Readiness probe on /health so rollouts wait for the runtime to
be alive.
- claim_next ordering: run_time ASC → DESC so the newest hourly
run drains before any stragglers from a broken prior run. Within
a run, forecast_hour ASC keeps nearest-first.
- grid_task_enqueuer sets kind="forecast" explicitly and updates
conflict_target to the new 3-column unique index introduced by
migration 20260419222624.
Replaces the serial claim→process loop with a JoinSet of N worker tasks,
each running the same claim→fetch→decode→score→complete flow against
grid_tasks. FOR UPDATE SKIP LOCKED already prevents contention between
workers so no additional coordination is needed.
talos5 is 4c/4t (i5-7260U); PROP_GRID_RS_PARALLELISM=4 turns the full
f01..f18 chain from ~18 min serial into roughly ~5 min. Per-task peak
memory after the streamed-bands refactor is ~250 Mi, so 4 concurrent is
~1 Gi — well under the 2 Gi container limit.
Postgres pool sized to parallelism + 2 so NOTIFY and an opportunistic
claim retry share the pool without blocking a worker transaction.
Introduces TaskKind { Forecast, Analysis } and threads it through the
claim flow. Forecast-only filter on the claim query keeps the claim
lane clean while Elixir still owns f00 — a half-deployed cluster won't
deadlock on an unfulfillable analysis row.
Prep for Stream A of Phase 3: once the Rust-side native-duct + NEXRAD +
commercial pipeline lands, a sibling claim_next_analysis function pulls
kind='analysis' rows and dispatches to the new pipeline.
Previous pipeline held three overlapping large structures in RAM:
1. `merged: HashMap<Key, HashMap<String, f32>>` — 92k outer × ~60 String
keys inner = ~200 MB of fragmented heap
2. `per_band: HashMap<u32, Vec<ScorePoint>>` — all 23 bands × 92k
ScorePoints before the first write
3. A `scores.clone()` per band during the write loop (doubled that
band's footprint transiently)
Consolidate into:
- Consume `merged` into a compact `Vec<(lat, lon, Conditions,
BandInvariants)>`; the huge String-keyed HashMaps drop as soon as
collection finishes.
- Loop over bands: score → write → vector drops at end of iteration.
One band's ~1.8 MB score vector lives at a time.
Peak heap goes from ~1.5 Gi to ~250 Mi on the chain step. 2 Gi limit
stays, but we now have 8× headroom instead of tight-fit.
Elixir's Ecto :utc_datetime maps to Postgres TIMESTAMP WITHOUT TIME
ZONE — Ecto's contract is that the value is UTC, but the column
itself is naive. sqlx's DateTime<Utc> binds/reads as TIMESTAMPTZ, so
the read failed with "mismatched types ... TIMESTAMPTZ is not
compatible with SQL type TIMESTAMP".
Read as NaiveDateTime and reattach UTC at the decode boundary, keeping
the downstream DateTime<Utc> signature. Test bind goes the other
direction — bind .naive_utc() against the TIMESTAMP column.
Previous Dockerfile compiled a 'fn main(){}' placeholder to warm the
dep cache, then was supposed to rebuild with real source — but cargo's
fingerprint wasn't picking up the re-added source after rm -rf, so the
placeholder artefact (~430 KB) was what got COPY'd into runtime.
Replace the stub-then-real dance with a single cache-mounted build.
BuildKit persists /usr/local/cargo/registry + git + /src/target across
runs, so dep changes are the only full-rebuild trigger. Copy the real
binary into /usr/local/bin/worker before the COPY --from= in runtime
because COPY --from= can't read from a cache mount.
wgrib2 isn't in Debian apt on trixie — the main prop Dockerfile builds
it from source in a dedicated stage. Rather than duplicate that 5-min
compile in the Rust image's CI path, copy the already-built
/usr/local/bin/wgrib2 + /usr/local/lib/libg2c.so* out of the published
Elixir image.
Also add the libgfortran5/libaec0/zlib1g/libpng16-16t64/libopenjp2-7
runtime deps that wgrib2 links dynamically against — same set the
Elixir runtime installs.
The previous approach (`docker run -v $(pwd)/rust/prop_grid_rs:/src`)
fails under act_runner because the runner container's pwd isn't a path
the host docker daemon can see — clippy installed, then cargo couldn't
find Cargo.toml.
Move the lint + test gate into the Dockerfile builder stage so the
build context ships over the socket the normal way. If clippy warns or
a test regresses, the image build fails and nothing is pushed.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
