Replaces the serial claim→process loop with a JoinSet of N worker tasks,
each running the same claim→fetch→decode→score→complete flow against
grid_tasks. FOR UPDATE SKIP LOCKED already prevents contention between
workers so no additional coordination is needed.
talos5 is 4c/4t (i5-7260U); PROP_GRID_RS_PARALLELISM=4 turns the full
f01..f18 chain from ~18 min serial into roughly ~5 min. Per-task peak
memory after the streamed-bands refactor is ~250 Mi, so 4 concurrent is
~1 Gi — well under the 2 Gi container limit.
Postgres pool sized to parallelism + 2 so NOTIFY and an opportunistic
claim retry share the pool without blocking a worker transaction.
Introduces TaskKind { Forecast, Analysis } and threads it through the
claim flow. Forecast-only filter on the claim query keeps the claim
lane clean while Elixir still owns f00 — a half-deployed cluster won't
deadlock on an unfulfillable analysis row.
Prep for Stream A of Phase 3: once the Rust-side native-duct + NEXRAD +
commercial pipeline lands, a sibling claim_next_analysis function pulls
kind='analysis' rows and dispatches to the new pipeline.
Prep for Stream A of Phase 3 rust migration: f00's native-duct merge,
NEXRAD, and commercial-link enrichment move to Rust via the same
grid_tasks work queue. kind='analysis' (single row per run) vs
kind='forecast' (18 rows per run) lets one Rust worker dispatch on
payload type without a second table.
Forecast default keeps existing Rust claim_next behaviour unchanged
during cutover.
MrmsFetchWorker fired every 2 min on the :propagation queue, decoding an
MRMS PrecipRate GRIB2 into MrmsCache. Its only consumer,
AsosAdjustmentWorker, is disabled in all configs (dev / config / runtime).
Net effect: 30 GRIB2 decodes/hour of dead work on hot pods.
Removed:
- MrmsFetchWorker, MrmsClient, MrmsCache and their tests
- cron entries in config.exs, dev.exs, runtime.exs
- mrms_req_options stub in test.exs
- MrmsCache from supervision tree in application.ex
- stale MRMS references in PropagationGridWorker docstring and test
Also adds docs/plans/2026-04-19-rust-migration-phase3.md which tracks
the broader Stream A (f00 port) and Stream C (HrrrFetchWorker port)
follow-on work.
Mirrors the Oban Job Queue Status table with a new section sourced
from the grid_tasks hand-off table. Columns: Running, Queued,
Retrying, Done (1h).
Active rows render an f## badge per in-flight forecast hour and a
detail line below with run_time / fh / attempt so it's obvious which
HRRR cycle the worker is on.
Pipes through the existing 2s debounced refresh plus a new PubSub
subscription to "propagation:updated" — PropagationNotifyListener
fans out the Rust worker's NOTIFY propagation_ready, so the panel
transitions running → done the moment Rust writes the score file.
Both the hot (`prop`) Deployment and the `prop-backfill` Deployment
labelled their pods with `app: prop` so both joined the same libcluster
Erlang cluster. The `prop` Service selector was `app: prop` too — which
meant the backfill pod (PHX_SERVER=false, no listener on :5000) was in
the Service endpoint pool.
With sessionAffinity: ClientIP, Cloudflared's source IP got hashed to
the backfill endpoint after a backfill pod roll and every request
from that client returned 502 (connection refused on :5000).
Fix: add `tier: hot` to hot-pod template labels; narrow the Service
selector to `{app: prop, tier: hot}`. Backfill keeps `tier: backfill`
and is cleanly excluded. libcluster selector (still `app: prop`) is
unchanged so backfill stays in the Erlang cluster for Oban queue
leader election and Oban.Pro's Smart engine.
Rust `prop-grid-rs` has been validated end-to-end on talos5 on the
streamed-bands image (main-1776635096-6d91461): real chain steps
complete in ~7 s each and the peak-heap refactor landed.
Changes:
- `PropagationGridWorker.seed_chain` now enqueues a single f00 Oban
job instead of f00–f18. The f00 analysis-hour keeps its enrichment
(native-level duct, NEXRAD composite, commercial-link boost,
ProfilesFile write) and the GridCache broadcast.
- `GridTaskEnqueuer.seed/1` is called again so grid_tasks gets the
18 forecast-hour rows the Rust worker claims.
- Test asserts only one Oban job is enqueued and the matching 18
grid_tasks rows land in the DB.
- `deployment-grid-rs.yaml` flips `PROP_SCORES_DIR` from
`/data/scores_shadow` to `/data/scores` so Rust writes to the live
score tree. No file collision — Elixir only writes the f00 files.
The hot pod memory shrink (6 Gi → 1 Gi) is deferred until one full
hourly cycle runs clean under the new split.
Previous pipeline held three overlapping large structures in RAM:
1. `merged: HashMap<Key, HashMap<String, f32>>` — 92k outer × ~60 String
keys inner = ~200 MB of fragmented heap
2. `per_band: HashMap<u32, Vec<ScorePoint>>` — all 23 bands × 92k
ScorePoints before the first write
3. A `scores.clone()` per band during the write loop (doubled that
band's footprint transiently)
Consolidate into:
- Consume `merged` into a compact `Vec<(lat, lon, Conditions,
BandInvariants)>`; the huge String-keyed HashMaps drop as soon as
collection finishes.
- Loop over bands: score → write → vector drops at end of iteration.
One band's ~1.8 MB score vector lives at a time.
Peak heap goes from ~1.5 Gi to ~250 Mi on the chain step. 2 Gi limit
stays, but we now have 8× headroom instead of tight-fit.
1 Gi OOM'd during a real chain step. Each forecast hour loads both a
surface and pressure-level HRRR GRIB2 (~40 MB combined), decodes via
wgrib2 subprocess (which allocates its own buffers), scores 92k grid
points × 23 bands (~2.1M intermediates), and writes 23 score files.
2 Gi absorbs the spike; still ~3× smaller than the Elixir-per-pod
budget we're replacing.
The real worker binary OOM-killed at 512 Mi on startup. The plan's
512 Mi assumption was speculative — actual Rust RSS at boot includes
sqlx pool buffers, reqwest connection pool, tokio runtime, and
tracing JSON subscriber. 1 Gi leaves room for all of that plus the
steady-state ~500 Mi during a chain step.
Elixir's Ecto :utc_datetime maps to Postgres TIMESTAMP WITHOUT TIME
ZONE — Ecto's contract is that the value is UTC, but the column
itself is naive. sqlx's DateTime<Utc> binds/reads as TIMESTAMPTZ, so
the read failed with "mismatched types ... TIMESTAMPTZ is not
compatible with SQL type TIMESTAMP".
Read as NaiveDateTime and reattach UTC at the decode boundary, keeping
the downstream DateTime<Utc> signature. Test bind goes the other
direction — bind .naive_utc() against the TIMESTAMP column.
Mirrors the Elixir wiring:
- ImageRepository + ImagePolicy scan git.mcintire.me/graham/prop-grid-rs
for main-<ts>-<sha> tags and pick the highest timestamp.
- ImageUpdateAutomation already covers ./k8s via the Setters strategy,
so the new setter marker on deployment-grid-rs.yaml means flux will
rewrite it on every CI push and commit the update back to main.
Option A from the shadow-mode duplication discussion — Elixir still
owns f00-f18 fan-out via Oban and writes to /data/scores. The Rust
worker sits idle polling an empty grid_tasks table instead of
re-fetching the same HRRR GRIB2 and writing a parallel tree to
/data/scores_shadow.
Brings the seed call back when Phase 2 cutover is approved (at which
point the Elixir fan-out shrinks to fh=0 and PROP_SCORES_DIR flips
from /data/scores_shadow to /data/scores).
Previous Dockerfile compiled a 'fn main(){}' placeholder to warm the
dep cache, then was supposed to rebuild with real source — but cargo's
fingerprint wasn't picking up the re-added source after rm -rf, so the
placeholder artefact (~430 KB) was what got COPY'd into runtime.
Replace the stub-then-real dance with a single cache-mounted build.
BuildKit persists /usr/local/cargo/registry + git + /src/target across
runs, so dep changes are the only full-rebuild trigger. Copy the real
binary into /usr/local/bin/worker before the COPY --from= in runtime
because COPY --from= can't read from a cache mount.
wgrib2 isn't in Debian apt on trixie — the main prop Dockerfile builds
it from source in a dedicated stage. Rather than duplicate that 5-min
compile in the Rust image's CI path, copy the already-built
/usr/local/bin/wgrib2 + /usr/local/lib/libg2c.so* out of the published
Elixir image.
Also add the libgfortran5/libaec0/zlib1g/libpng16-16t64/libopenjp2-7
runtime deps that wgrib2 links dynamically against — same set the
Elixir runtime installs.
The previous approach (`docker run -v $(pwd)/rust/prop_grid_rs:/src`)
fails under act_runner because the runner container's pwd isn't a path
the host docker daemon can see — clippy installed, then cargo couldn't
find Cargo.toml.
Move the lint + test gate into the Dockerfile builder stage so the
build context ships over the socket the normal way. If clippy warns or
a test regresses, the image build fails and nothing is pushed.
Runs `cargo clippy --all-targets -- -D warnings` + `cargo test` inside
a rust:1.94-trixie container, then builds `rust/prop_grid_rs/Dockerfile`
and pushes three tags: the timestamped `main-<ts>-<sha>`, `latest`, and
a static `main` tag that `k8s/deployment-grid-rs.yaml` pins to.
Path-filtered to `rust/prop_grid_rs/**` so Elixir-only pushes don't
trigger a redundant Rust build.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
Oban Pro 2.21.1's Smart engine rejects `local_limit: 0` with
"local_limit must be greater than 0", OOMing the backfill pod on every
boot since the upgrade. Switch the disable pattern to
`[local_limit: 1, paused: true]` so the queue still exists (satisfying
the validator) but processes nothing. Verified under PROP_ROLE=backfill
the four hot queues (propagation, commercial, solar, enqueue) resolve
to paused=true and the others keep their normal concurrency.
Telemetry showed the application-master process holding ~830 MiB of
terms from warm_grid_cache_from_latest_profile — the data lives in
the app master's heap and never GCs because the process is idle.
Running it in a Task.start lets the terms die with the task.
Mark GridCache, MrmsCache, NexradCache, and ScoreCache ETS tables
:compressed. The scored-band-map and HRRR grid data are map-heavy;
compression trims hundreds of MiB at a few percent CPU cost.
Memoise HRRR .idx responses in Microwaveprop.Cache. Published idx
files are immutable for a model run, but the hourly chain re-fetches
the same URL dozens of times across forecast hours. Cuts ~10s per
repeat out of hrrr_fetch_idx.
Force a garbage collect at the end of HrrrFetchWorker.perform to
reclaim the refc binary heap held from GRIB2 ranges before the Oban
producer hands the process its next job.
talos5 is a NUC7i5 with 32GB RAM and a Samsung 980 Pro NVMe. Adding
a 4th hot-path replica lets the anti-affinity rule place one pod
per physical host across node1/2/3/5 and gives the hourly f00-f18
grid chain 4-way parallel fan-out, with the NVMe-backed pod
handling its share of grib extraction.
config.exs registers Oban queues + the Cron plugin at compile time,
and Config deep-merges with runtime.exs by key. Simply omitting
propagation/commercial/solar from runtime.exs left them inheriting
from config.exs, so backfill pods were still processing hot-path
queues and holding the Cron plugin. Override the hot queues to
limit: 0 and re-register Cron with an empty crontab in backfill
mode so the compile-time defaults get neutralised.
PROP_ROLE=backfill drops the hot-path propagation/commercial/solar
queues and the Cron plugin. Paired with a dedicated Deployment
(nodeSelector: prop-backfill=true, 4Gi memory, no web) this lets 8GB
T620 workers drain the hrrr/weather/terrain/iemre/narr backfill
queues without competing with the hourly grid chain.
Two independent wins the latest prod telemetry pointed at:
1. Iowa Environmental Mesonet rate limiter. IEM throttles per source
IP across all in-flight requests, so dropping the `:weather` Oban
queue to 1/pod wasn't enough — workers were still issuing back-to-
back requests inside each job and drawing a steady stream of HTTP
429s (1,396 retryable on the queue). A GenServer-based token bucket
serialises acquire() with a 700ms min gap per pod; three pods give
~4 req/sec cluster-wide, well under the observed 429 threshold.
Wrapped around every IemClient fetch.
2. Parallel HRRR surface + pressure grid fetch. The two products live
in separate wrfsfcf / wrfprsf GRIB files, so their fetch → wgrib2
→ decode pipelines are independent; running them sequentially was
adding ~30s per forecast hour for no reason. Task.async the pair,
merge at the end. Halves per-fh wall time and should unblock some
of the missing-hour cases we saw on the 14:00Z chain.
Flagged contacts were only reachable from the contact detail page;
once the Flag column came off the /contacts index there was no
at-a-glance view of what's been flagged. Add a second table on the
admin contact-edits page (below the pending-edits table) that lists
every flagged contact newest-first with a direct View link. Hidden
entirely when nothing is flagged.
