wgrib2 isn't in Debian apt on trixie — the main prop Dockerfile builds
it from source in a dedicated stage. Rather than duplicate that 5-min
compile in the Rust image's CI path, copy the already-built
/usr/local/bin/wgrib2 + /usr/local/lib/libg2c.so* out of the published
Elixir image.
Also add the libgfortran5/libaec0/zlib1g/libpng16-16t64/libopenjp2-7
runtime deps that wgrib2 links dynamically against — same set the
Elixir runtime installs.
The previous approach (`docker run -v $(pwd)/rust/prop_grid_rs:/src`)
fails under act_runner because the runner container's pwd isn't a path
the host docker daemon can see — clippy installed, then cargo couldn't
find Cargo.toml.
Move the lint + test gate into the Dockerfile builder stage so the
build context ships over the socket the normal way. If clippy warns or
a test regresses, the image build fails and nothing is pushed.
Runs `cargo clippy --all-targets -- -D warnings` + `cargo test` inside
a rust:1.94-trixie container, then builds `rust/prop_grid_rs/Dockerfile`
and pushes three tags: the timestamped `main-<ts>-<sha>`, `latest`, and
a static `main` tag that `k8s/deployment-grid-rs.yaml` pins to.
Path-filtered to `rust/prop_grid_rs/**` so Elixir-only pushes don't
trigger a redundant Rust build.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
Oban Pro 2.21.1's Smart engine rejects `local_limit: 0` with
"local_limit must be greater than 0", OOMing the backfill pod on every
boot since the upgrade. Switch the disable pattern to
`[local_limit: 1, paused: true]` so the queue still exists (satisfying
the validator) but processes nothing. Verified under PROP_ROLE=backfill
the four hot queues (propagation, commercial, solar, enqueue) resolve
to paused=true and the others keep their normal concurrency.
Telemetry showed the application-master process holding ~830 MiB of
terms from warm_grid_cache_from_latest_profile — the data lives in
the app master's heap and never GCs because the process is idle.
Running it in a Task.start lets the terms die with the task.
Mark GridCache, MrmsCache, NexradCache, and ScoreCache ETS tables
:compressed. The scored-band-map and HRRR grid data are map-heavy;
compression trims hundreds of MiB at a few percent CPU cost.
Memoise HRRR .idx responses in Microwaveprop.Cache. Published idx
files are immutable for a model run, but the hourly chain re-fetches
the same URL dozens of times across forecast hours. Cuts ~10s per
repeat out of hrrr_fetch_idx.
Force a garbage collect at the end of HrrrFetchWorker.perform to
reclaim the refc binary heap held from GRIB2 ranges before the Oban
producer hands the process its next job.
talos5 is a NUC7i5 with 32GB RAM and a Samsung 980 Pro NVMe. Adding
a 4th hot-path replica lets the anti-affinity rule place one pod
per physical host across node1/2/3/5 and gives the hourly f00-f18
grid chain 4-way parallel fan-out, with the NVMe-backed pod
handling its share of grib extraction.
config.exs registers Oban queues + the Cron plugin at compile time,
and Config deep-merges with runtime.exs by key. Simply omitting
propagation/commercial/solar from runtime.exs left them inheriting
from config.exs, so backfill pods were still processing hot-path
queues and holding the Cron plugin. Override the hot queues to
limit: 0 and re-register Cron with an empty crontab in backfill
mode so the compile-time defaults get neutralised.
PROP_ROLE=backfill drops the hot-path propagation/commercial/solar
queues and the Cron plugin. Paired with a dedicated Deployment
(nodeSelector: prop-backfill=true, 4Gi memory, no web) this lets 8GB
T620 workers drain the hrrr/weather/terrain/iemre/narr backfill
queues without competing with the hourly grid chain.
Two independent wins the latest prod telemetry pointed at:
1. Iowa Environmental Mesonet rate limiter. IEM throttles per source
IP across all in-flight requests, so dropping the `:weather` Oban
queue to 1/pod wasn't enough — workers were still issuing back-to-
back requests inside each job and drawing a steady stream of HTTP
429s (1,396 retryable on the queue). A GenServer-based token bucket
serialises acquire() with a 700ms min gap per pod; three pods give
~4 req/sec cluster-wide, well under the observed 429 threshold.
Wrapped around every IemClient fetch.
2. Parallel HRRR surface + pressure grid fetch. The two products live
in separate wrfsfcf / wrfprsf GRIB files, so their fetch → wgrib2
→ decode pipelines are independent; running them sequentially was
adding ~30s per forecast hour for no reason. Task.async the pair,
merge at the end. Halves per-fh wall time and should unblock some
of the missing-hour cases we saw on the 14:00Z chain.
Flagged contacts were only reachable from the contact detail page;
once the Flag column came off the /contacts index there was no
at-a-glance view of what's been flagged. Add a second table on the
admin contact-edits page (below the pending-edits table) that lists
every flagged contact newest-first with a direct View link. Hidden
entirely when nothing is flagged.
The `!` flag column was a tiny sliver that rarely fired in practice
and was pushing more useful columns offscreen on narrow layouts. The
underlying `flagged_invalid` field stays on the schema for the detail
page; only the index column goes.
Row-level privacy was already correct: private rows only render for
the owner and admins. The header column was still showing for every
viewer, leaving an always-empty slot for anyone without private
contacts of their own. Drop `:private` from the fields list unless
the scope can actually see at least one private row.
When wall-clock is past :30, the hour *after* Now rounded to "0h" and
the Now slot kept its "Now" tag — two adjacent buttons both reading
as the current hour. Anchor the offset math to the Now slot itself so
the label sequence reads "-1h / Now / +1h / +2h" regardless of where
inside the hour the user loaded the page.
The host docker socket wasn't mounted consistently across act_runners,
so which runner picked up the push was the difference between a clean
build and "unable to connect to docker." Switch to a dockerd service
pinned to the same bridge network as the job; the CLI now talks to
it over TCP via DOCKER_HOST=tcp://dockerd:2375. Every runner has a
daemon, every time — no more re-runs for the scheduling lottery.
The 30s poll is enough to ride out a slow daemon startup, but not a
runner where the socket isn't mounted at all. Print DOCKER_HOST, the
socket stat, and the verbose `docker info` error on timeout so the
next flaky run tells us whether it's a startup race or a missing
mount.
Forgejo act_runner mounts the host docker socket into the job
container; the daemon sometimes takes a beat to answer, which has
been surfacing as intermittent "unable to connect to docker" failures
on login. Gate the build on a `docker info` poll (up to 30s) so later
steps don't race the socket, and wrap login + buildx push in a 3-try
exponential-backoff (2s/4s/8s) so a transient registry hiccup doesn't
require a manual workflow re-run.
Three UI fixes:
1. MapLive now ticks every 60s and, when the selected_time is still
the "Now" slot, advances to the newer latest-past-or-now hour as
the clock crosses into it. The URL is patched in place so shared
links stay accurate without polluting history. Manually picking a
specific past/future hour turns tracking off.
2. point_forecast/3 now reads its valid_times list straight from the
on-disk .ntms files (same source the main-map timeline uses), then
consults the cache per-hour for a fast score lookup. The old cache-
or-store branch could leave the click-to-detail sparkline at 13
hours while the bottom timeline showed 14+; both now line up.
3. Point-detail panel moves from bottom-14 → bottom-24 on desktop so
its lower edge clears the forecast-timeline bar (+ max-height tuned
to match), fixing the overlap visible in screenshots.
Four changes sized by measured prod telemetry (83m of spans):
1. propagation queue: 2 → 1 slot per pod. Two concurrent forecast-hour
steps per pod stacked HRRR grid + native duct grid + scored band
map into ~5-6 GiB RSS, OOM-killing every ~15 min. 3-way parallelism
cluster-wide still finishes the chain inside the hourly interval.
2. weather queue: 3 → 1 slot per pod. ASOS backfill was 429-thrashing
IEM (1,296 retryable jobs; logs were nothing but 429 backoffs).
3. PropagationGridWorker: skip native-level duct fetch on f01..f18.
At ~7-11 min/fh and 18 forecast hours, this was the largest single
cost per chain. Forecast hours fall back to
derived[:min_refractivity_gradient] from the pressure-level
profile. f00 still gets full native-level duct analysis.
4. HrrrClient.download_grib_ranges_to_file: parallelize with
Task.async_stream (max_concurrency 8). The file-backed variant was
sequential, dominating native-duct fetch time on the remaining f00
path. ~20s → ~3s per call.
The :score_band span was reading point_count via length(grid_data),
but grid_data is a %{{lat, lon} => profile} map. Every forecast-hour
job was crashing with ArgumentError before any scores landed, leaving
the map without current-hour forecasts. Swap to map_size/1.
Two wins on the hourly propagation pipeline:
1. Parallelize the chain. seed_chain was enqueuing only f00 and
each step self-enqueued f00+1, serializing ~2.5 min × 19
forecast hours into a ~48 min chain. Fan out all 19 jobs at
once — they're genuinely independent (different HRRR URLs,
different output files) — and let queue concurrency (2 slots
× 3 pods = 6 parallel workers) drop wall time to ~10 min.
Side effect: one permanently-failing step no longer takes out
the rest of the chain, so the rescue-chain logic I added last
commit becomes unnecessary. Removed.
The :final cleanup (retain_window + purge) used to run on the
fh=18 transition; with parallel execution we don't know which
step finishes last. Cleanup now relies on the existing
PropagationPruneWorker 15-min cron for time-based pruning.
Stale chain leftovers are already a minor concern with the
15-min prune; if file bloat becomes real, PruneWorker can
gain retain_window later.
2. Hoist band-invariant factors. score_time_of_day, score_sky,
score_wind, score_pressure depend on conditions only, not the
band — but composite_score was recomputing them 17 times per
point. Added Scorer.precompute_band_invariants/1, called once
per point before the bands loop; composite_score uses the
cached values when present and falls back to computing for
any caller that doesn't pre-warm (test harness, path
integrator). Saves ~30% of the scoring inner loop.
Telemetry showed ~66 PropagationGridWorker exceptions per 6h with
55 ArgumentErrors and 11 TimeoutErrors, producing ~13 discarded
chain steps. Each discard broke the chain: subsequent forecast
hours were never enqueued, leaving the score store with huge gaps
(e.g. at 14:11 UTC the earliest available forecast was 18:00,
because f00-f05 all failed somewhere upstream and nothing ran
after them).
Three changes:
1. PropagationGridWorker: on the final attempt, still enqueue
fh+1 even when this step failed. Oban discards the current
job normally — but the rest of the chain keeps running, so
one bad hour doesn't take out the remaining 12-18. The
rescue is factored into a tested public helper.
2. HrrrClient.parse_idx: skip malformed idx lines instead of
raising. NOAA S3 occasionally serves an HTML error page as
the idx body, and the old strict String.to_integer path
raised ArgumentError on the first non-numeric line and took
down the chain step. This is the root cause of the 55
ArgumentErrors.
3. JS renderTimeline: when no forecast hour is at-or-before
wall-clock (all times are future — the gap scenario the
fixes above are designed to prevent), stop labeling the
earliest future slot "Now". Lets the user see honest
"+Nh" offsets instead of a lie on the pill.
Telemetry puts nexrad_decode_png at 1.76 s/call × 37,847 calls/h
— ~18.5 h of CPU across the cluster every real hour, the single
biggest CPU consumer in the system. The map-click path
(fetch_rain_cells) already cached decoded frames by 5-min
bucket, but the per-contact CommonVolumeRadarWorker path
(fetch_decoded_frame) went straight to network + decode on every
call. Backfill means many contacts share a 5-min window, so the
same 66 MB frame was being decoded dozens of times.
Wire fetch_decoded_frame through NexradCache keyed on the
rounded timestamp. Add a 20-entry size cap in NexradCache so
backfill processing contacts in random timestamp order can't
grow the ETS table to hundreds of GB. Each frame is ~66 MB, so
20 = ~1.3 GB worst case, well under typical pod memory.
Expected impact: cuts sustained decode load by an order of
magnitude depending on backfill temporal locality; map-click
path is unchanged.
Every parser already returned nil on malformed input, so the ||
chains fell through to defaults — except parse_time_param, which
happily accepted any well-formed ISO8601 even if it pointed
outside the current HRRR forecast window (e.g. a stale shared
link from last week). Gate it behind the valid-times window and
fall back to the normal 'now' cursor when it's out of range.
Adds explicit coverage: non-numeric band, unknown band MHz,
non-numeric lat/lon/zoom, out-of-range values, malformed time,
ancient time, and all-params-garbage-at-once. Each case mounts
cleanly and lands on the defaults.
The main map page now reads band, selected time, and map
center/zoom from URL params and updates the URL as the user pans,
zooms, picks a band, or scrubs the timeline. Pan/zoom use
push_patch replace-mode so browser history isn't spammed, while
band and time changes push fresh entries.
Also fix the 'Up to date · just now ago' pipeline chip — 'just
now' already reads as an instant so the trailing 'ago' is
redundant. Any older duration keeps it.
The :propagation queue (2 slots) is shared by PropagationGridWorker,
MrmsFetchWorker (every 2 min), and PropagationPruneWorker (every
15 min). Oban dispatches priority-then-FIFO, so a post-deploy
MRMS/prune backlog could delay the hourly chain step until the
siblings drained.
Set PropagationGridWorker priority: 0 explicitly and drop the two
siblings to priority: 5 so new chain steps always jump ahead.
Backfill workers live on separate queues (:hrrr, :weather, :terrain,
:iemre, :narr, :radar, :mechanism) and already don't interact with
the predictor.
- Remove the 7-day outlook strip and daily_outlook_at/3. It was
derived from point_forecast, which is capped at HRRR's 18-hour
horizon, so it never had 7 days of data to show.
- factors_for: fall back to the nearest persisted analysis profile
at or before the requested time. Only f00 hours persist profiles,
so clicking any other hour used to yield an empty Analysis and
factor table. After the recent cursor fix lands 'Now' on a
forecast hour more often, this regressed further.
- available_valid_times/1: cap the forward horizon to 18h from now.
Leftover valid_times from prior cycles used to pile on the
timeline without adding information.
