HPA: prop-grid-rs and hrrr-point-rs were sitting at 4 pods for
~16 min after the hourly chain spike — 10-min scaleDown stabilization
window plus a 1-pod-per-2-min drain policy. Both deployments stayed
visibly over-provisioned even when CPU dropped to 1m/pod.
Tighten scale-down to: 2-min stabilization (long enough that a real
multi-cycle surge doesn't immediately collapse) + Percent: 100 per
60s drain (back to min=1 within 1 minute of stabilization clearing).
Net: spike → scale up to handle queue (unchanged) → back to 1 pod
within ~3 min of spike ending. scaleUp left untouched.
CLAUDE.md updates so this doesn't bite again:
- Document `cargo clippy --all-targets -- -D warnings` as part of
the local Rust precommit ritual. The HRDPS commits in this branch
passed `cargo build` locally but failed CI on lints (manual_range_
contains, iter_kv_map). Adding the recipe under Commands plus a
Rust subsection in Key Conventions so future Rust touches don't
ship clippy regressions.
- Added a Testing note about not anchoring relative-time fixtures on
utc_now-truncated-to-hour — the weather_map_live one-hour-cutoff
test (fix in 2768fc68) was a textbook example: 50% flake rate
driven entirely by which minute of the hour the test ran in.
CLAUDE.md adds HRDPS to Key Data Sources and the Background Job
Queues table with explicit "dormant until Rust ships" framing.
The 2026-04-29 plan gets a Status snapshot at the top: which stages
shipped (0-3 Elixir + 8) vs which are deferred to the Rust port
sessions (4-7, 9). Activation path is one-paragraph: land Rust HRDPS,
add cron entry, widen map bounds, extend FreshnessMonitor.
The /path calculator showed "0 / 9 HRRR points" in production despite
the on-disk profile store being current. Root cause: profile_from_cell/2
treated the cell's :profile key as a wrapper sub-map and called
Map.put_new on it — but the :profile key actually holds the vertical
pressure-level LIST. Every point sample crashed with BadMapError, the
crash propagated as {:exit, _} through Task.async_stream, and the
consumer silently dropped all 9 results.
Fix: stop wrapping. Cells are already flat HrrrProfile-shaped maps;
just stamp lat/lon (from the caller, since cells don't carry their
own coords — those are the map key) and valid_time onto the cell.
Audit + log every other async error path so the next silent failure
isn't invisible:
- PathLive HRRR point lookup
- Propagation.point_forecast per-hour reads
- Viewshed ray crashes
- IemClient ASOS network fetches
- RtmaClient range-download tasks
- Recalibrator factor-vector batches (positive + negative samples)
- MapLive forecast preload tasks
- RoverLive station resolution
LiveViews already had handle_async/3 exit clauses with logging. The
gap was always in Task.async_stream consumers that wrote {:exit, _} -> []
without surfacing the reason.
Add the rule to CLAUDE.md and project memory so this never repeats.
Also fix a pre-existing skewt_svg.ex compiler warning where
@critical_label_min_dy was used before being defined.
rover_map_hook and weather_map_hook share the same viewport-scoped
data flow as the propagation map — they hit the same blank-tile bug
after tab hide or socket reconnect. Add reconnected() + visibilitychange
handlers and document the requirement in CLAUDE.md so future hooks that
push map_bounds don't regress.
node3's NVMe SSD is the new shared-storage home. Mount the whole
/data export from 10.0.15.103 at /data inside the pod (instead of
the old skippy export of /data/srtm at /srtm) so there's room for
future shared stuff — Parquet score files, maybe a DuckDB database,
or whatever the next thing is — without re-exporting a new path.
SRTM tiles now live at /data/srtm rather than /srtm. Update the
srtm_tiles_dir runtime config to match, and fix the two doc
pointers (CLAUDE.md, duckdb plan) that referenced the old path.
Already applied to the cluster and rolled out; pods confirmed
running with tiles visible at /data/srtm.
Adds Microwaveprop.Propagation.AsosNudge: a pure IDW bias-field module
that takes ASOS observations + HRRR profiles and returns re-scored grid
rows for every cell within 250km of a reporting station. Upper-air
fields (min_refractivity_gradient, pwat_mm, hpbl_m, profile, duct
metadata) pass through unchanged so HRRR's signal isn't clobbered.
The old AsosAdjustmentWorker was unwired and buggy — nil'd out ~22% of
the scoring weight and wrote orphan timestamps. Replaced with a slim
worker that queries the latest HRRR valid_time, fetches live ASOS
currents, calls AsosNudge.compute/3, and upserts onto
(lat, lon, valid_time, band_mhz) so nudged values overwrite the HRRR
hour cleanly instead of polluting available_valid_times. After each
upsert it warms ScoreCache and broadcasts propagation:updated so live
/map clients refresh.
Cron hooked up every 10 minutes in config.exs and dev.exs. Also cleaned
up the stale "dev has propagation disabled" note in CLAUDE.md.
13 new AsosNudge unit tests cover: residual computation (co-located,
out-of-grid, nil fields), IDW weighting (single station, far station,
two equidistant stations, nil component handling), upper-air
preservation, and the compute/3 entry point's shape and radius filter.
Drive-by Styler formatting touched a handful of unrelated files from
`mix format`.
