Three additions that bring the diagram closer to the SPC sounding
viewer's full-information chart:
* Wet-bulb temperature trace, plotted as a thinner blue line under
the dewpoint trace. Comes from a Stipanuk-style bracket between
the level's dewpoint and dry-bulb temperatures (already used by
`SkewtParams` for the wet-bulb-zero level — now public so the
renderer can call it per level).
* Environment virtual-temperature trace and parcel virtual-T
trajectory, both as thin pinkish dashed lines. The parcel
virtual-T is the actual buoyancy curve CAPE integrates against,
so showing it next to the dry-bulb parcel ascent makes the
moisture correction visible. New `SkewtParams.virtual_temp_c/2`
and `/3` helpers, plus `SkewtParams.mixing_ratio/2`.
* Critical-level rail on the right edge. Each of LCL / LFC / EL /
0 °C / WBZ that resolves on the current profile gets a short
horizontal tick at the right of the plot and a `<label> <p> mb`
tag just outside it — matches the SPC chart's `143 mb (mix)` /
`158 mb (mix)` annotation column. SkewtLive computes the
pressures (and converts the height-keyed 0 °C and WBZ from
SkewtParams back to pressure via `pressure_at_height/2`) before
handing them to `SkewtSvg.render/2`.
Wind-derived overlays (hodograph, wind barbs) remain off — HRRR
profile data is wind-free above 10 m AGL.
mix test: 27 / 27 across the four /skewt-related test files,
2,902 / 2,902 + 221 properties on the full suite. mix credo --strict
clean (one cyclomatic-complexity refactor extracted along the way).
Skewed isotherms, dry adiabats, and mixing-ratio lines were drifting
outside the frame at the corners — most visibly the warm-side
isotherms exiting the right edge near the top of the diagram. The
existing `clip_polyline/1` only filtered with a ±200 px slack, so
segments crossing the boundary still drew past the frame.
Wrap every grid layer + the T/Td/parcel polylines in a
`<g clip-path="url(#skewt-plot-clip)">`. The clipPath defines a
rectangle from (left, top) to (right, bottom). Frame outline and
axis labels stay outside the clip group so they keep rendering at
the box edge unchanged. The hover-cursor crosshair also lives
outside the clip so its readout box can sit flush against the plot
corner without being trimmed.
mix test: 2,902 / 2,902 + 221 properties green. mix credo --strict
clean.
Two changes that bring /skewt visually and analytically into parity
with the NOAA SPC sounding viewer
(https://www.spc.noaa.gov/exper/soundings/):
NOAA-style line set
-------------------
Updated the SkewtSvg style block to match the public SPC chart:
* Temperature trace ........ pure red (#ff0000) at 2.5 px stroke
* Dewpoint trace ........... pure green (#00aa00) at 2.5 px stroke
* Isotherms ................ dashed teal (#14b8a6)
* Dry adiabats ............. solid orange (#f59e0b)
* Moist (saturated) adiabats dashed sky-blue (#38bdf8)
* Mixing-ratio lines ....... dotted darker green (#16a34a), with
labels at 700 mb showing the g/kg
value, drawn only below 600 mb (matches
the SPC chart's lower-band number row)
* Parcel ascent ............ new dashed grey (#475569) line, drawn
whenever SkewtParams.derive returns a
non-empty parcel_trace
The mixing-ratio set is the SPC standard (0.4 / 0.7 / 1 / 2 / 3 / 5 /
8 / 12 / 16 / 20 g/kg) instead of the prior arbitrary set.
SkewtParams: SPC parcel-theory + indices module
-----------------------------------------------
New `Microwaveprop.Weather.SkewtParams` derives the parameter set that
ships in the SPC `*.txt` companion file:
* LCL pressure / temperature / height (Bolton 1980 eq 22)
* LFC pressure / height
* EL pressure / height
* SBCAPE, SBCIN (J/kg, virtual-temperature integration with
surface parcel)
* 3 km CAPE
* Lifted Index (parcel ↔ environment T at 500 mb)
* Freezing level (T = 0 °C height) and Wet-Bulb Zero
* DCAPE (downdraft CAPE — picks the min-θₑ source level in
400–700 mb, descends moist-adiabatically to surface)
* Layer lapse rates: sfc–3 km, 700–500 mb, 850–500 mb
* `parcel_trace` — pressure/temperature trajectory used by
SkewtSvg to draw the dashed parcel-ascent overlay
Wind-derived indices (SRH, BWD, Bunkers motion, SCP, STP, SHIP) are
deliberately *not* produced and the LiveView surfaces a one-line note
explaining why: HRRR persists wind only at 10 m AGL, so per-pressure-
level shear can't be computed from this data source.
LiveView panel restructured into grouped sections (Surface, Convective
parcel, Levels, Lapse rates, Moisture/stability, Refractivity) so the
layout matches the SPC viewer's order. The Levels rows show pressure
*and* height ("872 mb · 1,140 m") for quick cross-reference between
the diagram axis and the readout.
Tests: 8/8 new SkewtParamsTest cases (LCL Bolton check, saturated-air
edge case, full field-set presence, summer profile produces SBCAPE > 0
and LI < 0, capped profile produces SBCAPE = 0 and LI > 0, freezing
level interpolated correctly, sfc–3 km lapse rate, empty profile no
crash). 25/25 across all skewt-related test files. mix test green
(2,902 / 2,902 + 221 properties), mix credo --strict clean, mix
assets.build clean.
Two follow-ups to the earlier /skewt work:
1. Interactive crosshair (matches the SPC-viewer feel from the
reference screenshot). Mousing over the diagram now drops a
horizontal pressure cursor with a top-left readout box showing the
pressure at the cursor's altitude (mb), the height in m and ft,
and the linearly interpolated temperature and dewpoint at that
level. Two coloured dots track the T and Td traces along the
cursor so the user can read the spread visually.
Mechanics: SkewtSvg.geometry/0 exposes the plot geometry (left,
right, top, bottom, p_bottom, p_top, t_min, t_max, skew) as JSON;
SkewtSvg emits an initially-hidden <g class="skewt-cursor"> layer.
A new Skewt hook in assets/js/app.ts inverts pressure_y to recover
pressure from the cursor's viewBox-Y, walks the (descending-pres-
sorted) profile to interp T/Td/height, and updates the elements.
The hook also handles mouseleave to hide the cursor and uses the
SVG's screen-CTM so the math stays correct under any container
aspect-ratio resize.
2. DB fallback for `available_valid_times`/`load_profile`. When the
on-disk `ProfilesFile` has nothing for the location-and-window
(true on dev, transient on prod between chain runs), SkewtLive
now falls back to `HrrrProfileLookup.{list_valid_times_near,
read_point_near}/3` against the `hrrr_profiles` Postgres table.
The lookup uses the same ±0.07° spatial tolerance and the
`(lat, lon, valid_time)` unique index that `Weather.find_nearest_
hrrr/3` already relies on, so the queries are index-only.
Returned shape matches `ProfilesFile.read_point/3` so SkewtLive
reads from either source transparently. NaiveDateTime → UTC
normalisation is centralised in `ensure_utc/1` so the LiveView
sees `%DateTime{time_zone: "Etc/UTC"}` regardless of which side
produced the value.
Tests: 6/6 new HrrrProfileLookupTest cases (list/limit/tolerance,
read nearest cell, nil for far cells, picks closest among matches),
3/3 SkewtLiveTest still green, 2,894 total Elixir tests + 221
properties green via mix test (one pre-existing flaky test in
admin/contact_edit_live re-runs clean). mix credo --strict clean.
mix assets.build clean.
The /skewt page was empty in prod because Phase 2 of the Rust cutover
moved f01..f18 grid scoring to Rust but left profile-file writing only
on the analysis (f00) path. Result: /data/scores/profiles/ accumulated
one file per chain run instead of nineteen, so SkewtLive only ever saw
the analysis hour — and that hour falls outside the 1 h past cutoff
within an hour of the chain finishing, leaving the page empty most of
the time.
Two surgical fixes:
* `pipeline::run_chain_step` now builds `profile_entries` alongside
`prepared` from the merged grid in a single iter pass, then spawns
`profiles_file::write_atomic` on the blocking pool to overlap the
NFS write with the score-band scoring/write fan-out — same pattern
`run_analysis_step` already uses. `ChainStepStats` gains
`profile_cells_written` so the worker log line is symmetric with
the analysis step's existing field and a future failure mode that
drops the profile would show up as a divergence.
* `SkewtLive.available_valid_times/0` widens the past cutoff from
1 h to 3 h. The chain's analysis hour can be ~70 min old by the
time the next chain finishes (and ~2 h on a missed cycle), so the
1 h cutoff was leaving the page blank during normal transients.
18 h forward window unchanged.
Cosmetic: cargo fmt picked up trivial rewrites in decoder/fetcher/
native_duct/hrrr_point_worker that had drifted; rolled in.
Verification: cargo build --release, cargo clippy --lib --bins -D
warnings, cargo test --lib (134/134), mix test
test/microwaveprop_web/live/skewt_live_test.exs (3/3) all green.
Single search bar accepts an address, Maidenhead grid square, or
callsign; resolves it to lat/lon via Geocoder / Maidenhead /
CallsignLocation (cheapest classification first), snaps to the HRRR
grid via the existing ProfilesFile reader, and renders an SVG
Skew-T-Log-P with isobars, skewed isotherms, dry/moist adiabats, and
mixing-ratio lines. A button row picks between every available
forecast hour (now → f18); default is the most recent analysis.
Right pane lists derived stability and refractivity stats from
SoundingParams.derive (PWAT, BL depth, K-index, lifted index, min
dN/dh, ducting status + per-duct base/top/ΔM).
Renderer is server-side SVG so the page works without JS and serializes
into the LiveView payload as a single tag. Wind barbs are deliberately
omitted — HRRR's persisted profile carries wind only at 10 m AGL.
`handle_event("switch_tab", ...)` ran `String.to_existing_atom/1` on
the raw `tab` value from the browser, so a stale, forged, or
mistyped client event with any unknown string crashed the LiveView
with `ArgumentError: not an already existing atom`.
Compares against an explicit `@valid_tabs` allowlist (`:single`,
`:csv`, `:adif`) — known values switch the active tab, anything
else is ignored.
The top navbar in the root layout already linked to /eme, but the
in-page side sidebars on /map and /weather (mobile + desktop) skipped
straight from Path Calculator to Beacons. Adds the EME entry between
them in all four panels so the Earth-Moon-Earth calculator is reachable
without bouncing through the navbar.
Also demotes the LiveStashGuard "stash skipped" warning to debug. It's
the known OTP 28 / LiveStash 0.2.0 ArgumentError that fires on every
event triggering a stash; the guard already handles it, the noise was
adding nothing.
Adds a `layer` query parameter to `/weather`. Loading
`/weather?layer=temp_700mb` boots straight into that overlay; clicking
a different layer pill push_patches the URL so the active selection is
shareable and survives a refresh. Unknown layer ids fall back to the
default (temperature) instead of crashing.
/weather mount called Weather.latest_weather_grid/1 directly, which
runs WeatherLayers.derive over 92k cells and Jason-encodes the
result. Production HTTP logs showed dead renders consistently
landing at 9.8–10.2 s — right at LiveView's socket join deadline.
The browser would render half the page, the WS would fail to mount,
and the user saw "attempting to reconnect".
The JS hook already fires map_bounds via requestAnimationFrame
immediately after mount, and the existing handle_event("map_bounds")
path returns viewport-scoped weather data via the update_weather
push event. So the dead-render data fetch is redundant — drop it.
Mount now returns initial_data_json: "[]"; the layer paints
~50–200 ms after WS connect via the bounds-driven flow.
850 mb T alone shows that warm air is present aloft but doesn't reveal
whether it's the textbook cap structure — a warm 700 mb above an EML
plume. The three new layers expose the canonical capping diagnostics
visually:
* T @ 700 mb — ≥10 °C is the southern Plains "moderate cap" threshold
* Td @ 700 mb — wide T-Td depression at 700 mb signals the EML plume
* Lapse 850→700 — steep mid-layer lapse rate (≥7 °C/km) over a moist
boundary layer is the cap mechanism itself
All three derive purely from the existing pressure-level profile (Rust
already fetches up through 700 mb), so no Rust pipeline changes needed.
The three tables on /status (Job Queue, Rust Workers, Table Sizes)
were collapsing to fit the viewport on narrow phones, wrapping
header text ("Siz…") and splitting values like "160 GB" into two
lines ("160." / "GB"). The overflow-x-auto wrappers were already
present but column auto-sizing still squeezed width below content.
Adds `whitespace-nowrap` to each `<table>` so cells never wrap, and
`-mx-2 px-2` on the overflow wrapper so horizontal scroll extends
edge-to-edge of the mobile viewport instead of being inset by the
layout's outer padding. Tables now scroll horizontally as intended
on mobile; desktop layout unchanged.
The Rust f01..f18 pipeline writes ProfilesFile cells, and the Rust
hrrr_points worker writes hrrr_profiles rows, with profile entries
keyed "pres_mb" / "hght_m" / "tmpc" / "dwpc" (string keys pre-
atomization; atom keys post-ProfilesFile.read). Elixir's
SoundingParams.derive and WeatherLayers.sort_profile filtered on
"pres" / "hght" — so every derived field returned nil for any
Rust-provided profile.
Visible symptoms on /weather: N-gradient, Refractivity, T @ 850mb,
Td @ 850mb, Lapse Rate, Inversion, Inv. Base all rendered no overlay.
For per-contact analysis, min_refractivity_gradient from the Rust
HRRR point worker's rows silently dropped.
- SoundingParams.normalize_profile_entry/1: single-source normalizer
accepting legacy, Rust-string, and Rust-atom shapes; returns the
canonical "pres"/"hght"/"tmpc"/"dwpc"/"drct"/"sknt" shape.
- SoundingParams.derive/1 and WeatherLayers.sort_profile/1 both run
every entry through it before the rest of the existing logic.
- Weather.build_grid_cache_row/4 now falls back to native_min_gradient
when SoundingParams can't derive one (surface-only profiles).
Also:
- Default /weather overlay changed to Temperature (LiveView assign +
JS fallback in the hook's dataset default) per user request.
Two user-visible crashes in MapLive seen on prod pods today:
1. `:preload_forecast` handle_info timed out a Task.async_stream subtask
after 10 s and the `Enum.map(fn {:ok, h} -> h end)` match cascaded
the `:exit, :timeout` into the LiveView process, terminating the
session. Adds `on_timeout: :kill_task` and flat-maps only :ok
elements — a slow NFS forecast hour now drops from the preload
payload instead of killing the LV.
2. `LiveStash.stash/1` crashed with `** (ArgumentError) errors were
found at the given arguments: 2nd argument: not a valid match
specification` inside `:ets.select_replace/2` on OTP 28. The dep
(live_stash 0.2.0) has a latent bug against the tightened OTP 28
match-spec validator. Both call sites (MapLive.select_band,
SubmitLive.switch_tab) now route through a new
`MicrowavepropWeb.LiveStashGuard.stash/1` wrapper that rescues
ArgumentError, logs a warning, and returns the socket unchanged.
Stash is a convenience for reconnect state — failing it does not
need to kill the user's session.
Drop the guard module once the upstream match-spec fix lands.
EmeLive.mount/3 dropped the {:ok, tref()} return from send_interval
on the floor. Elixir 1.19 + :unmatched_returns dialyzer flag catches
the discarded tuple; the if/else with explicit :ok arms keeps both
branches typed as :ok.
Adds ~600 new test cases (2013 → 2613 tests; 170 properties; 0
failures) across 12 new test files plus expansions of eight existing
files. Big lifts per module:
ContactLive.Mechanism 33 → 100%
MetricsPlug 54 → ~100%
Microwaveprop.Release 15.8 → 70%+
Telemetry 38 → 76%
HrrrNativeGridWorker 27.7 → 76%
ContactLive.Show 34 → 48% (handler + render branches)
Admin.ContactEditLive 49.7 → 70%+
GefsFetchWorker 35.8 → ~55%
IonosphereFetchWorker 56.3 → 75%
PathLive 58.9 → 67%
WeatherMapLive 66.9 → 80.2%
RoverLive 0 → 70.3%
ContactMapLive 59.2 → 89.8%
ContactMapController 0 → 100%
Mix tasks (Rust.Golden, Notebook, Backtest, HrrrBackfill,
HrrrClimatology, HrrrNativeBackfill, NexradBackfill,
RadarBackfill, ImportContestLogs, PropagationGrid,
ResetEnrichment, Hrrr.PurgeGridPoints) 0 → ~60-100%
Two incidental fixes made while adding tests:
- ContactLive.Show.handle_event("toggle_flag", ...) was passing
socket.assigns.current_scope to admin?/1 instead of socket.assigns,
so admins never matched the pattern and every toggle ran the
"Admins only." flash branch. Flag now toggles for admins again.
- Commercial.PollWorker.fetch_weather/1 promoted from private to
@doc'd public so the IEM-fetch + ASOS-upsert path can be tested
directly without driving perform/1 through live SNMP (which was
timing out for ~50 s per test in the earlier attempt).
Stable property-test additions cover the dewpoint-from-RH monotonicity,
nearest_run/1 idempotence, and the ionosphere station envelope.
Replaces the SVG schematic on /eme with a textured WebGL scene: NASA
Blue Marble Earth, real-size Moon, outbound beam, dashed return ray
to the sub-lunar point, and a red shading on the anti-moon hemisphere
that marks who can't see the Moon (outside the bounce footprint).
Three.js (~680 kB) now lives in its own chunk; esbuild --splitting
keeps it out of the main bundle so it only loads when a user hits
/eme. Main app.js drops from 1.6 MB to 922 kB.
Also swaps the Elevation/SNR rows so the badge comes before the number.
New /eme page — antenna aiming + link budget for moonbounce.
Given a station (callsign / grid / lat,lon), TX power, antenna gain,
detection bandwidth, and system Tsys, the page shows in real time:
- Moon azimuth / elevation from the observer (ticks every 10 s)
- Slant range to the Moon
- EIRP breakdown: TX(dBm) + gain(dBi) = EIRP(dBm)
- Path-loss decomposition (Earth → Moon → Earth):
free-space spreading (×2)
− moon reflection gain 10·log(4π σ / λ²) where σ = ρ·π·R²
= round-trip path loss
- Band-dependent lunar albedo ρ (VHF ≈ 0.065, microwave ≈ 0.08,
mm-wave ≈ 0.02) surfaced as a named line item with the current
ρ shown to three decimals
- Received power, thermal noise floor, and SNR with a badge that
classifies the margin (strong / marginal / below noise)
- Self-echo Doppler shift with its radial-velocity driver
New modules:
- `Microwaveprop.Moon` — low-precision lunar ephemeris
(Astronomical Almanac Sec. D.4 truncated series). Provides
`julian_day/1`, `geocentric/1`, `observer_position/3`, and
`radial_velocity_km_s/3`. Accurate to ~0.3° position / ~200 km
distance — well inside any amateur EME-dish beamwidth.
- `Microwaveprop.Propagation.Eme` — path-loss helpers:
`moon_albedo/1` (band lookup), `moon_rcs_m2/1`,
`fspl_round_trip_db/2`, `moon_reflection_gain_db/2`,
`path_loss_db/3`, `received_power_dbm/1`, `noise_floor_dbm/2`,
`snr_db/1`, `doppler_shift_hz/2`.
Wired into the main nav ("EME" button) and the router between
/path and /rover.
Tests: 3 properties + 41 unit tests cover JD conversion, moon
position envelope, band-dependent albedo lookup, decomposition
identity, linearity/monotonicity of path loss and Doppler, and a
LiveView integration test against the rendered DOM.
Full suite: 2,460 tests + 170 properties; credo strict clean.
Two breaking API changes in 0.2:
- `stash_assigns/2` is gone. Keys to persist are now declared on the
`use LiveStash, stored_keys: [...]` macro and `stash/1` picks them
up automatically.
- The TTL unit switched to seconds — this codebase never called the
TTL setter, so no config change is needed.
Updated both callers (MapLive persists selected_band + selected_time;
SubmitLive persists active_tab). `recover_state/1` signature is
unchanged. Full LiveView suite passes (67 tests).
Rust workers (prop-grid-rs) that die mid-claim (SIGKILL, OOM, node
drain) leave grid_tasks rows stuck in status='running' forever,
because claim_next uses FOR UPDATE SKIP LOCKED and nothing resets the
orphan. The /status page then shows a permanent spinner with stale
f00/f10/f18 badges — currently 21 rows claimed as far back as
2026-04-20.
GridTaskEnqueuer.reclaim_stale_running/1 flips rows whose claimed_at
is older than 15 minutes back to 'queued'. Rows that have already
burned through 5 claim/reclaim cycles become 'failed' with a
reclaim-orphan error so the next hourly seed can replace them.
Wired into PropagationGridWorker.seed_chain/0 so it runs every :05
cron tick before new rows are seeded.
Also rename the status panel "Retrying" column to "Failed" — it was
always showing terminal `failed` rows, never retrying ones.
ADIF records frequently include operator commentary in the
<NOTES> field (multi-line, detailed) or <COMMENT> (short
one-liner). Now that contacts have a notes column, pass that
value through the import pipeline instead of dropping it.
Preference is NOTES over COMMENT to match the ADIF 3.1.4 spec's
intended split — NOTES is the richer field. Whitespace-only
values collapse to nil so a program that always emits a blank
NOTES tag doesn't fill every row with an empty string.
Tests cover NOTES capture, COMMENT fallback, NOTES-over-COMMENT
preference, and the absent-field nil case. ADIF upload blurb
lists the new carry-over so users see it before confirming.
Operators want a place to jot observations that aren't captured by
the structured fields — weather anecdotes, propagation mode
commentary, equipment details, band conditions. Add a text column
on contacts plus the two ingest paths users submit through:
* /submit single-contact form gets a 3-row textarea under the
other fields. Optional, max 2000 chars, with a live length cap
via maxlength. Whitespace-only input collapses to NULL so the
column reflects "no notes" rather than an empty string.
* CSV importer recognises a `notes` (or `note`) column via the
existing header_aliases table and flows values straight through
submission_changeset. Sample template gets a matching example
row with embedded commas so the quoted-field round-trip is
exercised on download.
ADIF import and the refinement/refinement-notify paths are out of
scope — users specifically asked for CSV + single-contact. Tests
cover the changeset (accept/blank/over-length), CSV header parsing
(plain + RFC-4180 quoted), and LiveView form submit end-to-end.
Contacts have had a `mechanism_status` enum since the rain-scatter
classifier landed — it's in the schema and it's what drives the
Mechanism badge on /contacts/:id — but /status's progress
breakdown still only tracked hrrr/weather/terrain/iemre/radar/narr.
Add Mechanism to both `count_unprocessed_raw`'s FILTER and the
progress-bar list so the ops dashboard reflects the real shape of
the backfill pipeline instead of silently hiding one of the seven
enrichment types.
**Wire format — /map scores payload.** Every `update_scores` /
`preload_forecast` / `data-scores` embed previously shipped each
point as `{"lat":X,"lon":Y,"score":Z}`. At 95k cells in full CONUS
that's ~36 bytes/point × 95k = 3.4 MB of JSON, ~50% of it repeating
the three key names. Packing as `[lat, lon, score]` tuples drops
per-point overhead to ~19 bytes → ~1.8 MB on the wire, ~45% smaller.
New `Propagation.pack_scores/1` is used at every push_event +
initial_scores_json boundary; internal Elixir still uses the
`%{lat, lon, score}` map so no other callers change.
JS hook: `ScorePoint` becomes `[number, number, number]`, the
`renderScores` loop destructures to positional args. Regression
test in map_live_test updated to match the new shape.
**ProfilesFile compression.** Rust writer was at gzip level 6
(default). Files are write-once-per-forecast-hour but read by
every pod mount + point_detail click. Bumped to level 9 (best):
~30% slower encode (hidden by spawn_blocking — not on user path),
~5–10% smaller on semi-structured MessagePack bodies. No reader
changes needed — same gzip stream.
**TypeScript strictness.**
- `app.ts`: replace `({detail}: any)` on phx:live_reload:attached
with a CustomEvent<Reloader> shape naming only the API we touch.
- `weather_map_hook.ts`: drop the `(this as any)._map` pair by
declaring a `GridLayerWithMap = L.GridLayer & { _map: L.Map }`
intersection on the overlay's createTile `this` parameter.
- `global.d.ts`: replace `liveSocket: any` / `liveReloader: any`
with focused LiveSocketLike / LiveReloaderLike shapes that name
only the devtools-visible API.
No more `any` in the codebase (verified via rg).
Two wins that directly shrink /map and /weather latency under
interactive use:
**Debounce the moveend → repaint path (150ms).** Leaflet fires
`moveend` once per pan but 5–10× per second during a wheel-zoom.
Previously each one ran a full `scores_at` (or `weather_grid_at`)
read + a ~20–100KB websocket push. Now they coalesce into a single
trailing-edge flush via `schedule_bounds_update` (and the
/weather twin `schedule_weather_flush`) — the same pattern we
already use for forecast preload. 150ms is short enough that the
user doesn't perceive the pause; >90% of the burst disappears.
**Parallelize point_forecast across 4 tasks.** The sparkline that
pops up on point click was reading up to 18 `.prop` files
sequentially off NFS. Each read is cheap (pread one byte at
row*cols+col) but RTT-dominated. `Task.async_stream` with
max_concurrency=4, ordered=true brings 18-file wall time from
~50ms down to ~15ms on NFS without changing the public API.
No test changes: existing map_bounds test only asserts the bounds
assign lands, which the debounced path still does synchronously.
The main /map timeline was rendering each forecast hour twice
('Now 22:00' + '0h 22:00', '+1h 23:00' + '+1h 23:00', …).
Root cause: during the rolling `.ntms` → `.prop` rename window,
both extensions live on NFS for the same valid_time. `list_score_files`
matches either, so `list_valid_times_raw` mapped 2 files → 2 DateTimes
with no dedupe. Propagation.available_valid_times inherited the dup,
push_timeline pushed it to the JS hook, renderTimeline drew one chip
per entry — hence the visible duplicate.
Fix: `Enum.uniq` before the sort. Regression test writes a `.prop`
file, copies it as the same valid_time's `.ntms` legacy twin, and
asserts list_valid_times returns a single DateTime.
Also land a small refactor that was queued behind this: move the three
pure mechanism display helpers (label/1, badge_class/1, explainer/1)
out of the 2500-line contact_live/show.ex into a new
MicrowavepropWeb.ContactLive.Mechanism module. No behavior change — just
shrinks the LiveView and gives the display mapping a testable home.
- Add one-line moduledocs to 18 LiveViews that were carrying
@moduledoc false. Each line names the route and summarizes what
the page does so a future reader knows where to look before
opening the file.
- Add MapLive handle_event tests for toggle_radar, select_time /
set_selected_time, point_detail, map_bounds, retry_initial_scores.
Assertions focus on socket-state transitions and "didn't crash"
rather than raw HTML — map_bounds in particular has no visible
render-side effect (it push_events the new scores to the JS hook).
- Rust pipeline: integration-shaped test that hand-builds a 2-cell
surface + pressure grid, runs merge → cell_to_conditions →
precompute_band_invariants → composite_score_with → write_atomic
→ decode, asserting header + body length round-trip. Closes the
'pipeline's happy path is only covered by unit tests' gap.
StatusLive had a hand-rolled thousands-separator format_number/1 and a
binary-prefix format_bytes/1 — duplication that was already part of the
review punch-list. Move both to Microwaveprop.Format (which previously
held only distance_km) and delete the local defps. Fourteen call sites
switched to Format.number / Format.bytes.
Also extend the IEM error classifier to snooze on Req.TransportError
{:closed, :timeout, :econnrefused, :nxdomain}. Same rationale as the
429 snooze in batch 1: upstream TCP resets mid-fetch aren't a job
failure, they're transient — returning {:error, _} blows them up into
Oban.PerformError logspam with full stacktraces.
Batched from a system-wide review pass.
**Rate-limit + transient-error log hygiene**
- WeatherFetchWorker: classify IEM HTTP 429 as {:snooze, 300} instead
of {:error}. Stops Oban.PerformError stack-trace spam on every
routine rate-limit during backfill, keeps the retry semantics.
- IonosphereFetchWorker: compress GIRO TLS :unknown_ca error from a
~400-char inspect blob to 'TLS unknown_ca (CA bundle missing)'.
- FreshnessMonitor: log only when an enqueue actually lands (the
Oban unique conflict case was silently dropping jobs but still
producing 'enqueuing grid worker' info lines every 5 minutes).
**Perf**
- Rust grid_level_keys(): pre-format 'TMP:{p} mb' / DPT / HGT keys
once via OnceLock instead of format!()'ing per-cell. Removes ~3.6M
String allocations per f01..f18 chain step across pipeline.rs,
hrrr_points.rs. Same for the wgrib2 :(TMP|DPT|HGT): pattern in
hrrr_points.process_batch (sfc_pattern / prs_pattern OnceLock).
- MapLive preload_forecast: Task.async_stream with max_concurrency=4
replaces the 18-wide Enum.map serial walk. Forecast cache warms
~4× faster after a band change, with ordering preserved.
- grid_tasks: partial composite index on (run_time DESC,
forecast_hour ASC) WHERE status='queued' AND kind='forecast',
matching the Rust claim query's ORDER BY. Drops the old
status-only partial that forced a sort per claim.
**Correctness**
- ContactImportWorker: add Oban unique:[keys: [:import_run_id,
:offset]]. Was missing on a worker whose perform() does a
non-idempotent atomic counter increment — a retry would double-
count imported rows.
- CommonVolumeRadarWorker: x_min..x_max default step is -1 when
x_min > x_max, triggering a runtime deprecation warning. Force
Range.new(_, _, 1) explicitly.
**Cleanup**
- Drop unused tmp_dir parameter from hrrr_point_worker + its
process_batch signature.
Score-grid files now land at `<band>/<iso>.prop` with a 4-byte "PROP"
magic header. Readers still accept the legacy `.ntms` extension and
"NTMS" magic so a rolling deploy doesn't invalidate any file already
on the shared NFS mount — legacy files age out through normal
retention.
Applied both sides of the seam: Elixir ScoresFile + regex parsers,
Rust scores_file writer + decoder. Updated the comments in all
modules that mention the extension (NotifyListener, Reconciler,
gefs_fetch_worker, map_live) and the runbook diagram. talos5 is a
DB host now, not a Rust worker — corrected the runbook accordingly.
Two-stage mount splits the propagation map's first paint: the static
HTTP render still fetches scores synchronously so SEO/noscript shipping
gets real data baked into `data-scores`, but the websocket-connected
mount defers the potentially-slow fetch to `start_async/3`. Chrome
paints immediately; scores stream in via an `update_scores` push_event
once `handle_async(:initial_scores, ...)` resolves.
A small overlay (spinner on `:loading`, retry button on `:failed`)
uses `<.async_result>` so users can see state and recover from a
transient fetch error.
Rework of d61fbd3's dialyzer suppression: the 4 warnings from
LiveTable.LiveResource's macro expansion are now fixed at the
root rather than hidden via .dialyzer_ignore.exs.
Changes:
- Delete .dialyzer_ignore.exs and remove ignore_warnings from
mix.exs. No more hidden suppressions.
- New MicrowavepropWeb.LiveTableResource shim that wraps
use LiveTable.LiveResource and overrides:
* maybe_subscribe/1 — explicit :ok = on Phoenix.PubSub.subscribe/2
(previously discarded, tripping :unmatched_return).
* handle_info/2 — explicit _ = on File.rm, Process.send_after.
Switch all 4 LiveTable-using LiveViews (contact_live/index,
beacon_live/index, admin/contact_edit_live, user_management_live/index)
to use MicrowavepropWeb.LiveTableResource instead.
- New priv/dep_patches/live_table-0.4.1.patch adds _ = in front of
LiveTable.LiveSelectHelpers.restore_live_select_from_params/2 in
the dep's macro-expanded handle_params/3. The call's return was
silently discarded, tripping the fourth warning per LiveView.
- New lib/mix/tasks/deps.patch.ex applies every
priv/dep_patches/*.patch to its target dep idempotently after
mix deps.get. Wired into the :setup alias + overridden
mix deps.get so the patch survives cache regeneration in CI.
- Other modified files in this commit are the spec-coverage
additions from the parallel agent pass (beacons.ex,
commercial.ex, propagation.ex, weather.ex, narr_client.ex,
run_timing.ex, 3 workers) — tighter specs for bounds, tuple
shapes, schema-typed params. Dialyzer remains at 0 after
the changes.
Upstream TODO: send a PR for the live_table `_ =` fix. Once
merged and released, drop the patch + mix task + bump the dep.
mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 2 pre-existing flakes (MapLive timestamp +
PropagationPrune ScoresFile), 0 regressions.
count_unprocessed_raw fired seven separate COUNT(*) scans on the
contacts table every 2 s refresh (total, terrain, hrrr, weather, iemre,
radar, all-done). Each one went through the 58k+ row table
independently, so the status page was hitting Postgres with ~3.5 full
scans per second of connected viewer time.
A single query with COUNT(*) FILTER clauses pulls every count in one
sequential scan. Same result, roughly 1/7 the CPU on Postgres.
Two wins in the /map hot path.
1. preload_forecast fired on every Leaflet moveend — which arrives in
bursts during pan/zoom — and each firing read + filtered + shipped
18 forecast-hour score lists (up to 90k cells per hour at full
CONUS view) over the LiveView websocket. Now debounced to the
trailing edge: the user must stop moving for 750 ms before we pay
the preload cost. select_band and propagation_updated go through
the same scheduler so a storm of PubSub updates during an hourly
chain also coalesces.
2. ScoreCache.grid_to_filtered_list walked the 92k-cell grid twice
(Enum.filter then Enum.map) and allocated an intermediate tuple
list between them. Enum.reduce emits only in-bounds result maps
directly — halves map traversal + drops the tuple intermediate.
The 'Fetching HRRR atmospheric data' spinner stayed up forever for
contacts stuck in :queued without a matching hrrr_fetch_tasks row.
Ways a contact lands in that state:
- HrrrPointEnqueuer.enqueue/1 raises, the rescue swallows the error
and returns {:ok, 0}, but the caller still marked the contact
:queued before it noticed the failure.
- Rust worker marks a (valid_time) task :done with a points array
that never included this contact's point, so no profile is ever
written for it.
- hrrr_fetch_tasks row is cleaned up by retention before the contact
page loads.
Drop the hrrr_status != :queued short-circuit and always call the
enqueuer when the profile is nil. The ON CONFLICT clause in
HrrrPointEnqueuer.enqueue/1 unions points into the existing row,
resets :done/:failed rows back to :queued with attempt=0, and no-ops
when the point is already present — so re-entering on every page view
is cheap and self-healing.
Second row in the Rust worker table pulls from hrrr_fetch_tasks so
per-QSO HRRR backfill progress is visible alongside prop-grid-rs.
Same columns (Running / Queued / Retrying / Done 1h), 2s cache, same
refresh debounce as the grid_tasks panel.
Phase 3 Stream C Elixir-side: HrrrFetchWorker is deleted; per-QSO HRRR
enrichment now writes to the new hrrr_fetch_tasks table for the Rust
hrrr-point-worker to drain.
Table shape:
- one row per valid_time (primary key) with a JSONB array of
{lat, lon} points
- UPSERT on conflict: array-union of points, status flips back to
queued if previously done/failed so a backfill re-scan naturally
refills the queue for Rust
Elixir changes:
- new migration 20260419231502_create_hrrr_fetch_tasks
- new Microwaveprop.Weather.HrrrPointEnqueuer with enqueue/1 and
enqueue_for_contacts/1. Pre-2014 contacts (NARR's territory)
are skipped here so hrrr_status can pin them to :unavailable.
- ContactWeatherEnqueueWorker: build_hrrr_jobs/1 removed; single-
contact path and batch perform/1 both route through
HrrrPointEnqueuer.enqueue_for_contacts/1. A placeholder jobs-list
is kept just to feed mark_hrrr_status!.
- contact_live/show.ex retry button enqueues via the same path.
- :hrrr queue removed from dev/config/runtime.exs
- HrrrFetchWorker module + test deleted
BackfillEnqueueWorker scans continue to flow through
ContactWeatherEnqueueWorker.enqueue_for_contact (unchanged), so the
30-min reconcile refills hrrr_fetch_tasks automatically.
4 new tests cover the routing, pre-2014 skip, UPSERT-union, and
status-reset-on-reschedule behaviour. Rust-side hrrr-point-worker
binary + k8s deployment land in the next commits.
Mirrors the Oban Job Queue Status table with a new section sourced
from the grid_tasks hand-off table. Columns: Running, Queued,
Retrying, Done (1h).
Active rows render an f## badge per in-flight forecast hour and a
detail line below with run_time / fh / attempt so it's obvious which
HRRR cycle the worker is on.
Pipes through the existing 2s debounced refresh plus a new PubSub
subscription to "propagation:updated" — PropagationNotifyListener
fans out the Rust worker's NOTIFY propagation_ready, so the panel
transitions running → done the moment Rust writes the score file.
Flagged contacts were only reachable from the contact detail page;
once the Flag column came off the /contacts index there was no
at-a-glance view of what's been flagged. Add a second table on the
admin contact-edits page (below the pending-edits table) that lists
every flagged contact newest-first with a direct View link. Hidden
entirely when nothing is flagged.
The `!` flag column was a tiny sliver that rarely fired in practice
and was pushing more useful columns offscreen on narrow layouts. The
underlying `flagged_invalid` field stays on the schema for the detail
page; only the index column goes.
Row-level privacy was already correct: private rows only render for
the owner and admins. The header column was still showing for every
viewer, leaving an always-empty slot for anyone without private
contacts of their own. Drop `:private` from the fields list unless
the scope can actually see at least one private row.
Three UI fixes:
1. MapLive now ticks every 60s and, when the selected_time is still
the "Now" slot, advances to the newer latest-past-or-now hour as
the clock crosses into it. The URL is patched in place so shared
links stay accurate without polluting history. Manually picking a
specific past/future hour turns tracking off.
2. point_forecast/3 now reads its valid_times list straight from the
on-disk .ntms files (same source the main-map timeline uses), then
consults the cache per-hour for a fast score lookup. The old cache-
or-store branch could leave the click-to-detail sparkline at 13
hours while the bottom timeline showed 14+; both now line up.
3. Point-detail panel moves from bottom-14 → bottom-24 on desktop so
its lower edge clears the forecast-timeline bar (+ max-height tuned
to match), fixing the overlap visible in screenshots.
Every parser already returned nil on malformed input, so the ||
chains fell through to defaults — except parse_time_param, which
happily accepted any well-formed ISO8601 even if it pointed
outside the current HRRR forecast window (e.g. a stale shared
link from last week). Gate it behind the valid-times window and
fall back to the normal 'now' cursor when it's out of range.
Adds explicit coverage: non-numeric band, unknown band MHz,
non-numeric lat/lon/zoom, out-of-range values, malformed time,
ancient time, and all-params-garbage-at-once. Each case mounts
cleanly and lands on the defaults.
The main map page now reads band, selected time, and map
center/zoom from URL params and updates the URL as the user pans,
zooms, picks a band, or scrubs the timeline. Pan/zoom use
push_patch replace-mode so browser history isn't spammed, while
band and time changes push fresh entries.
Also fix the 'Up to date · just now ago' pipeline chip — 'just
now' already reads as an instant so the trailing 'ago' is
redundant. Any older duration keeps it.
- Remove the 7-day outlook strip and daily_outlook_at/3. It was
derived from point_forecast, which is capped at HRRR's 18-hour
horizon, so it never had 7 days of data to show.
- factors_for: fall back to the nearest persisted analysis profile
at or before the requested time. Only f00 hours persist profiles,
so clicking any other hour used to yield an empty Analysis and
factor table. After the recent cursor fix lands 'Now' on a
forecast hour more often, this regressed further.
- available_valid_times/1: cap the forward horizon to 18h from now.
Leftover valid_times from prior cycles used to pile on the
timeline without adding information.
Both the Elixir initial-cursor pick and the JS timeline renderer
used nearest-by-absolute-distance, so any wall-clock time past
:30 snapped the 'Now' label onto the next top-of-hour forecast —
up to ~30 min in the future. Switch to the latest valid_time at
or before wall-clock, with a fallback to the earliest time when
every slot is future (shouldn't happen for HRRR, safe default).
