Two user-visible crashes in MapLive seen on prod pods today:
1. `:preload_forecast` handle_info timed out a Task.async_stream subtask
after 10 s and the `Enum.map(fn {:ok, h} -> h end)` match cascaded
the `:exit, :timeout` into the LiveView process, terminating the
session. Adds `on_timeout: :kill_task` and flat-maps only :ok
elements — a slow NFS forecast hour now drops from the preload
payload instead of killing the LV.
2. `LiveStash.stash/1` crashed with `** (ArgumentError) errors were
found at the given arguments: 2nd argument: not a valid match
specification` inside `:ets.select_replace/2` on OTP 28. The dep
(live_stash 0.2.0) has a latent bug against the tightened OTP 28
match-spec validator. Both call sites (MapLive.select_band,
SubmitLive.switch_tab) now route through a new
`MicrowavepropWeb.LiveStashGuard.stash/1` wrapper that rescues
ArgumentError, logs a warning, and returns the socket unchanged.
Stash is a convenience for reconnect state — failing it does not
need to kill the user's session.
Drop the guard module once the upstream match-spec fix lands.
EmeLive.mount/3 dropped the {:ok, tref()} return from send_interval
on the floor. Elixir 1.19 + :unmatched_returns dialyzer flag catches
the discarded tuple; the if/else with explicit :ok arms keeps both
branches typed as :ok.
Adds ~600 new test cases (2013 → 2613 tests; 170 properties; 0
failures) across 12 new test files plus expansions of eight existing
files. Big lifts per module:
ContactLive.Mechanism 33 → 100%
MetricsPlug 54 → ~100%
Microwaveprop.Release 15.8 → 70%+
Telemetry 38 → 76%
HrrrNativeGridWorker 27.7 → 76%
ContactLive.Show 34 → 48% (handler + render branches)
Admin.ContactEditLive 49.7 → 70%+
GefsFetchWorker 35.8 → ~55%
IonosphereFetchWorker 56.3 → 75%
PathLive 58.9 → 67%
WeatherMapLive 66.9 → 80.2%
RoverLive 0 → 70.3%
ContactMapLive 59.2 → 89.8%
ContactMapController 0 → 100%
Mix tasks (Rust.Golden, Notebook, Backtest, HrrrBackfill,
HrrrClimatology, HrrrNativeBackfill, NexradBackfill,
RadarBackfill, ImportContestLogs, PropagationGrid,
ResetEnrichment, Hrrr.PurgeGridPoints) 0 → ~60-100%
Two incidental fixes made while adding tests:
- ContactLive.Show.handle_event("toggle_flag", ...) was passing
socket.assigns.current_scope to admin?/1 instead of socket.assigns,
so admins never matched the pattern and every toggle ran the
"Admins only." flash branch. Flag now toggles for admins again.
- Commercial.PollWorker.fetch_weather/1 promoted from private to
@doc'd public so the IEM-fetch + ASOS-upsert path can be tested
directly without driving perform/1 through live SNMP (which was
timing out for ~50 s per test in the earlier attempt).
Stable property-test additions cover the dewpoint-from-RH monotonicity,
nearest_run/1 idempotence, and the ionosphere station envelope.
Replaces the SVG schematic on /eme with a textured WebGL scene: NASA
Blue Marble Earth, real-size Moon, outbound beam, dashed return ray
to the sub-lunar point, and a red shading on the anti-moon hemisphere
that marks who can't see the Moon (outside the bounce footprint).
Three.js (~680 kB) now lives in its own chunk; esbuild --splitting
keeps it out of the main bundle so it only loads when a user hits
/eme. Main app.js drops from 1.6 MB to 922 kB.
Also swaps the Elevation/SNR rows so the badge comes before the number.
New /eme page — antenna aiming + link budget for moonbounce.
Given a station (callsign / grid / lat,lon), TX power, antenna gain,
detection bandwidth, and system Tsys, the page shows in real time:
- Moon azimuth / elevation from the observer (ticks every 10 s)
- Slant range to the Moon
- EIRP breakdown: TX(dBm) + gain(dBi) = EIRP(dBm)
- Path-loss decomposition (Earth → Moon → Earth):
free-space spreading (×2)
− moon reflection gain 10·log(4π σ / λ²) where σ = ρ·π·R²
= round-trip path loss
- Band-dependent lunar albedo ρ (VHF ≈ 0.065, microwave ≈ 0.08,
mm-wave ≈ 0.02) surfaced as a named line item with the current
ρ shown to three decimals
- Received power, thermal noise floor, and SNR with a badge that
classifies the margin (strong / marginal / below noise)
- Self-echo Doppler shift with its radial-velocity driver
New modules:
- `Microwaveprop.Moon` — low-precision lunar ephemeris
(Astronomical Almanac Sec. D.4 truncated series). Provides
`julian_day/1`, `geocentric/1`, `observer_position/3`, and
`radial_velocity_km_s/3`. Accurate to ~0.3° position / ~200 km
distance — well inside any amateur EME-dish beamwidth.
- `Microwaveprop.Propagation.Eme` — path-loss helpers:
`moon_albedo/1` (band lookup), `moon_rcs_m2/1`,
`fspl_round_trip_db/2`, `moon_reflection_gain_db/2`,
`path_loss_db/3`, `received_power_dbm/1`, `noise_floor_dbm/2`,
`snr_db/1`, `doppler_shift_hz/2`.
Wired into the main nav ("EME" button) and the router between
/path and /rover.
Tests: 3 properties + 41 unit tests cover JD conversion, moon
position envelope, band-dependent albedo lookup, decomposition
identity, linearity/monotonicity of path loss and Doppler, and a
LiveView integration test against the rendered DOM.
Full suite: 2,460 tests + 170 properties; credo strict clean.
Two breaking API changes in 0.2:
- `stash_assigns/2` is gone. Keys to persist are now declared on the
`use LiveStash, stored_keys: [...]` macro and `stash/1` picks them
up automatically.
- The TTL unit switched to seconds — this codebase never called the
TTL setter, so no config change is needed.
Updated both callers (MapLive persists selected_band + selected_time;
SubmitLive persists active_tab). `recover_state/1` signature is
unchanged. Full LiveView suite passes (67 tests).
Rust workers (prop-grid-rs) that die mid-claim (SIGKILL, OOM, node
drain) leave grid_tasks rows stuck in status='running' forever,
because claim_next uses FOR UPDATE SKIP LOCKED and nothing resets the
orphan. The /status page then shows a permanent spinner with stale
f00/f10/f18 badges — currently 21 rows claimed as far back as
2026-04-20.
GridTaskEnqueuer.reclaim_stale_running/1 flips rows whose claimed_at
is older than 15 minutes back to 'queued'. Rows that have already
burned through 5 claim/reclaim cycles become 'failed' with a
reclaim-orphan error so the next hourly seed can replace them.
Wired into PropagationGridWorker.seed_chain/0 so it runs every :05
cron tick before new rows are seeded.
Also rename the status panel "Retrying" column to "Failed" — it was
always showing terminal `failed` rows, never retrying ones.
ADIF records frequently include operator commentary in the
<NOTES> field (multi-line, detailed) or <COMMENT> (short
one-liner). Now that contacts have a notes column, pass that
value through the import pipeline instead of dropping it.
Preference is NOTES over COMMENT to match the ADIF 3.1.4 spec's
intended split — NOTES is the richer field. Whitespace-only
values collapse to nil so a program that always emits a blank
NOTES tag doesn't fill every row with an empty string.
Tests cover NOTES capture, COMMENT fallback, NOTES-over-COMMENT
preference, and the absent-field nil case. ADIF upload blurb
lists the new carry-over so users see it before confirming.
Operators want a place to jot observations that aren't captured by
the structured fields — weather anecdotes, propagation mode
commentary, equipment details, band conditions. Add a text column
on contacts plus the two ingest paths users submit through:
* /submit single-contact form gets a 3-row textarea under the
other fields. Optional, max 2000 chars, with a live length cap
via maxlength. Whitespace-only input collapses to NULL so the
column reflects "no notes" rather than an empty string.
* CSV importer recognises a `notes` (or `note`) column via the
existing header_aliases table and flows values straight through
submission_changeset. Sample template gets a matching example
row with embedded commas so the quoted-field round-trip is
exercised on download.
ADIF import and the refinement/refinement-notify paths are out of
scope — users specifically asked for CSV + single-contact. Tests
cover the changeset (accept/blank/over-length), CSV header parsing
(plain + RFC-4180 quoted), and LiveView form submit end-to-end.
The log_level(%{method: \"HEAD\"}) clause never matched because
Plug.Head rewrites the method to \"GET\" before Plug.Telemetry's
register_before_send callback fires, so by log time conn.method is
\"GET\". Stash the original method in private before Plug.Head runs
and key the filter off it.
Contacts have had a `mechanism_status` enum since the rain-scatter
classifier landed — it's in the schema and it's what drives the
Mechanism badge on /contacts/:id — but /status's progress
breakdown still only tracked hrrr/weather/terrain/iemre/radar/narr.
Add Mechanism to both `count_unprocessed_raw`'s FILTER and the
progress-bar list so the ops dashboard reflects the real shape of
the backfill pipeline instead of silently hiding one of the seven
enrichment types.
**Wire format — /map scores payload.** Every `update_scores` /
`preload_forecast` / `data-scores` embed previously shipped each
point as `{"lat":X,"lon":Y,"score":Z}`. At 95k cells in full CONUS
that's ~36 bytes/point × 95k = 3.4 MB of JSON, ~50% of it repeating
the three key names. Packing as `[lat, lon, score]` tuples drops
per-point overhead to ~19 bytes → ~1.8 MB on the wire, ~45% smaller.
New `Propagation.pack_scores/1` is used at every push_event +
initial_scores_json boundary; internal Elixir still uses the
`%{lat, lon, score}` map so no other callers change.
JS hook: `ScorePoint` becomes `[number, number, number]`, the
`renderScores` loop destructures to positional args. Regression
test in map_live_test updated to match the new shape.
**ProfilesFile compression.** Rust writer was at gzip level 6
(default). Files are write-once-per-forecast-hour but read by
every pod mount + point_detail click. Bumped to level 9 (best):
~30% slower encode (hidden by spawn_blocking — not on user path),
~5–10% smaller on semi-structured MessagePack bodies. No reader
changes needed — same gzip stream.
**TypeScript strictness.**
- `app.ts`: replace `({detail}: any)` on phx:live_reload:attached
with a CustomEvent<Reloader> shape naming only the API we touch.
- `weather_map_hook.ts`: drop the `(this as any)._map` pair by
declaring a `GridLayerWithMap = L.GridLayer & { _map: L.Map }`
intersection on the overlay's createTile `this` parameter.
- `global.d.ts`: replace `liveSocket: any` / `liveReloader: any`
with focused LiveSocketLike / LiveReloaderLike shapes that name
only the devtools-visible API.
No more `any` in the codebase (verified via rg).
Two wins that directly shrink /map and /weather latency under
interactive use:
**Debounce the moveend → repaint path (150ms).** Leaflet fires
`moveend` once per pan but 5–10× per second during a wheel-zoom.
Previously each one ran a full `scores_at` (or `weather_grid_at`)
read + a ~20–100KB websocket push. Now they coalesce into a single
trailing-edge flush via `schedule_bounds_update` (and the
/weather twin `schedule_weather_flush`) — the same pattern we
already use for forecast preload. 150ms is short enough that the
user doesn't perceive the pause; >90% of the burst disappears.
**Parallelize point_forecast across 4 tasks.** The sparkline that
pops up on point click was reading up to 18 `.prop` files
sequentially off NFS. Each read is cheap (pread one byte at
row*cols+col) but RTT-dominated. `Task.async_stream` with
max_concurrency=4, ordered=true brings 18-file wall time from
~50ms down to ~15ms on NFS without changing the public API.
No test changes: existing map_bounds test only asserts the bounds
assign lands, which the debounced path still does synchronously.
The main /map timeline was rendering each forecast hour twice
('Now 22:00' + '0h 22:00', '+1h 23:00' + '+1h 23:00', …).
Root cause: during the rolling `.ntms` → `.prop` rename window,
both extensions live on NFS for the same valid_time. `list_score_files`
matches either, so `list_valid_times_raw` mapped 2 files → 2 DateTimes
with no dedupe. Propagation.available_valid_times inherited the dup,
push_timeline pushed it to the JS hook, renderTimeline drew one chip
per entry — hence the visible duplicate.
Fix: `Enum.uniq` before the sort. Regression test writes a `.prop`
file, copies it as the same valid_time's `.ntms` legacy twin, and
asserts list_valid_times returns a single DateTime.
Also land a small refactor that was queued behind this: move the three
pure mechanism display helpers (label/1, badge_class/1, explainer/1)
out of the 2500-line contact_live/show.ex into a new
MicrowavepropWeb.ContactLive.Mechanism module. No behavior change — just
shrinks the LiveView and gives the display mapping a testable home.
- Add one-line moduledocs to 18 LiveViews that were carrying
@moduledoc false. Each line names the route and summarizes what
the page does so a future reader knows where to look before
opening the file.
- Add MapLive handle_event tests for toggle_radar, select_time /
set_selected_time, point_detail, map_bounds, retry_initial_scores.
Assertions focus on socket-state transitions and "didn't crash"
rather than raw HTML — map_bounds in particular has no visible
render-side effect (it push_events the new scores to the JS hook).
- Rust pipeline: integration-shaped test that hand-builds a 2-cell
surface + pressure grid, runs merge → cell_to_conditions →
precompute_band_invariants → composite_score_with → write_atomic
→ decode, asserting header + body length round-trip. Closes the
'pipeline's happy path is only covered by unit tests' gap.
StatusLive had a hand-rolled thousands-separator format_number/1 and a
binary-prefix format_bytes/1 — duplication that was already part of the
review punch-list. Move both to Microwaveprop.Format (which previously
held only distance_km) and delete the local defps. Fourteen call sites
switched to Format.number / Format.bytes.
Also extend the IEM error classifier to snooze on Req.TransportError
{:closed, :timeout, :econnrefused, :nxdomain}. Same rationale as the
429 snooze in batch 1: upstream TCP resets mid-fetch aren't a job
failure, they're transient — returning {:error, _} blows them up into
Oban.PerformError logspam with full stacktraces.
Batched from a system-wide review pass.
**Rate-limit + transient-error log hygiene**
- WeatherFetchWorker: classify IEM HTTP 429 as {:snooze, 300} instead
of {:error}. Stops Oban.PerformError stack-trace spam on every
routine rate-limit during backfill, keeps the retry semantics.
- IonosphereFetchWorker: compress GIRO TLS :unknown_ca error from a
~400-char inspect blob to 'TLS unknown_ca (CA bundle missing)'.
- FreshnessMonitor: log only when an enqueue actually lands (the
Oban unique conflict case was silently dropping jobs but still
producing 'enqueuing grid worker' info lines every 5 minutes).
**Perf**
- Rust grid_level_keys(): pre-format 'TMP:{p} mb' / DPT / HGT keys
once via OnceLock instead of format!()'ing per-cell. Removes ~3.6M
String allocations per f01..f18 chain step across pipeline.rs,
hrrr_points.rs. Same for the wgrib2 :(TMP|DPT|HGT): pattern in
hrrr_points.process_batch (sfc_pattern / prs_pattern OnceLock).
- MapLive preload_forecast: Task.async_stream with max_concurrency=4
replaces the 18-wide Enum.map serial walk. Forecast cache warms
~4× faster after a band change, with ordering preserved.
- grid_tasks: partial composite index on (run_time DESC,
forecast_hour ASC) WHERE status='queued' AND kind='forecast',
matching the Rust claim query's ORDER BY. Drops the old
status-only partial that forced a sort per claim.
**Correctness**
- ContactImportWorker: add Oban unique:[keys: [:import_run_id,
:offset]]. Was missing on a worker whose perform() does a
non-idempotent atomic counter increment — a retry would double-
count imported rows.
- CommonVolumeRadarWorker: x_min..x_max default step is -1 when
x_min > x_max, triggering a runtime deprecation warning. Force
Range.new(_, _, 1) explicitly.
**Cleanup**
- Drop unused tmp_dir parameter from hrrr_point_worker + its
process_batch signature.
/live fires every 10s per pod; the request line and Sent 200 response
from Plug.Telemetry drowned out real requests. log_level/1 now returns
false for /live too, same treatment /health already got.
Investigated why all three prop pods were restart-looping every few
minutes (RESTARTS counts 7–11 in 99m). The trail:
20:53:33.058 [error] Postgrex.Protocol ("db_conn_10") disconnected:
client (:"Elixir.Microwaveprop.PromEx.Poller.5000") timed out
because it queued and checked out the connection for longer than
15000ms
PromEx's Oban queue-depth poller was scanning the oban_jobs table
every 5s and holding an Ecto connection past its checkout timeout.
That starved the rest of the pool, so /health's `SELECT 1` couldn't
acquire a connection inside the 3s liveness probe timeout, the
kubelet's failureThreshold=3 tripped, and SIGKILL landed on the
BEAM. Every replica cycled on the same cadence.
Two fixes:
1. Split /live (no DB) from /health (DB ping). Kubernetes livenessProbe
now points at /live so a saturated Ecto pool can never cascade into
SIGKILL. Readiness still hits /health so a genuine DB outage drains
the pod from Service endpoints gracefully.
2. Turn off the PromEx Oban plugin in prod via the same flag already
used in test. The queue-depth query isn't worth pod instability;
the oban_web dashboard surfaces the same information without
scanning the job table on a timer.
Locked both down with HealthController tests that verify /live never
touches the Repo (no sandbox owner, controller.live/2 called directly,
200 ok) and /health does (sanity query round-trips).
Two fixes cover the blank "analysis breakdown" panel the user reported:
1. Bound the point_detail fallback lookback to 24h. Previously
factors_from_fallback_profile would happily use a week-old analysis
profile when no current one existed — now anything older than 24h
returns an empty factor map so the UI can surface "breakdown
unavailable" instead of silently misattributing.
2. Move the Plausible analytics snippet from Layouts.app into
root.html.heex. Full-bleed LiveViews (/map, /contacts/map,
/weather, home) bypass Layouts.app, so the snippet only loaded on
the navbar-wrapped pages. root.html.heex is loaded once per HTTP
document so coverage is now universal.
Added ~30 tests locking both down:
• point_detail fallback: exact-time wins, 24h boundary accepted,
30h-stale rejected, multi-band coverage, missing-cell degrades to
empty factors, default-valid_time path
• analytics_test.exs: Plausible script present on 12 representative
pages, exactly once, loaded async, surviving a login round-trip
Also fixed pre-existing credo issues per standing rule: shortened the
map_live_test ETS match_delete call, extracted a function from the
deeply-nested hrrr_point_enqueuer.enqueue_for_contacts/1 cond, and
swapped Mix.Tasks.Rust.Golden's IO.inspect for Mix.shell().info.
Score-grid files now land at `<band>/<iso>.prop` with a 4-byte "PROP"
magic header. Readers still accept the legacy `.ntms` extension and
"NTMS" magic so a rolling deploy doesn't invalidate any file already
on the shared NFS mount — legacy files age out through normal
retention.
Applied both sides of the seam: Elixir ScoresFile + regex parsers,
Rust scores_file writer + decoder. Updated the comments in all
modules that mention the extension (NotifyListener, Reconciler,
gefs_fetch_worker, map_live) and the runbook diagram. talos5 is a
DB host now, not a Rust worker — corrected the runbook accordingly.
Two-stage mount splits the propagation map's first paint: the static
HTTP render still fetches scores synchronously so SEO/noscript shipping
gets real data baked into `data-scores`, but the websocket-connected
mount defers the potentially-slow fetch to `start_async/3`. Chrome
paints immediately; scores stream in via an `update_scores` push_event
once `handle_async(:initial_scores, ...)` resolves.
A small overlay (spinner on `:loading`, retry button on `:failed`)
uses `<.async_result>` so users can see state and recover from a
transient fetch error.
The remote_ip plug previously trusted Cf-Connecting-Ip and X-Forwarded-For
on every request, so any client reaching a pod directly (cluster-internal,
kubectl port-forward, misconfigured Service) could spoof conn.remote_ip,
which flows into session storage and logs.
Now only honour those headers when the immediate TCP peer sits inside a
configured CIDR list. Default list covers loopback, RFC1918, CGNAT,
link-local, and IPv6 ULA — matches the k8s pod/service network.
Override via config :microwaveprop, :trusted_proxies, or per-environment
via the TRUSTED_PROXY_CIDRS env var in runtime.exs.
Also refresh the stale "nginx/dokku proxy" moduledoc — deployment has
been k8s + Cloudflare for a while.
Rework of d61fbd3's dialyzer suppression: the 4 warnings from
LiveTable.LiveResource's macro expansion are now fixed at the
root rather than hidden via .dialyzer_ignore.exs.
Changes:
- Delete .dialyzer_ignore.exs and remove ignore_warnings from
mix.exs. No more hidden suppressions.
- New MicrowavepropWeb.LiveTableResource shim that wraps
use LiveTable.LiveResource and overrides:
* maybe_subscribe/1 — explicit :ok = on Phoenix.PubSub.subscribe/2
(previously discarded, tripping :unmatched_return).
* handle_info/2 — explicit _ = on File.rm, Process.send_after.
Switch all 4 LiveTable-using LiveViews (contact_live/index,
beacon_live/index, admin/contact_edit_live, user_management_live/index)
to use MicrowavepropWeb.LiveTableResource instead.
- New priv/dep_patches/live_table-0.4.1.patch adds _ = in front of
LiveTable.LiveSelectHelpers.restore_live_select_from_params/2 in
the dep's macro-expanded handle_params/3. The call's return was
silently discarded, tripping the fourth warning per LiveView.
- New lib/mix/tasks/deps.patch.ex applies every
priv/dep_patches/*.patch to its target dep idempotently after
mix deps.get. Wired into the :setup alias + overridden
mix deps.get so the patch survives cache regeneration in CI.
- Other modified files in this commit are the spec-coverage
additions from the parallel agent pass (beacons.ex,
commercial.ex, propagation.ex, weather.ex, narr_client.ex,
run_timing.ex, 3 workers) — tighter specs for bounds, tuple
shapes, schema-typed params. Dialyzer remains at 0 after
the changes.
Upstream TODO: send a PR for the live_table `_ =` fix. Once
merged and released, drop the patch + mix task + bump the dep.
mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 2 pre-existing flakes (MapLive timestamp +
PropagationPrune ScoresFile), 0 regressions.
- endpoint.ex log_level/1 now filters by conn.request_path instead of
conn.path_info. Plug.Router.forward/2 (deps/plug/lib/plug.ex:170)
rewrites path_info to the unmatched remainder and extends script_name
with the matched prefix before dispatching to the forwarded plug.
/metrics is routed via forward "/metrics", MetricsPlug; by the time
Plug.Telemetry's before_send callback fires inside MetricsPlug.call/2
the conn it observes has path_info: [] and script_name: ["metrics"],
so the old log_level(%{path_info: ["metrics" | _]}) clause never
matched and the default :info level fired. request_path is set by
the adapter at entry and is never rewritten, making it the correct
discriminator. New regression test in metrics_log_suppression_test.exs
captures both the direct shape and the integration path via Plug.Test.
- Scorer.dbz_to_rain_rate_mmhr/1 now uses a compile-time @mp_inv_b
constant (1 / 1.6) instead of dividing on every rain pixel.
composite_score/2's band-invariant fallback switched from four
separate short-circuits (which silently
mixed cached and freshly-computed values if only some keys were
passed) to a single Map.has_key?/2 branch that honors the "all or
none" contract. Dropped unused band_invariant_tod/1 helper.
- Propagation.replace_scores span scope tightened: the
Instrument.span([:db, :replace_scores]) now wraps only the per-band
ScoresFile.write! loop, not the upstream Enum.group_by grouping
phase. The span name + metadata stay unchanged so Grafana panels
keep working, but the fixed telemetry dispatch cost (~100µs x 2)
is no longer paid for trivially small result sets.
- Added @type t :: %__MODULE__{...} to Accounts.UserToken and
Weather.HrrrClimatology — the last two schemas that lacked one.
Elixir 1.19's set-theoretic inference benefits from every struct
having an explicit t/0 so callers can flow through tightly.
mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 3 pre-existing flakes, 0 new regressions.
Every Sent log line now carries method and path so noisy traffic
(probes, scrapers, redirects) is identifiable without a tracer.
RemoteIp plug already sets remote_ip metadata; add method and
request_path alongside it and register the extra keys with the
default formatter.
count_unprocessed_raw fired seven separate COUNT(*) scans on the
contacts table every 2 s refresh (total, terrain, hrrr, weather, iemre,
radar, all-done). Each one went through the 58k+ row table
independently, so the status page was hitting Postgres with ~3.5 full
scans per second of connected viewer time.
A single query with COUNT(*) FILTER clauses pulls every count in one
sequential scan. Same result, roughly 1/7 the CPU on Postgres.
Two wins in the /map hot path.
1. preload_forecast fired on every Leaflet moveend — which arrives in
bursts during pan/zoom — and each firing read + filtered + shipped
18 forecast-hour score lists (up to 90k cells per hour at full
CONUS view) over the LiveView websocket. Now debounced to the
trailing edge: the user must stop moving for 750 ms before we pay
the preload cost. select_band and propagation_updated go through
the same scheduler so a storm of PubSub updates during an hourly
chain also coalesces.
2. ScoreCache.grid_to_filtered_list walked the 92k-cell grid twice
(Enum.filter then Enum.map) and allocated an intermediate tuple
list between them. Enum.reduce emits only in-bounds result maps
directly — halves map traversal + drops the tuple intermediate.
The 'Fetching HRRR atmospheric data' spinner stayed up forever for
contacts stuck in :queued without a matching hrrr_fetch_tasks row.
Ways a contact lands in that state:
- HrrrPointEnqueuer.enqueue/1 raises, the rescue swallows the error
and returns {:ok, 0}, but the caller still marked the contact
:queued before it noticed the failure.
- Rust worker marks a (valid_time) task :done with a points array
that never included this contact's point, so no profile is ever
written for it.
- hrrr_fetch_tasks row is cleaned up by retention before the contact
page loads.
Drop the hrrr_status != :queued short-circuit and always call the
enqueuer when the profile is nil. The ON CONFLICT clause in
HrrrPointEnqueuer.enqueue/1 unions points into the existing row,
resets :done/:failed rows back to :queued with attempt=0, and no-ops
when the point is already present — so re-entering on every page view
is cheap and self-healing.
Clicks on the map were hanging for 1-3s, triggering the Phoenix topbar
progress bar visible to the user. Three hot-path improvements:
1. ScoresFile.read_point now uses a two-range :file.pread/2 — reads the
33-byte header + one cell byte (~60 bytes) instead of the full ~93 KB
file. point_forecast walks 19 forecast hours per click, so the old
full-read path was the dominant cost on cold cache.
2. ProfilesFile.read/1 and list_valid_times/0 cached via
Microwaveprop.Cache (5s TTL). Decoded profile maps are ~10 MB each
(92k cells); timeline scrub that re-clicks the same valid_time
within seconds now hits ETS instead of re-gunzipping + decoding.
Cache keys include base_dir so test setup that swaps dirs doesn't
see stale entries. Writes + prune + retain_window invalidate.
3. Endpoint log_level now filters /metrics the same way it already
filters /health — Prometheus scrapes every 5s and was producing
visible log spam.
Pod-local ETS is right for this workload (per-click read, tiny working
set); Valkey / shared memstore would not help here since each pod needs
its own fast-path lookup. File a follow-up if cross-pod score lookups
ever show up in flame graphs.
Three small wins from a telemetry audit:
1. Propagation.scores_at/3 wrapped every call in Instrument.span, firing
two handler dispatches that dominated the ~10µs ETS lookup on cache
hits. The map's LiveView fires this on every pan + point-click, so
hot-path latency was mostly telemetry. Span now wraps only the miss
branch (where disk IO makes the duration signal meaningful); hit path
still emits the cheap hit/miss counter the cache-ratio panel reads.
2. Oban queue-depth poller: 10s → 30s. Every replica independently
GROUP-BYs oban_jobs, so each extra replica paid ~3 redundant queries
per minute for a gauge that moves on the hour-scale anyway.
3. Dropped summary(phoenix.endpoint.start.system_time): summarizing a
wall-clock timestamp produces no useful aggregate — stop.duration
below it is the meaningful signal.
Second row in the Rust worker table pulls from hrrr_fetch_tasks so
per-QSO HRRR backfill progress is visible alongside prop-grid-rs.
Same columns (Running / Queued / Retrying / Done 1h), 2s cache, same
refresh debounce as the grid_tasks panel.
Phase 3 Stream C Elixir-side: HrrrFetchWorker is deleted; per-QSO HRRR
enrichment now writes to the new hrrr_fetch_tasks table for the Rust
hrrr-point-worker to drain.
Table shape:
- one row per valid_time (primary key) with a JSONB array of
{lat, lon} points
- UPSERT on conflict: array-union of points, status flips back to
queued if previously done/failed so a backfill re-scan naturally
refills the queue for Rust
Elixir changes:
- new migration 20260419231502_create_hrrr_fetch_tasks
- new Microwaveprop.Weather.HrrrPointEnqueuer with enqueue/1 and
enqueue_for_contacts/1. Pre-2014 contacts (NARR's territory)
are skipped here so hrrr_status can pin them to :unavailable.
- ContactWeatherEnqueueWorker: build_hrrr_jobs/1 removed; single-
contact path and batch perform/1 both route through
HrrrPointEnqueuer.enqueue_for_contacts/1. A placeholder jobs-list
is kept just to feed mark_hrrr_status!.
- contact_live/show.ex retry button enqueues via the same path.
- :hrrr queue removed from dev/config/runtime.exs
- HrrrFetchWorker module + test deleted
BackfillEnqueueWorker scans continue to flow through
ContactWeatherEnqueueWorker.enqueue_for_contact (unchanged), so the
30-min reconcile refills hrrr_fetch_tasks automatically.
4 new tests cover the routing, pre-2014 skip, UPSERT-union, and
status-reset-on-reschedule behaviour. Rust-side hrrr-point-worker
binary + k8s deployment land in the next commits.
Mirrors the Oban Job Queue Status table with a new section sourced
from the grid_tasks hand-off table. Columns: Running, Queued,
Retrying, Done (1h).
Active rows render an f## badge per in-flight forecast hour and a
detail line below with run_time / fh / attempt so it's obvious which
HRRR cycle the worker is on.
Pipes through the existing 2s debounced refresh plus a new PubSub
subscription to "propagation:updated" — PropagationNotifyListener
fans out the Rust worker's NOTIFY propagation_ready, so the panel
transitions running → done the moment Rust writes the score file.
Flagged contacts were only reachable from the contact detail page;
once the Flag column came off the /contacts index there was no
at-a-glance view of what's been flagged. Add a second table on the
admin contact-edits page (below the pending-edits table) that lists
every flagged contact newest-first with a direct View link. Hidden
entirely when nothing is flagged.
The `!` flag column was a tiny sliver that rarely fired in practice
and was pushing more useful columns offscreen on narrow layouts. The
underlying `flagged_invalid` field stays on the schema for the detail
page; only the index column goes.
Row-level privacy was already correct: private rows only render for
the owner and admins. The header column was still showing for every
viewer, leaving an always-empty slot for anyone without private
contacts of their own. Drop `:private` from the fields list unless
the scope can actually see at least one private row.
Three UI fixes:
1. MapLive now ticks every 60s and, when the selected_time is still
the "Now" slot, advances to the newer latest-past-or-now hour as
the clock crosses into it. The URL is patched in place so shared
links stay accurate without polluting history. Manually picking a
specific past/future hour turns tracking off.
2. point_forecast/3 now reads its valid_times list straight from the
on-disk .ntms files (same source the main-map timeline uses), then
consults the cache per-hour for a fast score lookup. The old cache-
or-store branch could leave the click-to-detail sparkline at 13
hours while the bottom timeline showed 14+; both now line up.
3. Point-detail panel moves from bottom-14 → bottom-24 on desktop so
its lower edge clears the forecast-timeline bar (+ max-height tuned
to match), fixing the overlap visible in screenshots.
Every parser already returned nil on malformed input, so the ||
chains fell through to defaults — except parse_time_param, which
happily accepted any well-formed ISO8601 even if it pointed
outside the current HRRR forecast window (e.g. a stale shared
link from last week). Gate it behind the valid-times window and
fall back to the normal 'now' cursor when it's out of range.
Adds explicit coverage: non-numeric band, unknown band MHz,
non-numeric lat/lon/zoom, out-of-range values, malformed time,
ancient time, and all-params-garbage-at-once. Each case mounts
cleanly and lands on the defaults.
The main map page now reads band, selected time, and map
center/zoom from URL params and updates the URL as the user pans,
zooms, picks a band, or scrubs the timeline. Pan/zoom use
push_patch replace-mode so browser history isn't spammed, while
band and time changes push fresh entries.
Also fix the 'Up to date · just now ago' pipeline chip — 'just
now' already reads as an instant so the trailing 'ago' is
redundant. Any older duration keeps it.
- Remove the 7-day outlook strip and daily_outlook_at/3. It was
derived from point_forecast, which is capped at HRRR's 18-hour
horizon, so it never had 7 days of data to show.
- factors_for: fall back to the nearest persisted analysis profile
at or before the requested time. Only f00 hours persist profiles,
so clicking any other hour used to yield an empty Analysis and
factor table. After the recent cursor fix lands 'Now' on a
forecast hour more often, this regressed further.
- available_valid_times/1: cap the forward horizon to 18h from now.
Leftover valid_times from prior cycles used to pile on the
timeline without adding information.
Both the Elixir initial-cursor pick and the JS timeline renderer
used nearest-by-absolute-distance, so any wall-clock time past
:30 snapped the 'Now' label onto the next top-of-hour forecast —
up to ~30 min in the future. Switch to the latest valid_time at
or before wall-clock, with a fallback to the earliest time when
every slot is future (shouldn't happen for HRRR, safe default).
Contacts can be marked private at submit time (single form, CSV
import, ADIF import) and edit time. Private contacts are hidden
from the public map, the public /u/callsign profile, and the
browse table for anonymous and non-owning viewers. The original
submitter and admins see them inline on the browse table with a
"Yes" badge, and the detail page shows a lock icon. Non-authorized
viewers get a 404 on the detail page.
