- SwpcClient: unparseable time_tag rows drop silently, integer / string
numerics both cast, already-decoded list bodies (Req auto-decode) are
accepted, non-JSON-array bodies return {:error, :not_a_list}, X-ray
rows whose energy isn't the long wavelength are filtered, and the
misspelled-upstream electron_contaminaton key flags
electron_contaminated.
- UwyoSoundingClient: every month abbreviation routes to its correct
number, bogus month tokens fall through to an empty result, and
short-of-fixed-width lines are skipped without raising.
- Grid.wgrib2_grid_spec/0: lon/lat_start anchor to the SW corner,
steps match Grid.step/0, cell counts span CONUS inclusive, and the
spec's cell product equals length(Grid.conus_points/0).
- PollWorker: empty-link list is a no-op, and poll_fn receives host /
community / radio_type straight from the link row.
- Contact submission_changeset: antenna heights are bounded to
[0, 1000] ft, whitespace-only mode normalises without firing an
invalid-mode error, plus a pair of StreamData properties — every
sanctioned band validates and every out-of-list band string is
rejected.
Suite: 2,380 tests + 148 properties (was 2,370 + 146); coverage
70.38% → 71.08% total; credo strict clean.
Two breaking API changes in 0.2:
- `stash_assigns/2` is gone. Keys to persist are now declared on the
`use LiveStash, stored_keys: [...]` macro and `stash/1` picks them
up automatically.
- The TTL unit switched to seconds — this codebase never called the
TTL setter, so no config change is needed.
Updated both callers (MapLive persists selected_band + selected_time;
SubmitLive persists active_tab). `recover_state/1` signature is
unchanged. Full LiveView suite passes (67 tests).
Changes under lib/ (source):
- iemre_fetch_worker: replace `if transient_failure?/1` boolean predicate
+ `if/else` with a classifier that pattern-matches the error reason
directly, returning `:transient | :permanent`, and dispatch via
`case`. Three-clause head is clearer than the boolean predicate.
- scores_file.extract_points and nexrad_client.extract_box: force `//1`
step on the row/col comprehensions so an image box whose centre
lands outside the raster collapses to an empty iteration instead
of firing Elixir 1.19's ambiguous-range warning. Adds a regression
test for the right-edge box.
Test coverage added:
- Feature wrappers in Backtest.Features (theta_e_jump, shear_at_top,
duct_thickness, best_duct_freq, duct_usable_for_band,
distance_to_front / parallel_to_front stubs, all_features/0).
- NotifyListener handle_info tolerance for malformed payloads +
unknown messages, plus the PubSub broadcast side effect.
- Viewshed.effective_reach_km across every verdict / diffraction
tier / ducting-score tier combination.
- SnmpClient parse_af60_output unit conversions + unknown-column
handling; parse_snmpget_output OID normalisation and junk-line
tolerance.
- HrrrNativeClient native_level_count, native_variables,
native_messages shape, duct_messages / duct_byte_ranges.
- Changeset coverage for GeomagneticObservation, SolarFluxObservation,
SolarXrayObservation, Ionosphere.Observation, HrrrClimatology, and
Metar5minObservation — lifted each from ~50% / 0% to 100%.
- Property tests: GefsClient.dewpoint_from_rh invariants (Td ≤ T,
monotonic in RH, finite across the operating envelope) + idempotent
nearest_run. NexradClient.pixel_to_dbz monotonicity + bounded
output; latlon_to_pixel round-trip for every grid cell.
Also cleared several unrelated compiler/linter warnings:
- Three private test helpers (create_contact, insert_contact,
current_hour) had `\\ %{}` / `\\ 0` defaults that no caller used.
- profiles_file_test bound `dir` in a pattern match without using it.
Suite: 2,359 tests + 146 properties pass (was 2,300 / 139); coverage
70.38% → 70.89%; credo strict clean.
Pure-function invariants that example tests can't express at scale:
- Weather.round_to_iemre_grid/2, HrrrClient.nearest_hrrr_hour/1,
NexradClient.round_to_5min/1, NarrClient.snap_to_analysis_hour/1:
each is idempotent, its output satisfies a grid/slot-membership
predicate, and lies within a bounded distance of the input.
- NarrClient.in_coverage?/1: coverage_end is an exclusive upper
bound; any timestamp after it is out of coverage.
- IemreFetchWorker.backoff/1: result is in [120, 21_600], monotonic
non-decreasing in attempt, equals 120*2^(attempt-1) pre-cap.
- Contact submission_changeset notes field: every non-blank string
up to 2000 chars validates; any longer string is rejected.
22 new properties (139 → 161 including existing); all 2,300 tests
green, credo clean.
Rust workers (prop-grid-rs) that die mid-claim (SIGKILL, OOM, node
drain) leave grid_tasks rows stuck in status='running' forever,
because claim_next uses FOR UPDATE SKIP LOCKED and nothing resets the
orphan. The /status page then shows a permanent spinner with stale
f00/f10/f18 badges — currently 21 rows claimed as far back as
2026-04-20.
GridTaskEnqueuer.reclaim_stale_running/1 flips rows whose claimed_at
is older than 15 minutes back to 'queued'. Rows that have already
burned through 5 claim/reclaim cycles become 'failed' with a
reclaim-orphan error so the next hourly seed can replace them.
Wired into PropagationGridWorker.seed_chain/0 so it runs every :05
cron tick before new rows are seeded.
Also rename the status panel "Retrying" column to "Failed" — it was
always showing terminal `failed` rows, never retrying ones.
IemreFetchWorker now writes an empty stub observation when IEM returns
a permanent-failure status (404, 422), mirroring the existing {:ok, []}
path. The backfill cron's next tick sees the stub via
has_iemre_observation?/3, generates no job, and mark_status!/3 flips
the contact's iemre_status from :queued to :complete — draining the
51 contacts that have been stuck behind cancelled out-of-grid IEMRE
jobs.
Raise hot pod CPU limit 2 → 3 so BEAM gets 3 schedulers online.
Observed run queues of [2, 10, 0, 0, 0, 0] on the 2-scheduler
configuration during the :05 propagation chain, starving /live and
/health probe handlers and tripping intermittent liveness timeouts.
ADIF records frequently include operator commentary in the
<NOTES> field (multi-line, detailed) or <COMMENT> (short
one-liner). Now that contacts have a notes column, pass that
value through the import pipeline instead of dropping it.
Preference is NOTES over COMMENT to match the ADIF 3.1.4 spec's
intended split — NOTES is the richer field. Whitespace-only
values collapse to nil so a program that always emits a blank
NOTES tag doesn't fill every row with an empty string.
Tests cover NOTES capture, COMMENT fallback, NOTES-over-COMMENT
preference, and the absent-field nil case. ADIF upload blurb
lists the new carry-over so users see it before confirming.
Operators want a place to jot observations that aren't captured by
the structured fields — weather anecdotes, propagation mode
commentary, equipment details, band conditions. Add a text column
on contacts plus the two ingest paths users submit through:
* /submit single-contact form gets a 3-row textarea under the
other fields. Optional, max 2000 chars, with a live length cap
via maxlength. Whitespace-only input collapses to NULL so the
column reflects "no notes" rather than an empty string.
* CSV importer recognises a `notes` (or `note`) column via the
existing header_aliases table and flows values straight through
submission_changeset. Sample template gets a matching example
row with embedded commas so the quoted-field round-trip is
exercised on download.
ADIF import and the refinement/refinement-notify paths are out of
scope — users specifically asked for CSV + single-contact. Tests
cover the changeset (accept/blank/over-length), CSV header parsing
(plain + RFC-4180 quoted), and LiveView form submit end-to-end.
Hot pods were running weather/narr/rtma/nexrad/contact_import
alongside LiveView and the hourly propagation chain. The weather
queue in particular hammers IEM with ASOS backfills; each Req
retry against a 429 stacks an in-worker Process.sleep, and enough
concurrent retries saturated the Ecto pool. Once DB checkouts
started queueing past 15 s the cascade was always the same:
Postgrex client timeouts → Oban.Notifier/Peer 5 s health checks
failing → /health readiness timeout → kubelet flipping the pod
out of the service endpoint set. One hot replica accumulated 11
restarts over 16 h.
Split the shared queue block: anything that's truly per-QSO
enrichment on demand (terrain, iemre, radar, mechanism, gefs,
ionosphere, space_weather, admin, exports, backfill_enqueue) stays
on both tiers, while weather/narr/rtma/nexrad/contact_import run
only on prop-backfill. The hot pods mark those as paused so the
queue entries still exist cluster-wide (Oban Pro's Smart engine
rejects `limit: 0`). Cluster capacity for the backfill queues is
unchanged; they just execute on the dedicated t620.
When the Rust hrrr-point-worker marks a task as `failed` (upstream
`idx HTTP 404` or wgrib2 decode error on old archive formats), the
corresponding contacts sit at `hrrr_status = :queued` until the
generic 3-day stale-queued reconcile flips them. Speed that up: on
every BackfillEnqueue tick, join contacts to `hrrr_fetch_tasks` by
rounded HRRR hour and flip any `:queued` contact whose cycle is
permanently `failed` to `:unavailable` immediately.
Motivation: a handful of 2018-08-04/05 cycles (t20z/t21z/t15z/t17z)
are genuinely missing `wrfsfcf`/`wrfprsf` files on the NOAA S3
archive — other hours on the same dates are fine, so this is an
upstream data gap rather than a structural pre-2019 issue. Users
see an accurate "Partial" marker on those contacts instead of a
three-day `:queued` stall.
Tested: 4 new cases covering rounding, failed-vs-done, and the
`hrrr not in types` skip branch; full suite 2283/0.
podAntiAffinity on app=prop already keeps the backfill pod on its own
physical host, so the dedicated nodeSelector + taint were redundant.
Removing them lets talos4 be used as a normal worker for anything else
the cluster wants to place there. The live taint and label on talos4
were cleared alongside this commit.
The log_level(%{method: \"HEAD\"}) clause never matched because
Plug.Head rewrites the method to \"GET\" before Plug.Telemetry's
register_before_send callback fires, so by log time conn.method is
\"GET\". Stash the original method in private before Plug.Head runs
and key the filter off it.
Per-pod IEM rate limiter was 700ms (~1.4 req/sec); four pods put the
fleet at ~5.7 req/sec, above the observed 429 threshold. Retry storm
(Req exponential backoff up to 32s) piled concurrent sleeps on top of
live workers and tripped Bandit's acceptor + Oban's leader heartbeat.
- Raise default interval to 1500ms (~2.7 req/sec fleet).
- Hot pod /live timeout 3s → 5s (acceptor stalls under retry sleeps).
- prop-backfill exec probe period 30s → 120s, timeout 15s → 30s; the
release CLI round-trips through distributed Erlang and 15s wasn't
enough under load, restart-looping every ~10 min.
hrrr-point-rs was marking fetch tasks 'done' with 0 profiles inserted
for ~13,900 historical contacts. The wgrib2 subprocess resamples HRRR
to the 0.125° CONUS prop grid, but the lookup keyed on the requested
point (rounded to 0.01° by Weather.round_to_hrrr_grid/2) — those keys
never match a 0.125° cell exactly, so every point fell through the
`cell.is_empty()` skip. Snap to the nearest 0.125° cell for lookup,
store at the original (lat, lon) so Weather.has_hrrr_profile?/3's
0.07° search window finds it on the next backfill tick.
TDD: added snap_key unit tests.
The exec probe runs `/app/bin/microwaveprop rpc ':ok'` which spawns a
release CLI and round-trips through distributed Erlang. Under an IEM
429 retry storm (WeatherFetchWorker sleeps 1-16 s between attempts),
the BEAM scheduler stays responsive but probe latency spikes past 5 s
and the kubelet SIGKILLs the pod every ~10 min. 15 s is still well
below the 30 s period so the probe still catches a genuinely wedged
VM inside one cycle.
Three bugs were letting mechanism/terrain/radar contacts ping-pong
between :complete and :queued on every backfill cron:
1. enqueue_for_contact/2 now filters out types that are already
:complete on the contact before building jobs. Previously
mark_status!/3 demoted every non-empty jobs_by_type entry back to
:queued, overwriting the :complete that workers had just set.
2. reconcile_stale_queued/1 now also flips mechanism_status and
radar_status back to :complete when the companion data
(propagation_mechanism / contact_common_volume_radar) already
exists. Drains the existing ~14k row backlog on the first post-
deploy tick.
3. IemClient.parse_iemre_json/1 accepts "" and nil so a 200-with-
empty-body IEMRE response doesn't FunctionClauseError the worker
through four retry attempts.
lgdc.uml.edu's TLS handshake doesn't include the InCommon RSA Server
CA 2 intermediate, and Erlang/OTP's chain decode trips on an ASN.1
table-constraint mismatch even when the intermediate is merged into
the trust store. The data is public, read-only, unauthenticated
ionosonde measurements, so verify_none is scoped to this one client
rather than let the pipeline stay dark.
The in-memory idx cache kept entries around forever: get() returned
None on expiry but left the HashMap entry in place. Each hourly
HRRR cycle adds ~38 new idx URLs (19 forecast hours × surface +
pressure), so the map grew ~4 MiB/day indefinitely on each Rust
replica. Not the main cause of the OOM loop (glibc fragmentation
was; jemalloc landed last commit) but worth sealing so a long-
lived pod's idx cache can't drift toward the 3 GiB ceiling over
weeks.
Evicts on-touch via `get()` — simple, amortized, no background
sweep thread needed.
Pods were OOMKilled at :05 every hour at the 3 GiB cgroup limit.
Idle RSS climbed 750 MiB → 1.5 GiB across pod lifetime as the
hourly f00 analysis cycled surface + pressure + native-duct blobs
(50 MB / 400 MB / 530 MB) simultaneously during decode. glibc
malloc retains dirty pages after freeing those large transient
allocations, so RSS drifted upward with each cycle and the next
hourly peak eventually crossed the limit.
jemalloc's background decay returns pages to the kernel on a
short schedule, so RSS tracks the actual working set. Wired in
at lib.rs so both `worker` and `hrrr_point_worker` inherit it.
Skipped on MSVC targets (Windows CI if we ever add one); dev
builds on macOS still pick it up via the generic target_env
gate.
Hot pods were restart-looping every ~25 minutes on liveness probe
timeouts. Root cause: ScoreCacheReconciler mirrored every .prop file
from NFS into ETS, including 7 days of GEFS Day 2-7 forecasts. With
23 bands × 43 valid_times × ~2 MB per grid = 1.95 GB in the
propagation_score_cache table alone; GC sweeps starved the
scheduler enough that /live dropped its 3 s budget.
The /map UI only ever requests the [now-1h, now+18h] window. Share
that bound as Propagation.hot_cache_window/0 and apply it in the
reconciler's disk-scan path plus NotifyListener's post-warm prune.
Long-horizon GEFS files stay on disk and are still served via the
lazy read_from_disk_and_cache path when requested directly.
Adds ScoreCache.prune_outside_window/2 (inclusive bounds) and
updates the reconciler tests to use relative-to-now timestamps
since hardcoded fixture dates now drift out of window.
**Rust workers exit after 30 consecutive claim_next failures.**
Both `prop-grid-rs` and `hrrr-point-worker` ran a `while` loop that
slept 10s and retried forever on `claim_next` errors. If Postgres
went unreachable, pods would stay alive but useless — no circuit
breaker, no liveness probe, just an infinite stream of error logs.
Now we increment a consecutive-failure counter and exit with an
error once it hits 30 (~5 minutes of DB unavailability). k8s
restart policy then cycles the pod, which both triggers an
operator alert and gives us a fresh sqlx pool on the way back up.
A successful claim (Some or None) resets the counter so one
flaky query doesn't poison it.
**BeaconMonitorsTest: drop the Process.sleep(1100).** The test
relied on wall-clock gaps to disambiguate `inserted_at` values on
the `:utc_datetime` (second-precision) timestamps — one full
second of sleep per run, and on shared CI it could still flake
when both inserts landed in the same second. Replaced with an
explicit `Repo.update_all` that back-dates the first row by one
second. 1.1s faster, deterministic.
Every prop pod restart was dumping a pair of ArgumentError stack
traces from grpcbox_channel / grpcbox_subchannel terminate/3:
:ets.select(:gproc, …)
* 1st argument: the table identifier does not refer to an
existing ETS table
This is a known grpcbox shutdown race: the OTLP/gRPC transport's
terminate callback tries to look itself up in gproc's ETS just
after gproc's supervisor has already torn the table down. Noisy,
not harmful — but one ArgumentError per restart × N pods adds up.
The OTLP/HTTP transport (port 4318 on the same otel-collector
Service) has no gproc/ETS channel machinery, so shutdown is
quiet. Spans are identical — our collector already accepts both.
- `config/runtime.exs`: default `otlp_protocol: :http_protobuf`,
with an `OTEL_EXPORTER_OTLP_PROTOCOL=grpc` escape hatch.
- `k8s/deployment.yaml`: point `OTEL_EXPORTER_OTLP_ENDPOINT` at
:4318 instead of :4317. The Rust workers stay on :4317 — their
native opentelemetry-otlp transport doesn't share the grpcbox
bug.
The decoder's `parse_lola_binary` inserts the same "VAR:LEVEL"
string into ~92k cell maps for every one of ~60 GRIB2 messages in
a chain step — previously a String::clone per cell, yielding
~5.5M heap allocations and frees per hourly run. Switching
`CellValues` from `HashMap<String, f32>` to
`HashMap<Arc<str>, f32>` makes each insertion a single atomic
refcount increment: the string is allocated once per message, and
`Arc::clone` is cheap from there.
Callers were mostly unchanged — `cell.get("...")` still works via
`Arc<str>: Borrow<str>`. Two touch-ups needed: native_duct.rs had
a couple of `cell.get(&format!(...))` lookups that were passing
`&String` (triggering the wrong Borrow impl) — switched to
.as_str(), and the test fixtures' `.insert(format!(...), _)` now
need a `.into()` to coerce into `Arc<str>`. Zero behavior change.
Contacts have had a `mechanism_status` enum since the rain-scatter
classifier landed — it's in the schema and it's what drives the
Mechanism badge on /contacts/:id — but /status's progress
breakdown still only tracked hrrr/weather/terrain/iemre/radar/narr.
Add Mechanism to both `count_unprocessed_raw`'s FILTER and the
progress-bar list so the ops dashboard reflects the real shape of
the backfill pipeline instead of silently hiding one of the seven
enrichment types.
**Wire format — /map scores payload.** Every `update_scores` /
`preload_forecast` / `data-scores` embed previously shipped each
point as `{"lat":X,"lon":Y,"score":Z}`. At 95k cells in full CONUS
that's ~36 bytes/point × 95k = 3.4 MB of JSON, ~50% of it repeating
the three key names. Packing as `[lat, lon, score]` tuples drops
per-point overhead to ~19 bytes → ~1.8 MB on the wire, ~45% smaller.
New `Propagation.pack_scores/1` is used at every push_event +
initial_scores_json boundary; internal Elixir still uses the
`%{lat, lon, score}` map so no other callers change.
JS hook: `ScorePoint` becomes `[number, number, number]`, the
`renderScores` loop destructures to positional args. Regression
test in map_live_test updated to match the new shape.
**ProfilesFile compression.** Rust writer was at gzip level 6
(default). Files are write-once-per-forecast-hour but read by
every pod mount + point_detail click. Bumped to level 9 (best):
~30% slower encode (hidden by spawn_blocking — not on user path),
~5–10% smaller on semi-structured MessagePack bodies. No reader
changes needed — same gzip stream.
**TypeScript strictness.**
- `app.ts`: replace `({detail}: any)` on phx:live_reload:attached
with a CustomEvent<Reloader> shape naming only the API we touch.
- `weather_map_hook.ts`: drop the `(this as any)._map` pair by
declaring a `GridLayerWithMap = L.GridLayer & { _map: L.Map }`
intersection on the overlay's createTile `this` parameter.
- `global.d.ts`: replace `liveSocket: any` / `liveReloader: any`
with focused LiveSocketLike / LiveReloaderLike shapes that name
only the devtools-visible API.
No more `any` in the codebase (verified via rg).
Two wins that directly shrink /map and /weather latency under
interactive use:
**Debounce the moveend → repaint path (150ms).** Leaflet fires
`moveend` once per pan but 5–10× per second during a wheel-zoom.
Previously each one ran a full `scores_at` (or `weather_grid_at`)
read + a ~20–100KB websocket push. Now they coalesce into a single
trailing-edge flush via `schedule_bounds_update` (and the
/weather twin `schedule_weather_flush`) — the same pattern we
already use for forecast preload. 150ms is short enough that the
user doesn't perceive the pause; >90% of the burst disappears.
**Parallelize point_forecast across 4 tasks.** The sparkline that
pops up on point click was reading up to 18 `.prop` files
sequentially off NFS. Each read is cheap (pread one byte at
row*cols+col) but RTT-dominated. `Task.async_stream` with
max_concurrency=4, ordered=true brings 18-file wall time from
~50ms down to ~15ms on NFS without changing the public API.
No test changes: existing map_bounds test only asserts the bounds
assign lands, which the debounced path still does synchronously.
The main /map timeline was rendering each forecast hour twice
('Now 22:00' + '0h 22:00', '+1h 23:00' + '+1h 23:00', …).
Root cause: during the rolling `.ntms` → `.prop` rename window,
both extensions live on NFS for the same valid_time. `list_score_files`
matches either, so `list_valid_times_raw` mapped 2 files → 2 DateTimes
with no dedupe. Propagation.available_valid_times inherited the dup,
push_timeline pushed it to the JS hook, renderTimeline drew one chip
per entry — hence the visible duplicate.
Fix: `Enum.uniq` before the sort. Regression test writes a `.prop`
file, copies it as the same valid_time's `.ntms` legacy twin, and
asserts list_valid_times returns a single DateTime.
Also land a small refactor that was queued behind this: move the three
pure mechanism display helpers (label/1, badge_class/1, explainer/1)
out of the 2500-line contact_live/show.ex into a new
MicrowavepropWeb.ContactLive.Mechanism module. No behavior change — just
shrinks the LiveView and gives the display mapping a testable home.
