Mix.Task.run isn't available in a compiled release, so the in-prod
`bin/microwaveprop eval 'Mix.Tasks.Weather.RebatchAsos.run(...)'`
dispatch crashes. Moves the real logic into a plain module
(`Microwaveprop.Weather.RebatchAsos`) that both the Mix task and a
release eval can call. Uses IO.puts over Mix.shell for the same
reason.
- IemRateLimiter gains AIMD-style adaptive spacing. signal_429/0
widens the current gap (*= 1.5, capped at max_interval_ms default
10s); signal_success/0 narrows it back toward the configured base
(*= 0.92, floored at interval_ms). Self-tunes to IEM's moving
ceiling without needing the manual "safe for 4 pods" constant.
- IemClient now routes every response through a central handle_response
helper that fires the widen/narrow feedback signals, eliminating the
four near-identical case blocks.
- Mix.Tasks.Weather.RebatchAsos collapses any already-queued single-
station "asos" jobs into the batched "asos_batch" shape the
enqueuer now emits, so the pending backfill queue converts to the
new per-request-efficient path instead of draining at the old rate.
Idempotent; supports --dry-run.
2833 tests + credo green.
IEM's ASOS CSV endpoint accepts multiple station= params in a single
request. Prior code enqueued one WeatherFetchWorker job per nearby
station per QSO endpoint — N jobs × 1 station each — which paid the
IemRateLimiter's 1500ms gap and the IEM 429-retry tail N times per
contact.
Changes:
- IemClient.asos_url/3 accepts a list of station codes.
- IemClient.fetch_asos_batch/3 fetches N stations in one call and
returns rows grouped by station_code (with absent codes filled as
empty lists so callers can stub them).
- parse_asos_csv/1 now exposes the station_code column it was
previously discarding.
- WeatherFetchWorker gains an "asos_batch" fetch_type clause that
unpacks rows per (station_id, station_code), upserting or stubbing
each. The single-station "asos" clause stays for already-queued
retryable jobs.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one batch
job per (lat, lon, 4h window) covering every uncovered nearby
station (sorted for deterministic unique-args).
Expected effect on backfill: ~10-20x fewer IEM requests per contact
enrichment cycle, matching drop in 429 retry traffic.
Removes the six OTel packages (opentelemetry, _api, _exporter,
_phoenix, _oban, _bandit) and their 9 transitive deps. Tracing was
only wired for Phoenix/Bandit/Oban auto-instrumentation — PromEx
already covers the metrics we actually use on the status page +
Grafana. Also drops the OTEL_EXPORTER_OTLP_ENDPOINT env var and
its comment block from both deployment manifests.
Adds ~600 new test cases (2013 → 2613 tests; 170 properties; 0
failures) across 12 new test files plus expansions of eight existing
files. Big lifts per module:
ContactLive.Mechanism 33 → 100%
MetricsPlug 54 → ~100%
Microwaveprop.Release 15.8 → 70%+
Telemetry 38 → 76%
HrrrNativeGridWorker 27.7 → 76%
ContactLive.Show 34 → 48% (handler + render branches)
Admin.ContactEditLive 49.7 → 70%+
GefsFetchWorker 35.8 → ~55%
IonosphereFetchWorker 56.3 → 75%
PathLive 58.9 → 67%
WeatherMapLive 66.9 → 80.2%
RoverLive 0 → 70.3%
ContactMapLive 59.2 → 89.8%
ContactMapController 0 → 100%
Mix tasks (Rust.Golden, Notebook, Backtest, HrrrBackfill,
HrrrClimatology, HrrrNativeBackfill, NexradBackfill,
RadarBackfill, ImportContestLogs, PropagationGrid,
ResetEnrichment, Hrrr.PurgeGridPoints) 0 → ~60-100%
Two incidental fixes made while adding tests:
- ContactLive.Show.handle_event("toggle_flag", ...) was passing
socket.assigns.current_scope to admin?/1 instead of socket.assigns,
so admins never matched the pattern and every toggle ran the
"Admins only." flash branch. Flag now toggles for admins again.
- Commercial.PollWorker.fetch_weather/1 promoted from private to
@doc'd public so the IEM-fetch + ASOS-upsert path can be tested
directly without driving perform/1 through live SNMP (which was
timing out for ~50 s per test in the earlier attempt).
Stable property-test additions cover the dewpoint-from-RH monotonicity,
nearest_run/1 idempotence, and the ionosphere station envelope.
New /eme page — antenna aiming + link budget for moonbounce.
Given a station (callsign / grid / lat,lon), TX power, antenna gain,
detection bandwidth, and system Tsys, the page shows in real time:
- Moon azimuth / elevation from the observer (ticks every 10 s)
- Slant range to the Moon
- EIRP breakdown: TX(dBm) + gain(dBi) = EIRP(dBm)
- Path-loss decomposition (Earth → Moon → Earth):
free-space spreading (×2)
− moon reflection gain 10·log(4π σ / λ²) where σ = ρ·π·R²
= round-trip path loss
- Band-dependent lunar albedo ρ (VHF ≈ 0.065, microwave ≈ 0.08,
mm-wave ≈ 0.02) surfaced as a named line item with the current
ρ shown to three decimals
- Received power, thermal noise floor, and SNR with a badge that
classifies the margin (strong / marginal / below noise)
- Self-echo Doppler shift with its radial-velocity driver
New modules:
- `Microwaveprop.Moon` — low-precision lunar ephemeris
(Astronomical Almanac Sec. D.4 truncated series). Provides
`julian_day/1`, `geocentric/1`, `observer_position/3`, and
`radial_velocity_km_s/3`. Accurate to ~0.3° position / ~200 km
distance — well inside any amateur EME-dish beamwidth.
- `Microwaveprop.Propagation.Eme` — path-loss helpers:
`moon_albedo/1` (band lookup), `moon_rcs_m2/1`,
`fspl_round_trip_db/2`, `moon_reflection_gain_db/2`,
`path_loss_db/3`, `received_power_dbm/1`, `noise_floor_dbm/2`,
`snr_db/1`, `doppler_shift_hz/2`.
Wired into the main nav ("EME" button) and the router between
/path and /rover.
Tests: 3 properties + 41 unit tests cover JD conversion, moon
position envelope, band-dependent albedo lookup, decomposition
identity, linearity/monotonicity of path loss and Doppler, and a
LiveView integration test against the rendered DOM.
Full suite: 2,460 tests + 170 properties; credo strict clean.
Root cause of 14 stuck hrrr-queued contacts: hrrr_point_rs's CONUS-only
grid silently writes zero profiles for out-of-bounds points (UK,
Winnipeg, Alberta, >50° lat), but the fetch task still completes
successfully. ContactWeatherEnqueueWorker's hrrr_placeholder_jobs
always emitted a queued placeholder for any contact with a pos1,
which meant mark_hrrr_status! re-flagged :queued every backfill tick
forever.
Introduce Propagation.Grid.contains?/1 as the canonical in-CONUS
check (inclusive on all four edges, nil-safe). Use it in two places
of ContactWeatherEnqueueWorker:
- hrrr_placeholder_jobs/1 now returns [] for OCONUS contacts, same
as pre-2014 / already-complete ones.
- mark_hrrr_status!/3 empty-list clause now branches on Grid.contains?
first (→ :unavailable) before falling through to the existing
NARR-coverage / :complete logic.
Next backfill tick will flip the 14 stuck OCONUS contacts in prod to
:unavailable automatically. The remaining 13 "CONUS" stuck ones are
right at the 50° edge (lat=50.1875 rounds outside the box) — same
reconciliation path.
Coverage additions:
- Grid.contains?/1 — corner-inclusive, OCONUS positive/negative
fixtures based on the actual stuck-contact pos1 values, nil/missing-
key/nil-value handling.
- enqueue_for_contact regression tests: pos1 in the UK and pos1 just
north of the lat cap both land at :unavailable, not :queued.
Suite: 2,416 tests + 159 properties (was 2,409 + 159); credo strict
clean.
Changes under lib/ (source):
- iemre_fetch_worker: replace `if transient_failure?/1` boolean predicate
+ `if/else` with a classifier that pattern-matches the error reason
directly, returning `:transient | :permanent`, and dispatch via
`case`. Three-clause head is clearer than the boolean predicate.
- scores_file.extract_points and nexrad_client.extract_box: force `//1`
step on the row/col comprehensions so an image box whose centre
lands outside the raster collapses to an empty iteration instead
of firing Elixir 1.19's ambiguous-range warning. Adds a regression
test for the right-edge box.
Test coverage added:
- Feature wrappers in Backtest.Features (theta_e_jump, shear_at_top,
duct_thickness, best_duct_freq, duct_usable_for_band,
distance_to_front / parallel_to_front stubs, all_features/0).
- NotifyListener handle_info tolerance for malformed payloads +
unknown messages, plus the PubSub broadcast side effect.
- Viewshed.effective_reach_km across every verdict / diffraction
tier / ducting-score tier combination.
- SnmpClient parse_af60_output unit conversions + unknown-column
handling; parse_snmpget_output OID normalisation and junk-line
tolerance.
- HrrrNativeClient native_level_count, native_variables,
native_messages shape, duct_messages / duct_byte_ranges.
- Changeset coverage for GeomagneticObservation, SolarFluxObservation,
SolarXrayObservation, Ionosphere.Observation, HrrrClimatology, and
Metar5minObservation — lifted each from ~50% / 0% to 100%.
- Property tests: GefsClient.dewpoint_from_rh invariants (Td ≤ T,
monotonic in RH, finite across the operating envelope) + idempotent
nearest_run. NexradClient.pixel_to_dbz monotonicity + bounded
output; latlon_to_pixel round-trip for every grid cell.
Also cleared several unrelated compiler/linter warnings:
- Three private test helpers (create_contact, insert_contact,
current_hour) had `\\ %{}` / `\\ 0` defaults that no caller used.
- profiles_file_test bound `dir` in a pattern match without using it.
Suite: 2,359 tests + 146 properties pass (was 2,300 / 139); coverage
70.38% → 70.89%; credo strict clean.
Rust workers (prop-grid-rs) that die mid-claim (SIGKILL, OOM, node
drain) leave grid_tasks rows stuck in status='running' forever,
because claim_next uses FOR UPDATE SKIP LOCKED and nothing resets the
orphan. The /status page then shows a permanent spinner with stale
f00/f10/f18 badges — currently 21 rows claimed as far back as
2026-04-20.
GridTaskEnqueuer.reclaim_stale_running/1 flips rows whose claimed_at
is older than 15 minutes back to 'queued'. Rows that have already
burned through 5 claim/reclaim cycles become 'failed' with a
reclaim-orphan error so the next hourly seed can replace them.
Wired into PropagationGridWorker.seed_chain/0 so it runs every :05
cron tick before new rows are seeded.
Also rename the status panel "Retrying" column to "Failed" — it was
always showing terminal `failed` rows, never retrying ones.
IemreFetchWorker now writes an empty stub observation when IEM returns
a permanent-failure status (404, 422), mirroring the existing {:ok, []}
path. The backfill cron's next tick sees the stub via
has_iemre_observation?/3, generates no job, and mark_status!/3 flips
the contact's iemre_status from :queued to :complete — draining the
51 contacts that have been stuck behind cancelled out-of-grid IEMRE
jobs.
Raise hot pod CPU limit 2 → 3 so BEAM gets 3 schedulers online.
Observed run queues of [2, 10, 0, 0, 0, 0] on the 2-scheduler
configuration during the :05 propagation chain, starving /live and
/health probe handlers and tripping intermittent liveness timeouts.
ADIF records frequently include operator commentary in the
<NOTES> field (multi-line, detailed) or <COMMENT> (short
one-liner). Now that contacts have a notes column, pass that
value through the import pipeline instead of dropping it.
Preference is NOTES over COMMENT to match the ADIF 3.1.4 spec's
intended split — NOTES is the richer field. Whitespace-only
values collapse to nil so a program that always emits a blank
NOTES tag doesn't fill every row with an empty string.
Tests cover NOTES capture, COMMENT fallback, NOTES-over-COMMENT
preference, and the absent-field nil case. ADIF upload blurb
lists the new carry-over so users see it before confirming.
Operators want a place to jot observations that aren't captured by
the structured fields — weather anecdotes, propagation mode
commentary, equipment details, band conditions. Add a text column
on contacts plus the two ingest paths users submit through:
* /submit single-contact form gets a 3-row textarea under the
other fields. Optional, max 2000 chars, with a live length cap
via maxlength. Whitespace-only input collapses to NULL so the
column reflects "no notes" rather than an empty string.
* CSV importer recognises a `notes` (or `note`) column via the
existing header_aliases table and flows values straight through
submission_changeset. Sample template gets a matching example
row with embedded commas so the quoted-field round-trip is
exercised on download.
ADIF import and the refinement/refinement-notify paths are out of
scope — users specifically asked for CSV + single-contact. Tests
cover the changeset (accept/blank/over-length), CSV header parsing
(plain + RFC-4180 quoted), and LiveView form submit end-to-end.
When the Rust hrrr-point-worker marks a task as `failed` (upstream
`idx HTTP 404` or wgrib2 decode error on old archive formats), the
corresponding contacts sit at `hrrr_status = :queued` until the
generic 3-day stale-queued reconcile flips them. Speed that up: on
every BackfillEnqueue tick, join contacts to `hrrr_fetch_tasks` by
rounded HRRR hour and flip any `:queued` contact whose cycle is
permanently `failed` to `:unavailable` immediately.
Motivation: a handful of 2018-08-04/05 cycles (t20z/t21z/t15z/t17z)
are genuinely missing `wrfsfcf`/`wrfprsf` files on the NOAA S3
archive — other hours on the same dates are fine, so this is an
upstream data gap rather than a structural pre-2019 issue. Users
see an accurate "Partial" marker on those contacts instead of a
three-day `:queued` stall.
Tested: 4 new cases covering rounding, failed-vs-done, and the
`hrrr not in types` skip branch; full suite 2283/0.
Per-pod IEM rate limiter was 700ms (~1.4 req/sec); four pods put the
fleet at ~5.7 req/sec, above the observed 429 threshold. Retry storm
(Req exponential backoff up to 32s) piled concurrent sleeps on top of
live workers and tripped Bandit's acceptor + Oban's leader heartbeat.
- Raise default interval to 1500ms (~2.7 req/sec fleet).
- Hot pod /live timeout 3s → 5s (acceptor stalls under retry sleeps).
- prop-backfill exec probe period 30s → 120s, timeout 15s → 30s; the
release CLI round-trips through distributed Erlang and 15s wasn't
enough under load, restart-looping every ~10 min.
Three bugs were letting mechanism/terrain/radar contacts ping-pong
between :complete and :queued on every backfill cron:
1. enqueue_for_contact/2 now filters out types that are already
:complete on the contact before building jobs. Previously
mark_status!/3 demoted every non-empty jobs_by_type entry back to
:queued, overwriting the :complete that workers had just set.
2. reconcile_stale_queued/1 now also flips mechanism_status and
radar_status back to :complete when the companion data
(propagation_mechanism / contact_common_volume_radar) already
exists. Drains the existing ~14k row backlog on the first post-
deploy tick.
3. IemClient.parse_iemre_json/1 accepts "" and nil so a 200-with-
empty-body IEMRE response doesn't FunctionClauseError the worker
through four retry attempts.
lgdc.uml.edu's TLS handshake doesn't include the InCommon RSA Server
CA 2 intermediate, and Erlang/OTP's chain decode trips on an ASN.1
table-constraint mismatch even when the intermediate is merged into
the trust store. The data is public, read-only, unauthenticated
ionosonde measurements, so verify_none is scoped to this one client
rather than let the pipeline stay dark.
Hot pods were restart-looping every ~25 minutes on liveness probe
timeouts. Root cause: ScoreCacheReconciler mirrored every .prop file
from NFS into ETS, including 7 days of GEFS Day 2-7 forecasts. With
23 bands × 43 valid_times × ~2 MB per grid = 1.95 GB in the
propagation_score_cache table alone; GC sweeps starved the
scheduler enough that /live dropped its 3 s budget.
The /map UI only ever requests the [now-1h, now+18h] window. Share
that bound as Propagation.hot_cache_window/0 and apply it in the
reconciler's disk-scan path plus NotifyListener's post-warm prune.
Long-horizon GEFS files stay on disk and are still served via the
lazy read_from_disk_and_cache path when requested directly.
Adds ScoreCache.prune_outside_window/2 (inclusive bounds) and
updates the reconciler tests to use relative-to-now timestamps
since hardcoded fixture dates now drift out of window.
**Wire format — /map scores payload.** Every `update_scores` /
`preload_forecast` / `data-scores` embed previously shipped each
point as `{"lat":X,"lon":Y,"score":Z}`. At 95k cells in full CONUS
that's ~36 bytes/point × 95k = 3.4 MB of JSON, ~50% of it repeating
the three key names. Packing as `[lat, lon, score]` tuples drops
per-point overhead to ~19 bytes → ~1.8 MB on the wire, ~45% smaller.
New `Propagation.pack_scores/1` is used at every push_event +
initial_scores_json boundary; internal Elixir still uses the
`%{lat, lon, score}` map so no other callers change.
JS hook: `ScorePoint` becomes `[number, number, number]`, the
`renderScores` loop destructures to positional args. Regression
test in map_live_test updated to match the new shape.
**ProfilesFile compression.** Rust writer was at gzip level 6
(default). Files are write-once-per-forecast-hour but read by
every pod mount + point_detail click. Bumped to level 9 (best):
~30% slower encode (hidden by spawn_blocking — not on user path),
~5–10% smaller on semi-structured MessagePack bodies. No reader
changes needed — same gzip stream.
**TypeScript strictness.**
- `app.ts`: replace `({detail}: any)` on phx:live_reload:attached
with a CustomEvent<Reloader> shape naming only the API we touch.
- `weather_map_hook.ts`: drop the `(this as any)._map` pair by
declaring a `GridLayerWithMap = L.GridLayer & { _map: L.Map }`
intersection on the overlay's createTile `this` parameter.
- `global.d.ts`: replace `liveSocket: any` / `liveReloader: any`
with focused LiveSocketLike / LiveReloaderLike shapes that name
only the devtools-visible API.
No more `any` in the codebase (verified via rg).
Two wins that directly shrink /map and /weather latency under
interactive use:
**Debounce the moveend → repaint path (150ms).** Leaflet fires
`moveend` once per pan but 5–10× per second during a wheel-zoom.
Previously each one ran a full `scores_at` (or `weather_grid_at`)
read + a ~20–100KB websocket push. Now they coalesce into a single
trailing-edge flush via `schedule_bounds_update` (and the
/weather twin `schedule_weather_flush`) — the same pattern we
already use for forecast preload. 150ms is short enough that the
user doesn't perceive the pause; >90% of the burst disappears.
**Parallelize point_forecast across 4 tasks.** The sparkline that
pops up on point click was reading up to 18 `.prop` files
sequentially off NFS. Each read is cheap (pread one byte at
row*cols+col) but RTT-dominated. `Task.async_stream` with
max_concurrency=4, ordered=true brings 18-file wall time from
~50ms down to ~15ms on NFS without changing the public API.
No test changes: existing map_bounds test only asserts the bounds
assign lands, which the debounced path still does synchronously.
The main /map timeline was rendering each forecast hour twice
('Now 22:00' + '0h 22:00', '+1h 23:00' + '+1h 23:00', …).
Root cause: during the rolling `.ntms` → `.prop` rename window,
both extensions live on NFS for the same valid_time. `list_score_files`
matches either, so `list_valid_times_raw` mapped 2 files → 2 DateTimes
with no dedupe. Propagation.available_valid_times inherited the dup,
push_timeline pushed it to the JS hook, renderTimeline drew one chip
per entry — hence the visible duplicate.
Fix: `Enum.uniq` before the sort. Regression test writes a `.prop`
file, copies it as the same valid_time's `.ntms` legacy twin, and
asserts list_valid_times returns a single DateTime.
Also land a small refactor that was queued behind this: move the three
pure mechanism display helpers (label/1, badge_class/1, explainer/1)
out of the 2500-line contact_live/show.ex into a new
MicrowavepropWeb.ContactLive.Mechanism module. No behavior change — just
shrinks the LiveView and gives the display mapping a testable home.
StatusLive had a hand-rolled thousands-separator format_number/1 and a
binary-prefix format_bytes/1 — duplication that was already part of the
review punch-list. Move both to Microwaveprop.Format (which previously
held only distance_km) and delete the local defps. Fourteen call sites
switched to Format.number / Format.bytes.
Also extend the IEM error classifier to snooze on Req.TransportError
{:closed, :timeout, :econnrefused, :nxdomain}. Same rationale as the
429 snooze in batch 1: upstream TCP resets mid-fetch aren't a job
failure, they're transient — returning {:error, _} blows them up into
Oban.PerformError logspam with full stacktraces.
Batched from a system-wide review pass.
**Rate-limit + transient-error log hygiene**
- WeatherFetchWorker: classify IEM HTTP 429 as {:snooze, 300} instead
of {:error}. Stops Oban.PerformError stack-trace spam on every
routine rate-limit during backfill, keeps the retry semantics.
- IonosphereFetchWorker: compress GIRO TLS :unknown_ca error from a
~400-char inspect blob to 'TLS unknown_ca (CA bundle missing)'.
- FreshnessMonitor: log only when an enqueue actually lands (the
Oban unique conflict case was silently dropping jobs but still
producing 'enqueuing grid worker' info lines every 5 minutes).
**Perf**
- Rust grid_level_keys(): pre-format 'TMP:{p} mb' / DPT / HGT keys
once via OnceLock instead of format!()'ing per-cell. Removes ~3.6M
String allocations per f01..f18 chain step across pipeline.rs,
hrrr_points.rs. Same for the wgrib2 :(TMP|DPT|HGT): pattern in
hrrr_points.process_batch (sfc_pattern / prs_pattern OnceLock).
- MapLive preload_forecast: Task.async_stream with max_concurrency=4
replaces the 18-wide Enum.map serial walk. Forecast cache warms
~4× faster after a band change, with ordering preserved.
- grid_tasks: partial composite index on (run_time DESC,
forecast_hour ASC) WHERE status='queued' AND kind='forecast',
matching the Rust claim query's ORDER BY. Drops the old
status-only partial that forced a sort per claim.
**Correctness**
- ContactImportWorker: add Oban unique:[keys: [:import_run_id,
:offset]]. Was missing on a worker whose perform() does a
non-idempotent atomic counter increment — a retry would double-
count imported rows.
- CommonVolumeRadarWorker: x_min..x_max default step is -1 when
x_min > x_max, triggering a runtime deprecation warning. Force
Range.new(_, _, 1) explicitly.
**Cleanup**
- Drop unused tmp_dir parameter from hrrr_point_worker + its
process_batch signature.
Two unrelated production warnings from the same log window:
WeatherFetchWorker raised Postgrex 21000 cardinality_violation on
ASOS upserts when IEM returned two METAR records for the same
timestamp (routine + special at the same minute). Dedupe by
(station_id, observed_at) in upsert_surface_observations, keeping
the last occurrence since IEM's ordering is chronological.
SwpcClient.fetch_xrays warned with the full truncated response
body (~650 KB of GOES X-ray JSON) dumped into the log aggregator
whenever SWPC's CDN cut the response mid-stream. Intercept
Jason.DecodeError in fetch_and_parse and format it short: keep
the byte position and an 80-char preview, drop the rest.
Two fixes cover the blank "analysis breakdown" panel the user reported:
1. Bound the point_detail fallback lookback to 24h. Previously
factors_from_fallback_profile would happily use a week-old analysis
profile when no current one existed — now anything older than 24h
returns an empty factor map so the UI can surface "breakdown
unavailable" instead of silently misattributing.
2. Move the Plausible analytics snippet from Layouts.app into
root.html.heex. Full-bleed LiveViews (/map, /contacts/map,
/weather, home) bypass Layouts.app, so the snippet only loaded on
the navbar-wrapped pages. root.html.heex is loaded once per HTTP
document so coverage is now universal.
Added ~30 tests locking both down:
• point_detail fallback: exact-time wins, 24h boundary accepted,
30h-stale rejected, multi-band coverage, missing-cell degrades to
empty factors, default-valid_time path
• analytics_test.exs: Plausible script present on 12 representative
pages, exactly once, loaded async, surviving a login round-trip
Also fixed pre-existing credo issues per standing rule: shortened the
map_live_test ETS match_delete call, extracted a function from the
deeply-nested hrrr_point_enqueuer.enqueue_for_contacts/1 cond, and
swapped Mix.Tasks.Rust.Golden's IO.inspect for Mix.shell().info.
Score-grid files now land at `<band>/<iso>.prop` with a 4-byte "PROP"
magic header. Readers still accept the legacy `.ntms` extension and
"NTMS" magic so a rolling deploy doesn't invalidate any file already
on the shared NFS mount — legacy files age out through normal
retention.
Applied both sides of the seam: Elixir ScoresFile + regex parsers,
Rust scores_file writer + decoder. Updated the comments in all
modules that mention the extension (NotifyListener, Reconciler,
gefs_fetch_worker, map_live) and the runbook diagram. talos5 is a
DB host now, not a Rust worker — corrected the runbook accordingly.
Convert every context/worker upsert that used the catch-all
`:replace_all_except` form to the explicit `{:replace, [:col1, ...]}`
form. Reviewers can now see exactly which columns an upsert overwrites,
and adding a new column to a schema no longer silently opts it into the
update path.
Behaviour is preserved bit-for-bit: each new explicit list contains
every column the old `:replace_all_except` would have overwritten.
Touched:
- Microwaveprop.SpaceWeather (Kp / F10.7 / X-ray shared helper)
- Microwaveprop.Ionosphere (observation upsert)
- NexradWorker, HrrrNativeGridWorker (insert_all grids)
- CommonVolumeRadarWorker, RadarFrameWorker (per-contact stats)
Also pins the conflict behaviour for the ContactCommonVolumeRadar
upsert path in RadarFrameWorkerTest so a future column addition that
isn't reflected in the explicit list fails loudly.
Wrap a Task.Supervisor in a PartitionSupervisor (one partition per
scheduler) and route the four sustained async_stream callers through
`{:via, PartitionSupervisor, {Microwaveprop.TaskSupervisor, self()}}`.
Concurrent work — the hourly PropagationGridWorker pulling f00-f18
HRRR range downloads, the GEFS worker scoring its grid, and the
Recalibrator's 20-way positive/negative fan-out — no longer funnels
through a single supervisor PID.
Light/one-shot Task.async + Task.start sites (hrrr_client surface/
pressure fan-out, application boot warmup, weather fire-and-forget)
are left alone; partitioning only helps under sustained concurrency.
PromEx.Plugins.Oban counts job exceptions as a Prometheus counter but
does not tell us which worker/queue failed, whether the job is about to
be retried, or the exception class. This adds a telemetry handler on
[:oban, :job, :exception] that:
- emits a single structured Logger.error with worker, queue, job_id,
attempt, max_attempts, retry_exhausted?, kind, reason class, and
args-keys summary (never values — args can contain PII/tokens);
- dispatches opt-in per-worker recovery callbacks
(on_permanent_failure/1 when retries exhausted,
on_transient_failure/1 otherwise);
- catches callback failures so a bug in one worker's recovery path
cannot detach the handler or break logging for other jobs;
- does not emit a second telemetry event, so PromEx counts are not
double-counted.
Handler is attached from Application.start/2 after the Oban supervisor
starts.
Every pod was rescheduling its sweep on a fixed 60 s `Process.send_after`
tick, so N replicas hit the shared NFS `/data/scores` mount and Postgres
in lock-step every minute. Add a uniform jitter of up to 20 s on top of
the 60 s base (effective range 60-80 s) so per-replica schedules drift
apart.
Expose `next_sweep_interval/0,2` as a public helper so the randomized
window can be tested deterministically with `:rand` draws rather than
mocking timers. Existing reschedule test passes `jitter_max_ms: 0` to
keep its 2 s wait window honest.
- HrrrClient idx-cache test only invalidated the surface idx URL, but
fetch_profile also fetches a pressure idx. Previous runs' state for
the pressure key decided whether the counter landed at 1 (prior run
cached it, pass) or 2 (cold, fail). Invalidate both + assert 2 total
fetches to reflect the actual code path.
- CsvImportTest deadlocked against other async DataCase tests when
inline Oban child jobs upserted iemre_observations/terrain_profiles
with a shared conflict target. Flip to async: false — same fix as
ContactWeatherEnqueueWorkerTest earlier this session.
- PromEx.Plugins.Oban runs a 5s telemetry_poller that queries the DB,
but its poller PID has no sandbox connection in test and crashed
with DBConnection.OwnershipError on every tick, spamming the log.
Gate the plugin on a config flag and skip it in config/test.exs;
prod behaviour unchanged.
Band-warm failures in the Rust propagation_ready pipeline were only
surfaced via Logger.warning, so Prometheus had no way to see them.
The /weather GridCache hit/miss was likewise invisible while its
/map counterpart (ScoreCache) already reported cache ratio.
- NotifyListener emits [:microwaveprop, :propagation, :notify_listener, :warm]
with %{ok, err} measurements and %{valid_time} metadata after every
propagation_ready notification. Public handle_propagation_ready/1 so
tests can exercise the warm+telemetry path without Postgrex.
- GridCache.fetch/1, fetch_bounds/2, and fetch_point/3 emit
[:microwaveprop, :weather, :grid_cache, :lookup] with a :hit | :miss
metadata key on every ETS lookup.
Wraps the hourly grid_tasks seed with
[:microwaveprop, :propagation, :grid_worker, :perform] so Prometheus
can see duration, success, and failure for the chain-seed cron.
Emits a :seeded event with the queued_tasks count and an :exception
event on GridTaskEnqueuer failure so silent seed errors show up in
the existing PromEx dashboards instead of dying as a lone Logger.info.
IemClient.fetch_asos returns 24-288 rows per station per call, each
previously triggering an individual UPDATE-conflict round-trip via
Weather.upsert_surface_observation/2. On the Turing Pi 2 Postgres
node that per-row latency dominates ingestion time.
Add Weather.upsert_surface_observations/2, a bulk variant that collapses
the rows into a single Repo.insert_all with the same
update-only-when-changed on_conflict predicate and (station_id,
observed_at) conflict target. Switch WeatherFetchWorker to use it.
The Microwaveprop.Weather.GridCache ETS table grew unbounded — each
hourly PropagationGridWorker run plus peer broadcast_put added ~92k
points per forecast hour, so long-lived pods were strong OOM
candidates. GridCache already exposed prune_older_than/1 but nothing
called it.
Adds an Oban worker on the :propagation queue (lower priority than
the grid chain) that prunes entries older than 3 hours. Scheduled
hourly at :15 in dev, prod, and the base crontab.
CommonVolumeRadarWorker and MechanismClassifyWorker were both
returning :ok on upstream failures, so Prometheus job-failure counters
stayed at zero during multi-hour NEXRAD outages and classifier bugs
went unnoticed.
- CommonVolumeRadarWorker: distinguish permanent (4xx) from transient
errors using the same pattern as RadarFrameWorker. Permanent errors
still mark :unavailable + :ok so the backfill stops re-queueing them.
Transient errors return {:error, reason} and leave radar_status alone
so the next retry can succeed.
- MechanismClassifyWorker: keep the row-status :failed update so
operators can see which contacts blew up, but return
{:error, inspect(e)} from the rescue instead of swallowing it as :ok.
Rework of d61fbd3's dialyzer suppression: the 4 warnings from
LiveTable.LiveResource's macro expansion are now fixed at the
root rather than hidden via .dialyzer_ignore.exs.
Changes:
- Delete .dialyzer_ignore.exs and remove ignore_warnings from
mix.exs. No more hidden suppressions.
- New MicrowavepropWeb.LiveTableResource shim that wraps
use LiveTable.LiveResource and overrides:
* maybe_subscribe/1 — explicit :ok = on Phoenix.PubSub.subscribe/2
(previously discarded, tripping :unmatched_return).
* handle_info/2 — explicit _ = on File.rm, Process.send_after.
Switch all 4 LiveTable-using LiveViews (contact_live/index,
beacon_live/index, admin/contact_edit_live, user_management_live/index)
to use MicrowavepropWeb.LiveTableResource instead.
- New priv/dep_patches/live_table-0.4.1.patch adds _ = in front of
LiveTable.LiveSelectHelpers.restore_live_select_from_params/2 in
the dep's macro-expanded handle_params/3. The call's return was
silently discarded, tripping the fourth warning per LiveView.
- New lib/mix/tasks/deps.patch.ex applies every
priv/dep_patches/*.patch to its target dep idempotently after
mix deps.get. Wired into the :setup alias + overridden
mix deps.get so the patch survives cache regeneration in CI.
- Other modified files in this commit are the spec-coverage
additions from the parallel agent pass (beacons.ex,
commercial.ex, propagation.ex, weather.ex, narr_client.ex,
run_timing.ex, 3 workers) — tighter specs for bounds, tuple
shapes, schema-typed params. Dialyzer remains at 0 after
the changes.
Upstream TODO: send a PR for the live_table `_ =` fix. Once
merged and released, drop the patch + mix task + bump the dep.
mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 2 pre-existing flakes (MapLive timestamp +
PropagationPrune ScoresFile), 0 regressions.
- endpoint.ex log_level/1 now filters by conn.request_path instead of
conn.path_info. Plug.Router.forward/2 (deps/plug/lib/plug.ex:170)
rewrites path_info to the unmatched remainder and extends script_name
with the matched prefix before dispatching to the forwarded plug.
/metrics is routed via forward "/metrics", MetricsPlug; by the time
Plug.Telemetry's before_send callback fires inside MetricsPlug.call/2
the conn it observes has path_info: [] and script_name: ["metrics"],
so the old log_level(%{path_info: ["metrics" | _]}) clause never
matched and the default :info level fired. request_path is set by
the adapter at entry and is never rewritten, making it the correct
discriminator. New regression test in metrics_log_suppression_test.exs
captures both the direct shape and the integration path via Plug.Test.
- Scorer.dbz_to_rain_rate_mmhr/1 now uses a compile-time @mp_inv_b
constant (1 / 1.6) instead of dividing on every rain pixel.
composite_score/2's band-invariant fallback switched from four
separate short-circuits (which silently
mixed cached and freshly-computed values if only some keys were
passed) to a single Map.has_key?/2 branch that honors the "all or
none" contract. Dropped unused band_invariant_tod/1 helper.
- Propagation.replace_scores span scope tightened: the
Instrument.span([:db, :replace_scores]) now wraps only the per-band
ScoresFile.write! loop, not the upstream Enum.group_by grouping
phase. The span name + metadata stay unchanged so Grafana panels
keep working, but the fixed telemetry dispatch cost (~100µs x 2)
is no longer paid for trivially small result sets.
- Added @type t :: %__MODULE__{...} to Accounts.UserToken and
Weather.HrrrClimatology — the last two schemas that lacked one.
Elixir 1.19's set-theoretic inference benefits from every struct
having an explicit t/0 so callers can flow through tightly.
mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 3 pre-existing flakes, 0 new regressions.
Fixes flagged by the code-reviewer agent's pass over the session's
commits (cc9220b..7b78a25):
- Propagation.warm_cache_and_broadcast/2 now uses ScoresFile.read/2
directly and returns {:ok, :ok} | {:error, :enoent | :invalid_format}.
Previously it called ScoresFile.read_bounds/3 which silently returns
[] on missing/corrupt files, poisoning ScoreCache with an empty grid
that the reconciler couldn't heal.
- NotifyListener.warm_band/2 and ScoreCacheReconciler.warm_one/2 now
pattern-match {:error, reason} and skip (log, return :error) instead
of caching empty. rescue clauses kept as defense-in-depth for
unexpected faults in PubSub.broadcast / ETS writes.
- ScoresFile.extract_points/2 promoted to @doc public — callers that
need to distinguish missing file from empty grid can feed read/2
payloads here themselves.
- Weather.build_grid_cache_row/4: replaced || fallbacks with prefer/3
helper (Map.fetch) so a legitimate persisted ducting_detected: false
is not clobbered by a derived-from-sounding true.
- Weather.hrrr_data_fully_present?/1 @spec tightened from map() to
Contact.t() | field-constrained map, plus is_nil(qso_timestamp)
guard so callers with partial contacts get a clean false rather
than a HrrrClient.nearest_hrrr_hour/1 crash.
- AdminTaskWorker.native_derive bulk-UPDATE: chunk reduced 2000 → 500
and wrapped in try/rescue with per-row fallback on Postgrex errors
so a single bad row doesn't kill the remaining 1999 in its chunk.
- Runbook FM3 rewritten to match the actual code path (no rescue;
explicit {:error, _} pattern match). FM5 clarifies the surviving
PropagationGridWorker is a cron-fired seed worker, not a fallback
compute path.
- Dialyzer: strict flags added (:error_handling, :unknown,
:unmatched_returns, :extra_return, :missing_return). Baseline
emitted 130 warnings, mostly discarded Task.start/PubSub/Logger
returns; a follow-up will tighten those and backfill @specs.
New tests:
- ScoreCacheReconciler GenServer lifecycle: run_on_start true/false,
interval_ms rescheduling, info-level log line.
- Weather.hrrr_data_fully_present?/1: nil qso_timestamp returns false.
- Weather.build_grid_cache_rows/2: explicit ducting_detected: false
on profile beats derived true from sounding params.
- RadarFrameWorker: pins the NexradClient "NEXRAD n0q HTTP <code>"
error string contract so permanent_error?/1 classification doesn't
silently regress if the client's error format changes.
1. GridCache: auto-release fill lock when the claimer process crashes.
claim_fill/1 + release_fill/1 go through the GenServer so the
server can Process.monitor the caller and clean up the ETS entry
on :DOWN. Clear/0 now resets both the data table and the lock
table. Fixes a latent bug where a crashed fill leaked the lock
indefinitely, preventing every subsequent /weather mount for that
valid_time from claiming and leaving cache cold.
2. RadarFrameWorker: distinguish permanent vs transient fetch errors.
404 from the IEM n0q archive is permanent (file will never exist)
and marks contacts :unavailable as before. Any other error shape
(5xx, timeout, transport failure) now returns {:error, reason}
so Oban retries — previously those also pinned contacts at
:unavailable after a transient outage.
3. AdminTaskWorker.native_derive: replace per-row Repo.update_all
(N round-trips + N fsyncs) with one UPDATE ... FROM unnest(...)
per 2000-row batch. For the 10k-profile budget this is one
network round trip per chunk instead of 10k, and one fsync per
chunk instead of 10k. Restructured the clause to separate
derivation (pure) from persistence (I/O).
All three changes are test-covered (grid_cache_test auto-release
test, radar_frame_worker_test 5xx + transport tests, existing
admin_task_worker_test native_derive coverage exercises the new
bulk path). Also drops the scorer_diff no-op test that was
verifying the clause removed in 61da51c.
- AdminTaskWorker: remove scorer_diff no-op clause (propagation_scores table
was dropped pre-cutover; queue is drained).
- Weather: remove load_weather_grid_from_db/1 fallback — Rust writes
ProfilesFile for every forecast hour, so the DB path was unreachable
in practice and loaded stale pre-cutover data when it fired.
- Weather.build_grid_cache_row/4 now prefers persisted fields from the
ProfilesFile profile map (surface_refractivity, min_refractivity_gradient,
ducting_detected, duct_characteristics) over re-deriving from the raw
profile, matching the old DB-path semantics.
- NotifyListener: wrap per-band warm in try/rescue so one corrupt or
mid-rename .ntms file doesn't abort the remaining bands. Failures log
a warning with band + valid_time; successes still emit one info line.
- weather_grid_test: rewrite insert_hrrr_profile helper to write
ProfilesFile directly and drop the "only returns points on 0.125 grid"
test (filter was a property of the deleted DB fallback; Rust writes
only grid points by construction).
Column name now matches the type key used everywhere else:
:hrrr -> hrrr_status, :weather -> weather_status,
:terrain -> terrain_status, :mechanism -> mechanism_status
Drops the @status_column_overrides remap in BackfillEnqueueWorker
(status_column/1 is now a straight :"#{type}_status" derivation) and
the long-form references in Contact, ContactWeatherEnqueueWorker,
and MechanismClassifyWorker. Migration is a metadata-only RENAME
COLUMN + RENAME INDEX — safe to land in prod (no table rewrite,
brief catalog lock only).
Periodic 60s sweep re-warms ScoreCache from /data/scores whenever
{band, valid_time} pairs are on disk but absent in local ETS.
Covers the three silent-fail modes of the NOTIFY/LISTEN path:
dropped notifies on reconnect, missing listener, and stale NFS
reads. Each pod reconciles its own cache — no PubSub fan-out,
no coordination required. Rescue on File.Error handles the
rare window where an atomic-rename write races the reader.
docs/runbook_propagation_pipeline.md names every failure mode
in the Rust↔Elixir seam and what recovers it.
Pure domain check — "does this contact already have HRRR profiles at
every path point for its QSO hour" — has no business living in an
Oban worker. Weather already owns has_hrrr_profile?/3 and
round_to_hrrr_grid/2, so consolidation keeps HRRR knowledge in one
context and leaves ContactWeatherEnqueueWorker focused on job
coordination.
Phase 2 cutover completed in 65693ed — Rust prop-grid-rs owns
f01-f18 production writes and the shadow path has been dead
for weeks. No callers in lib/, test/, or mix tasks.
hrrr_placeholder_jobs previously returned [contact.id] for any
post-2014 contact with pos1, regardless of whether the HRRR
profiles were already in the DB. That meant BackfillEnqueueWorker
re-flagged the same ~22k contacts as :queued on every 30 min
cron tick forever — HrrrPointEnqueuer saw 0 missing points and
inserted nothing, but the status field never advanced.
Now the placeholder also returns [] when every path point has
an hrrr_profile row, so mark_hrrr_status!/3's empty-list branch
flips the contact to :complete directly. Genuinely-missing
contacts still flow through HrrrPointEnqueuer unchanged.
Two wins in the /map hot path.
1. preload_forecast fired on every Leaflet moveend — which arrives in
bursts during pan/zoom — and each firing read + filtered + shipped
18 forecast-hour score lists (up to 90k cells per hour at full
CONUS view) over the LiveView websocket. Now debounced to the
trailing edge: the user must stop moving for 750 ms before we pay
the preload cost. select_band and propagation_updated go through
the same scheduler so a storm of PubSub updates during an hourly
chain also coalesces.
2. ScoreCache.grid_to_filtered_list walked the 92k-cell grid twice
(Enum.filter then Enum.map) and allocated an intermediate tuple
list between them. Enum.reduce emits only in-bounds result maps
directly — halves map traversal + drops the tuple intermediate.
