Passes `capture_log: true` to ExUnit.start/1 so log output emitted
during a passing test stays in the per-test capture buffer instead of
stdout. Drops ~360 lines of noise from the suite: Postgrex teardown
disconnects, expected worker failures, NexradClient 404 stubs,
NotifyListener warm-skip messages, SNMP poll failures, PromEx tag-
drop warnings, and OTel-handler boot errors. Failing tests still
surface their full logs on report, so debugging is unchanged.
Side-effect fixes:
- LoggerFormatTest opts out via @moduletag capture_log: false — it
patches the :default :logger handler and capture_log hot-swaps the
same handler, so the setup lookup would 404.
- TelemetryTest.start_link/1 unlinks the supervisor before the brutal
kill so the exit signal doesn't propagate to the test process.
- Drop unused `import Ecto.Query` in a top_hours mix-task test.
- Drop unused default-arg `attrs \\ %{}` on create_contact/1 in
contact_map_live_test (every caller passes attrs explicitly).
Remaining noise is ~3 lines from oban_pro vendored-dep @impl warnings
(require an upstream patch) plus rare intermittent async teardowns.
Adds 30 more test cases (2594 → 2624 tests) targeting the last
low-coverage hot spots.
Notable additions:
- ContactLive.Show fully-hydrated render path (seeds HrrrProfile,
TerrainProfile, ContactCommonVolumeRadar, Sounding, SurfaceObservation,
IemreObservation) exercises refresh_computed, build_propagation_analysis,
build_data_sources, compute_elevation_profile, and the downstream
render branches they unlock.
- handle_info :terrain_ready and :sounding_fetched paths for same-contact
and unrelated-contact dispatch.
- Flagged-invalid badge + line-through header style.
- Owner-label button path (Edit vs Suggest Edit).
- NotifyListener init/1 subscribes on schedule; handle_info(:subscribe)
succeeds against the live Repo.
- RecalibrateScorer runs cleanly against an empty corpus via the
Recalibrator insufficient-data fallback.
- HrrrNativeClient.extract_native_profiles/2 tolerates malformed input
without crashing.
Adds ~600 new test cases (2013 → 2613 tests; 170 properties; 0
failures) across 12 new test files plus expansions of eight existing
files. Big lifts per module:
ContactLive.Mechanism 33 → 100%
MetricsPlug 54 → ~100%
Microwaveprop.Release 15.8 → 70%+
Telemetry 38 → 76%
HrrrNativeGridWorker 27.7 → 76%
ContactLive.Show 34 → 48% (handler + render branches)
Admin.ContactEditLive 49.7 → 70%+
GefsFetchWorker 35.8 → ~55%
IonosphereFetchWorker 56.3 → 75%
PathLive 58.9 → 67%
WeatherMapLive 66.9 → 80.2%
RoverLive 0 → 70.3%
ContactMapLive 59.2 → 89.8%
ContactMapController 0 → 100%
Mix tasks (Rust.Golden, Notebook, Backtest, HrrrBackfill,
HrrrClimatology, HrrrNativeBackfill, NexradBackfill,
RadarBackfill, ImportContestLogs, PropagationGrid,
ResetEnrichment, Hrrr.PurgeGridPoints) 0 → ~60-100%
Two incidental fixes made while adding tests:
- ContactLive.Show.handle_event("toggle_flag", ...) was passing
socket.assigns.current_scope to admin?/1 instead of socket.assigns,
so admins never matched the pattern and every toggle ran the
"Admins only." flash branch. Flag now toggles for admins again.
- Commercial.PollWorker.fetch_weather/1 promoted from private to
@doc'd public so the IEM-fetch + ASOS-upsert path can be tested
directly without driving perform/1 through live SNMP (which was
timing out for ~50 s per test in the earlier attempt).
Stable property-test additions cover the dewpoint-from-RH monotonicity,
nearest_run/1 idempotence, and the ionosphere station envelope.
New /eme page — antenna aiming + link budget for moonbounce.
Given a station (callsign / grid / lat,lon), TX power, antenna gain,
detection bandwidth, and system Tsys, the page shows in real time:
- Moon azimuth / elevation from the observer (ticks every 10 s)
- Slant range to the Moon
- EIRP breakdown: TX(dBm) + gain(dBi) = EIRP(dBm)
- Path-loss decomposition (Earth → Moon → Earth):
free-space spreading (×2)
− moon reflection gain 10·log(4π σ / λ²) where σ = ρ·π·R²
= round-trip path loss
- Band-dependent lunar albedo ρ (VHF ≈ 0.065, microwave ≈ 0.08,
mm-wave ≈ 0.02) surfaced as a named line item with the current
ρ shown to three decimals
- Received power, thermal noise floor, and SNR with a badge that
classifies the margin (strong / marginal / below noise)
- Self-echo Doppler shift with its radial-velocity driver
New modules:
- `Microwaveprop.Moon` — low-precision lunar ephemeris
(Astronomical Almanac Sec. D.4 truncated series). Provides
`julian_day/1`, `geocentric/1`, `observer_position/3`, and
`radial_velocity_km_s/3`. Accurate to ~0.3° position / ~200 km
distance — well inside any amateur EME-dish beamwidth.
- `Microwaveprop.Propagation.Eme` — path-loss helpers:
`moon_albedo/1` (band lookup), `moon_rcs_m2/1`,
`fspl_round_trip_db/2`, `moon_reflection_gain_db/2`,
`path_loss_db/3`, `received_power_dbm/1`, `noise_floor_dbm/2`,
`snr_db/1`, `doppler_shift_hz/2`.
Wired into the main nav ("EME" button) and the router between
/path and /rover.
Tests: 3 properties + 41 unit tests cover JD conversion, moon
position envelope, band-dependent albedo lookup, decomposition
identity, linearity/monotonicity of path loss and Doppler, and a
LiveView integration test against the rendered DOM.
Full suite: 2,460 tests + 170 properties; credo strict clean.
- identify_var: full round-trip across every known (discipline,
category, number) triple, plus a property that any triple outside
the 12-entry lookup falls back to the "cat:num" default.
- identify_level: hybrid-level and entire-atmosphere branches, plus
two properties — pressure levels always render as
`div(value_pa, 100) <> " mb"` and unknown surface types always
render as `unknown:<type>:<value>` (generator samples only from
types outside the 6-entry lookup set).
- sign_magnitude_16: round-trip property over [-32_767, 32_767] (±0
both decode to 0 as intended) and a monotonicity property inside
a single sign band.
Suite: 2,419 tests + 164 properties (was 2,416 + 159); credo strict
clean.
Root cause of 14 stuck hrrr-queued contacts: hrrr_point_rs's CONUS-only
grid silently writes zero profiles for out-of-bounds points (UK,
Winnipeg, Alberta, >50° lat), but the fetch task still completes
successfully. ContactWeatherEnqueueWorker's hrrr_placeholder_jobs
always emitted a queued placeholder for any contact with a pos1,
which meant mark_hrrr_status! re-flagged :queued every backfill tick
forever.
Introduce Propagation.Grid.contains?/1 as the canonical in-CONUS
check (inclusive on all four edges, nil-safe). Use it in two places
of ContactWeatherEnqueueWorker:
- hrrr_placeholder_jobs/1 now returns [] for OCONUS contacts, same
as pre-2014 / already-complete ones.
- mark_hrrr_status!/3 empty-list clause now branches on Grid.contains?
first (→ :unavailable) before falling through to the existing
NARR-coverage / :complete logic.
Next backfill tick will flip the 14 stuck OCONUS contacts in prod to
:unavailable automatically. The remaining 13 "CONUS" stuck ones are
right at the 50° edge (lat=50.1875 rounds outside the box) — same
reconciliation path.
Coverage additions:
- Grid.contains?/1 — corner-inclusive, OCONUS positive/negative
fixtures based on the actual stuck-contact pos1 values, nil/missing-
key/nil-value handling.
- enqueue_for_contact regression tests: pos1 in the UK and pos1 just
north of the lat cap both land at :unavailable, not :queued.
Suite: 2,416 tests + 159 properties (was 2,409 + 159); credo strict
clean.
- Format.number/1: boundary cases (0, 999, 1_000), float rounding,
nil/atom fallback to to_string, plus two properties — comma
insertion is lossless round-trip and the comma count follows
`div(digits - 1, 3)` for |n| ≥ 1000.
- Format.bytes/1: per-unit boundary cases plus a monotonicity
property (parse-back preserves ordering across the B/KB/MB/GB
transitions).
- Instrument.span/2,3 (0→100%): wraps the block in a telemetry span
returning its value, defaults metadata to %{}, emits the
:exception event and re-raises on a raise inside the span, and
prefixes every event suffix with :microwaveprop.
- SolarClient: short-line / empty-line / blank-day handling for
parse_gfz_row; blank-line + short-line tolerance in parse_gfz_file;
filter_since boundary semantics (>= is inclusive, empty list
passes through).
Suite: 2,397 tests + 159 properties (was 2,382 + 155); coverage
71.12% → 71.16%; Format + Instrument now at 100%; credo strict clean.
- PageController: cover the legacy `/qsos` → `/contacts` redirect pair
(both the index and the id-preserving per-resource redirect), which
had no coverage since the route rename.
- Scorer property tests:
- f_to_c / c_to_f are inverses up to float epsilon, both propagate
nil, and the C→F slope is exactly 5/9.
- wind_speed_kts is non-negative, propagates nil on either axis,
and matches sqrt(u²+v²)·1.94384 (the 5-digit kts factor the
scorer uses).
- precip_to_rate_mmhr is non-negative, 0 for nil/zero/negative.
- dbz_to_rain_rate_mmhr is 0 below 5 dBZ (and for nil), and
monotonically non-decreasing above — guards against accidental
inversion of the Marshall-Palmer lookup.
Suite: 2,382 tests + 155 properties (was 2,380 + 148); coverage
71.08% → 71.12%; credo strict clean.
- SwpcClient: unparseable time_tag rows drop silently, integer / string
numerics both cast, already-decoded list bodies (Req auto-decode) are
accepted, non-JSON-array bodies return {:error, :not_a_list}, X-ray
rows whose energy isn't the long wavelength are filtered, and the
misspelled-upstream electron_contaminaton key flags
electron_contaminated.
- UwyoSoundingClient: every month abbreviation routes to its correct
number, bogus month tokens fall through to an empty result, and
short-of-fixed-width lines are skipped without raising.
- Grid.wgrib2_grid_spec/0: lon/lat_start anchor to the SW corner,
steps match Grid.step/0, cell counts span CONUS inclusive, and the
spec's cell product equals length(Grid.conus_points/0).
- PollWorker: empty-link list is a no-op, and poll_fn receives host /
community / radio_type straight from the link row.
- Contact submission_changeset: antenna heights are bounded to
[0, 1000] ft, whitespace-only mode normalises without firing an
invalid-mode error, plus a pair of StreamData properties — every
sanctioned band validates and every out-of-list band string is
rejected.
Suite: 2,380 tests + 148 properties (was 2,370 + 146); coverage
70.38% → 71.08% total; credo strict clean.
Changes under lib/ (source):
- iemre_fetch_worker: replace `if transient_failure?/1` boolean predicate
+ `if/else` with a classifier that pattern-matches the error reason
directly, returning `:transient | :permanent`, and dispatch via
`case`. Three-clause head is clearer than the boolean predicate.
- scores_file.extract_points and nexrad_client.extract_box: force `//1`
step on the row/col comprehensions so an image box whose centre
lands outside the raster collapses to an empty iteration instead
of firing Elixir 1.19's ambiguous-range warning. Adds a regression
test for the right-edge box.
Test coverage added:
- Feature wrappers in Backtest.Features (theta_e_jump, shear_at_top,
duct_thickness, best_duct_freq, duct_usable_for_band,
distance_to_front / parallel_to_front stubs, all_features/0).
- NotifyListener handle_info tolerance for malformed payloads +
unknown messages, plus the PubSub broadcast side effect.
- Viewshed.effective_reach_km across every verdict / diffraction
tier / ducting-score tier combination.
- SnmpClient parse_af60_output unit conversions + unknown-column
handling; parse_snmpget_output OID normalisation and junk-line
tolerance.
- HrrrNativeClient native_level_count, native_variables,
native_messages shape, duct_messages / duct_byte_ranges.
- Changeset coverage for GeomagneticObservation, SolarFluxObservation,
SolarXrayObservation, Ionosphere.Observation, HrrrClimatology, and
Metar5minObservation — lifted each from ~50% / 0% to 100%.
- Property tests: GefsClient.dewpoint_from_rh invariants (Td ≤ T,
monotonic in RH, finite across the operating envelope) + idempotent
nearest_run. NexradClient.pixel_to_dbz monotonicity + bounded
output; latlon_to_pixel round-trip for every grid cell.
Also cleared several unrelated compiler/linter warnings:
- Three private test helpers (create_contact, insert_contact,
current_hour) had `\\ %{}` / `\\ 0` defaults that no caller used.
- profiles_file_test bound `dir` in a pattern match without using it.
Suite: 2,359 tests + 146 properties pass (was 2,300 / 139); coverage
70.38% → 70.89%; credo strict clean.
Pure-function invariants that example tests can't express at scale:
- Weather.round_to_iemre_grid/2, HrrrClient.nearest_hrrr_hour/1,
NexradClient.round_to_5min/1, NarrClient.snap_to_analysis_hour/1:
each is idempotent, its output satisfies a grid/slot-membership
predicate, and lies within a bounded distance of the input.
- NarrClient.in_coverage?/1: coverage_end is an exclusive upper
bound; any timestamp after it is out of coverage.
- IemreFetchWorker.backoff/1: result is in [120, 21_600], monotonic
non-decreasing in attempt, equals 120*2^(attempt-1) pre-cap.
- Contact submission_changeset notes field: every non-blank string
up to 2000 chars validates; any longer string is rejected.
22 new properties (139 → 161 including existing); all 2,300 tests
green, credo clean.
Rust workers (prop-grid-rs) that die mid-claim (SIGKILL, OOM, node
drain) leave grid_tasks rows stuck in status='running' forever,
because claim_next uses FOR UPDATE SKIP LOCKED and nothing resets the
orphan. The /status page then shows a permanent spinner with stale
f00/f10/f18 badges — currently 21 rows claimed as far back as
2026-04-20.
GridTaskEnqueuer.reclaim_stale_running/1 flips rows whose claimed_at
is older than 15 minutes back to 'queued'. Rows that have already
burned through 5 claim/reclaim cycles become 'failed' with a
reclaim-orphan error so the next hourly seed can replace them.
Wired into PropagationGridWorker.seed_chain/0 so it runs every :05
cron tick before new rows are seeded.
Also rename the status panel "Retrying" column to "Failed" — it was
always showing terminal `failed` rows, never retrying ones.
IemreFetchWorker now writes an empty stub observation when IEM returns
a permanent-failure status (404, 422), mirroring the existing {:ok, []}
path. The backfill cron's next tick sees the stub via
has_iemre_observation?/3, generates no job, and mark_status!/3 flips
the contact's iemre_status from :queued to :complete — draining the
51 contacts that have been stuck behind cancelled out-of-grid IEMRE
jobs.
Raise hot pod CPU limit 2 → 3 so BEAM gets 3 schedulers online.
Observed run queues of [2, 10, 0, 0, 0, 0] on the 2-scheduler
configuration during the :05 propagation chain, starving /live and
/health probe handlers and tripping intermittent liveness timeouts.
ADIF records frequently include operator commentary in the
<NOTES> field (multi-line, detailed) or <COMMENT> (short
one-liner). Now that contacts have a notes column, pass that
value through the import pipeline instead of dropping it.
Preference is NOTES over COMMENT to match the ADIF 3.1.4 spec's
intended split — NOTES is the richer field. Whitespace-only
values collapse to nil so a program that always emits a blank
NOTES tag doesn't fill every row with an empty string.
Tests cover NOTES capture, COMMENT fallback, NOTES-over-COMMENT
preference, and the absent-field nil case. ADIF upload blurb
lists the new carry-over so users see it before confirming.
Operators want a place to jot observations that aren't captured by
the structured fields — weather anecdotes, propagation mode
commentary, equipment details, band conditions. Add a text column
on contacts plus the two ingest paths users submit through:
* /submit single-contact form gets a 3-row textarea under the
other fields. Optional, max 2000 chars, with a live length cap
via maxlength. Whitespace-only input collapses to NULL so the
column reflects "no notes" rather than an empty string.
* CSV importer recognises a `notes` (or `note`) column via the
existing header_aliases table and flows values straight through
submission_changeset. Sample template gets a matching example
row with embedded commas so the quoted-field round-trip is
exercised on download.
ADIF import and the refinement/refinement-notify paths are out of
scope — users specifically asked for CSV + single-contact. Tests
cover the changeset (accept/blank/over-length), CSV header parsing
(plain + RFC-4180 quoted), and LiveView form submit end-to-end.
When the Rust hrrr-point-worker marks a task as `failed` (upstream
`idx HTTP 404` or wgrib2 decode error on old archive formats), the
corresponding contacts sit at `hrrr_status = :queued` until the
generic 3-day stale-queued reconcile flips them. Speed that up: on
every BackfillEnqueue tick, join contacts to `hrrr_fetch_tasks` by
rounded HRRR hour and flip any `:queued` contact whose cycle is
permanently `failed` to `:unavailable` immediately.
Motivation: a handful of 2018-08-04/05 cycles (t20z/t21z/t15z/t17z)
are genuinely missing `wrfsfcf`/`wrfprsf` files on the NOAA S3
archive — other hours on the same dates are fine, so this is an
upstream data gap rather than a structural pre-2019 issue. Users
see an accurate "Partial" marker on those contacts instead of a
three-day `:queued` stall.
Tested: 4 new cases covering rounding, failed-vs-done, and the
`hrrr not in types` skip branch; full suite 2283/0.
Three bugs were letting mechanism/terrain/radar contacts ping-pong
between :complete and :queued on every backfill cron:
1. enqueue_for_contact/2 now filters out types that are already
:complete on the contact before building jobs. Previously
mark_status!/3 demoted every non-empty jobs_by_type entry back to
:queued, overwriting the :complete that workers had just set.
2. reconcile_stale_queued/1 now also flips mechanism_status and
radar_status back to :complete when the companion data
(propagation_mechanism / contact_common_volume_radar) already
exists. Drains the existing ~14k row backlog on the first post-
deploy tick.
3. IemClient.parse_iemre_json/1 accepts "" and nil so a 200-with-
empty-body IEMRE response doesn't FunctionClauseError the worker
through four retry attempts.
lgdc.uml.edu's TLS handshake doesn't include the InCommon RSA Server
CA 2 intermediate, and Erlang/OTP's chain decode trips on an ASN.1
table-constraint mismatch even when the intermediate is merged into
the trust store. The data is public, read-only, unauthenticated
ionosonde measurements, so verify_none is scoped to this one client
rather than let the pipeline stay dark.
Hot pods were restart-looping every ~25 minutes on liveness probe
timeouts. Root cause: ScoreCacheReconciler mirrored every .prop file
from NFS into ETS, including 7 days of GEFS Day 2-7 forecasts. With
23 bands × 43 valid_times × ~2 MB per grid = 1.95 GB in the
propagation_score_cache table alone; GC sweeps starved the
scheduler enough that /live dropped its 3 s budget.
The /map UI only ever requests the [now-1h, now+18h] window. Share
that bound as Propagation.hot_cache_window/0 and apply it in the
reconciler's disk-scan path plus NotifyListener's post-warm prune.
Long-horizon GEFS files stay on disk and are still served via the
lazy read_from_disk_and_cache path when requested directly.
Adds ScoreCache.prune_outside_window/2 (inclusive bounds) and
updates the reconciler tests to use relative-to-now timestamps
since hardcoded fixture dates now drift out of window.
**Rust workers exit after 30 consecutive claim_next failures.**
Both `prop-grid-rs` and `hrrr-point-worker` ran a `while` loop that
slept 10s and retried forever on `claim_next` errors. If Postgres
went unreachable, pods would stay alive but useless — no circuit
breaker, no liveness probe, just an infinite stream of error logs.
Now we increment a consecutive-failure counter and exit with an
error once it hits 30 (~5 minutes of DB unavailability). k8s
restart policy then cycles the pod, which both triggers an
operator alert and gives us a fresh sqlx pool on the way back up.
A successful claim (Some or None) resets the counter so one
flaky query doesn't poison it.
**BeaconMonitorsTest: drop the Process.sleep(1100).** The test
relied on wall-clock gaps to disambiguate `inserted_at` values on
the `:utc_datetime` (second-precision) timestamps — one full
second of sleep per run, and on shared CI it could still flake
when both inserts landed in the same second. Replaced with an
explicit `Repo.update_all` that back-dates the first row by one
second. 1.1s faster, deterministic.
The main /map timeline was rendering each forecast hour twice
('Now 22:00' + '0h 22:00', '+1h 23:00' + '+1h 23:00', …).
Root cause: during the rolling `.ntms` → `.prop` rename window,
both extensions live on NFS for the same valid_time. `list_score_files`
matches either, so `list_valid_times_raw` mapped 2 files → 2 DateTimes
with no dedupe. Propagation.available_valid_times inherited the dup,
push_timeline pushed it to the JS hook, renderTimeline drew one chip
per entry — hence the visible duplicate.
Fix: `Enum.uniq` before the sort. Regression test writes a `.prop`
file, copies it as the same valid_time's `.ntms` legacy twin, and
asserts list_valid_times returns a single DateTime.
Also land a small refactor that was queued behind this: move the three
pure mechanism display helpers (label/1, badge_class/1, explainer/1)
out of the 2500-line contact_live/show.ex into a new
MicrowavepropWeb.ContactLive.Mechanism module. No behavior change — just
shrinks the LiveView and gives the display mapping a testable home.
Two unrelated production warnings from the same log window:
WeatherFetchWorker raised Postgrex 21000 cardinality_violation on
ASOS upserts when IEM returned two METAR records for the same
timestamp (routine + special at the same minute). Dedupe by
(station_id, observed_at) in upsert_surface_observations, keeping
the last occurrence since IEM's ordering is chronological.
SwpcClient.fetch_xrays warned with the full truncated response
body (~650 KB of GOES X-ray JSON) dumped into the log aggregator
whenever SWPC's CDN cut the response mid-stream. Intercept
Jason.DecodeError in fetch_and_parse and format it short: keep
the byte position and an 80-char preview, drop the rest.
/live fires every 10s per pod; the request line and Sent 200 response
from Plug.Telemetry drowned out real requests. log_level/1 now returns
false for /live too, same treatment /health already got.
Two fixes cover the blank "analysis breakdown" panel the user reported:
1. Bound the point_detail fallback lookback to 24h. Previously
factors_from_fallback_profile would happily use a week-old analysis
profile when no current one existed — now anything older than 24h
returns an empty factor map so the UI can surface "breakdown
unavailable" instead of silently misattributing.
2. Move the Plausible analytics snippet from Layouts.app into
root.html.heex. Full-bleed LiveViews (/map, /contacts/map,
/weather, home) bypass Layouts.app, so the snippet only loaded on
the navbar-wrapped pages. root.html.heex is loaded once per HTTP
document so coverage is now universal.
Added ~30 tests locking both down:
• point_detail fallback: exact-time wins, 24h boundary accepted,
30h-stale rejected, multi-band coverage, missing-cell degrades to
empty factors, default-valid_time path
• analytics_test.exs: Plausible script present on 12 representative
pages, exactly once, loaded async, surviving a login round-trip
Also fixed pre-existing credo issues per standing rule: shortened the
map_live_test ETS match_delete call, extracted a function from the
deeply-nested hrrr_point_enqueuer.enqueue_for_contacts/1 cond, and
swapped Mix.Tasks.Rust.Golden's IO.inspect for Mix.shell().info.
Score-grid files now land at `<band>/<iso>.prop` with a 4-byte "PROP"
magic header. Readers still accept the legacy `.ntms` extension and
"NTMS" magic so a rolling deploy doesn't invalidate any file already
on the shared NFS mount — legacy files age out through normal
retention.
Applied both sides of the seam: Elixir ScoresFile + regex parsers,
Rust scores_file writer + decoder. Updated the comments in all
modules that mention the extension (NotifyListener, Reconciler,
gefs_fetch_worker, map_live) and the runbook diagram. talos5 is a
DB host now, not a Rust worker — corrected the runbook accordingly.
Convert every context/worker upsert that used the catch-all
`:replace_all_except` form to the explicit `{:replace, [:col1, ...]}`
form. Reviewers can now see exactly which columns an upsert overwrites,
and adding a new column to a schema no longer silently opts it into the
update path.
Behaviour is preserved bit-for-bit: each new explicit list contains
every column the old `:replace_all_except` would have overwritten.
Touched:
- Microwaveprop.SpaceWeather (Kp / F10.7 / X-ray shared helper)
- Microwaveprop.Ionosphere (observation upsert)
- NexradWorker, HrrrNativeGridWorker (insert_all grids)
- CommonVolumeRadarWorker, RadarFrameWorker (per-contact stats)
Also pins the conflict behaviour for the ContactCommonVolumeRadar
upsert path in RadarFrameWorkerTest so a future column addition that
isn't reflected in the explicit list fails loudly.
Wrap a Task.Supervisor in a PartitionSupervisor (one partition per
scheduler) and route the four sustained async_stream callers through
`{:via, PartitionSupervisor, {Microwaveprop.TaskSupervisor, self()}}`.
Concurrent work — the hourly PropagationGridWorker pulling f00-f18
HRRR range downloads, the GEFS worker scoring its grid, and the
Recalibrator's 20-way positive/negative fan-out — no longer funnels
through a single supervisor PID.
Light/one-shot Task.async + Task.start sites (hrrr_client surface/
pressure fan-out, application boot warmup, weather fire-and-forget)
are left alone; partitioning only helps under sustained concurrency.
The ingress + Grafana pipeline parses JSON-formatted log lines; Elixir's
default text formatter was the odd format everywhere else, forcing Loki
to fall back to free-text parsing. Swap the `:default` :logger handler's
formatter to `LoggerJSON.Formatters.Basic` when `LOG_FORMAT=json`.
Defaults: `LOG_FORMAT=text` for dev/test (no behaviour change), `json`
for prod. Whitelisted metadata covers the fields we actually correlate
on — `request_id`, `trace_id`, `worker`, `queue`, `job_id`.
OTP 28 / Elixir 1.19 exposes `Logger.Formatter.new/1` but ships no
JSON encoder of its own, so `logger_json` (~> 7.0) carries the rendering.
The test pokes the formatter directly rather than round-tripping through
ExUnit's capture_log, which replaces the handler and so can't see the
installed formatter's output.
PromEx.Plugins.Oban counts job exceptions as a Prometheus counter but
does not tell us which worker/queue failed, whether the job is about to
be retried, or the exception class. This adds a telemetry handler on
[:oban, :job, :exception] that:
- emits a single structured Logger.error with worker, queue, job_id,
attempt, max_attempts, retry_exhausted?, kind, reason class, and
args-keys summary (never values — args can contain PII/tokens);
- dispatches opt-in per-worker recovery callbacks
(on_permanent_failure/1 when retries exhausted,
on_transient_failure/1 otherwise);
- catches callback failures so a bug in one worker's recovery path
cannot detach the handler or break logging for other jobs;
- does not emit a second telemetry event, so PromEx counts are not
double-counted.
Handler is attached from Application.start/2 after the Oban supervisor
starts.
Every pod was rescheduling its sweep on a fixed 60 s `Process.send_after`
tick, so N replicas hit the shared NFS `/data/scores` mount and Postgres
in lock-step every minute. Add a uniform jitter of up to 20 s on top of
the 60 s base (effective range 60-80 s) so per-replica schedules drift
apart.
Expose `next_sweep_interval/0,2` as a public helper so the randomized
window can be tested deterministically with `:rand` draws rather than
mocking timers. Existing reschedule test passes `jitter_max_ms: 0` to
keep its 2 s wait window honest.
Cover the pure-math invariants of the ITU-R P.526-16 diffraction
helpers, the haversine metric on Geo, and the Magnus / dry-adiabat
helpers in the skew-T renderer. 29 properties, one invariant each.
- HrrrClient idx-cache test only invalidated the surface idx URL, but
fetch_profile also fetches a pressure idx. Previous runs' state for
the pressure key decided whether the counter landed at 1 (prior run
cached it, pass) or 2 (cold, fail). Invalidate both + assert 2 total
fetches to reflect the actual code path.
- CsvImportTest deadlocked against other async DataCase tests when
inline Oban child jobs upserted iemre_observations/terrain_profiles
with a shared conflict target. Flip to async: false — same fix as
ContactWeatherEnqueueWorkerTest earlier this session.
- PromEx.Plugins.Oban runs a 5s telemetry_poller that queries the DB,
but its poller PID has no sandbox connection in test and crashed
with DBConnection.OwnershipError on every tick, spamming the log.
Gate the plugin on a config flag and skip it in config/test.exs;
prod behaviour unchanged.
Band-warm failures in the Rust propagation_ready pipeline were only
surfaced via Logger.warning, so Prometheus had no way to see them.
The /weather GridCache hit/miss was likewise invisible while its
/map counterpart (ScoreCache) already reported cache ratio.
- NotifyListener emits [:microwaveprop, :propagation, :notify_listener, :warm]
with %{ok, err} measurements and %{valid_time} metadata after every
propagation_ready notification. Public handle_propagation_ready/1 so
tests can exercise the warm+telemetry path without Postgrex.
- GridCache.fetch/1, fetch_bounds/2, and fetch_point/3 emit
[:microwaveprop, :weather, :grid_cache, :lookup] with a :hit | :miss
metadata key on every ETS lookup.
Wraps the hourly grid_tasks seed with
[:microwaveprop, :propagation, :grid_worker, :perform] so Prometheus
can see duration, success, and failure for the chain-seed cron.
Emits a :seeded event with the queued_tasks count and an :exception
event on GridTaskEnqueuer failure so silent seed errors show up in
the existing PromEx dashboards instead of dying as a lone Logger.info.
IemClient.fetch_asos returns 24-288 rows per station per call, each
previously triggering an individual UPDATE-conflict round-trip via
Weather.upsert_surface_observation/2. On the Turing Pi 2 Postgres
node that per-row latency dominates ingestion time.
Add Weather.upsert_surface_observations/2, a bulk variant that collapses
the rows into a single Repo.insert_all with the same
update-only-when-changed on_conflict predicate and (station_id,
observed_at) conflict target. Switch WeatherFetchWorker to use it.
The Microwaveprop.Weather.GridCache ETS table grew unbounded — each
hourly PropagationGridWorker run plus peer broadcast_put added ~92k
points per forecast hour, so long-lived pods were strong OOM
candidates. GridCache already exposed prune_older_than/1 but nothing
called it.
Adds an Oban worker on the :propagation queue (lower priority than
the grid chain) that prunes entries older than 3 hours. Scheduled
hourly at :15 in dev, prod, and the base crontab.
CommonVolumeRadarWorker and MechanismClassifyWorker were both
returning :ok on upstream failures, so Prometheus job-failure counters
stayed at zero during multi-hour NEXRAD outages and classifier bugs
went unnoticed.
- CommonVolumeRadarWorker: distinguish permanent (4xx) from transient
errors using the same pattern as RadarFrameWorker. Permanent errors
still mark :unavailable + :ok so the backfill stops re-queueing them.
Transient errors return {:error, reason} and leave radar_status alone
so the next retry can succeed.
- MechanismClassifyWorker: keep the row-status :failed update so
operators can see which contacts blew up, but return
{:error, inspect(e)} from the rescue instead of swallowing it as :ok.
- MapLiveTest timestamp indicator: earlier tests in the suite leave
score files in the shared `:propagation_scores_dir` and warm the 5 s
`list_valid_times` cache; clear both in setup so the "No data"
assertion sees the state the test documents.
- PropagationPruneWorkerTest: `old = now - 3h` collides with the
worker's cutoff (also `now - 3h`) once both are truncated to the
second, so they land equal when test + worker run in the same wall
clock second. Shift `old` by an extra -1s for unconditional ordering.
- ContactWeatherEnqueueWorkerTest: flip to async: false — perform/1
runs inline Oban child jobs that upsert rows with a shared conflict
target, and parallel DataCase siblings hit "ShareLock deadlock_detected"
intermittently.
Fixes flagged by the code-reviewer agent's pass over the session's
commits (cc9220b..7b78a25):
- Propagation.warm_cache_and_broadcast/2 now uses ScoresFile.read/2
directly and returns {:ok, :ok} | {:error, :enoent | :invalid_format}.
Previously it called ScoresFile.read_bounds/3 which silently returns
[] on missing/corrupt files, poisoning ScoreCache with an empty grid
that the reconciler couldn't heal.
- NotifyListener.warm_band/2 and ScoreCacheReconciler.warm_one/2 now
pattern-match {:error, reason} and skip (log, return :error) instead
of caching empty. rescue clauses kept as defense-in-depth for
unexpected faults in PubSub.broadcast / ETS writes.
- ScoresFile.extract_points/2 promoted to @doc public — callers that
need to distinguish missing file from empty grid can feed read/2
payloads here themselves.
- Weather.build_grid_cache_row/4: replaced || fallbacks with prefer/3
helper (Map.fetch) so a legitimate persisted ducting_detected: false
is not clobbered by a derived-from-sounding true.
- Weather.hrrr_data_fully_present?/1 @spec tightened from map() to
Contact.t() | field-constrained map, plus is_nil(qso_timestamp)
guard so callers with partial contacts get a clean false rather
than a HrrrClient.nearest_hrrr_hour/1 crash.
- AdminTaskWorker.native_derive bulk-UPDATE: chunk reduced 2000 → 500
and wrapped in try/rescue with per-row fallback on Postgrex errors
so a single bad row doesn't kill the remaining 1999 in its chunk.
- Runbook FM3 rewritten to match the actual code path (no rescue;
explicit {:error, _} pattern match). FM5 clarifies the surviving
PropagationGridWorker is a cron-fired seed worker, not a fallback
compute path.
- Dialyzer: strict flags added (:error_handling, :unknown,
:unmatched_returns, :extra_return, :missing_return). Baseline
emitted 130 warnings, mostly discarded Task.start/PubSub/Logger
returns; a follow-up will tighten those and backfill @specs.
New tests:
- ScoreCacheReconciler GenServer lifecycle: run_on_start true/false,
interval_ms rescheduling, info-level log line.
- Weather.hrrr_data_fully_present?/1: nil qso_timestamp returns false.
- Weather.build_grid_cache_rows/2: explicit ducting_detected: false
on profile beats derived true from sounding params.
- RadarFrameWorker: pins the NexradClient "NEXRAD n0q HTTP <code>"
error string contract so permanent_error?/1 classification doesn't
silently regress if the client's error format changes.
1. GridCache: auto-release fill lock when the claimer process crashes.
claim_fill/1 + release_fill/1 go through the GenServer so the
server can Process.monitor the caller and clean up the ETS entry
on :DOWN. Clear/0 now resets both the data table and the lock
table. Fixes a latent bug where a crashed fill leaked the lock
indefinitely, preventing every subsequent /weather mount for that
valid_time from claiming and leaving cache cold.
2. RadarFrameWorker: distinguish permanent vs transient fetch errors.
404 from the IEM n0q archive is permanent (file will never exist)
and marks contacts :unavailable as before. Any other error shape
(5xx, timeout, transport failure) now returns {:error, reason}
so Oban retries — previously those also pinned contacts at
:unavailable after a transient outage.
3. AdminTaskWorker.native_derive: replace per-row Repo.update_all
(N round-trips + N fsyncs) with one UPDATE ... FROM unnest(...)
per 2000-row batch. For the 10k-profile budget this is one
network round trip per chunk instead of 10k, and one fsync per
chunk instead of 10k. Restructured the clause to separate
derivation (pure) from persistence (I/O).
All three changes are test-covered (grid_cache_test auto-release
test, radar_frame_worker_test 5xx + transport tests, existing
admin_task_worker_test native_derive coverage exercises the new
bulk path). Also drops the scorer_diff no-op test that was
verifying the clause removed in 61da51c.
- AdminTaskWorker: remove scorer_diff no-op clause (propagation_scores table
was dropped pre-cutover; queue is drained).
- Weather: remove load_weather_grid_from_db/1 fallback — Rust writes
ProfilesFile for every forecast hour, so the DB path was unreachable
in practice and loaded stale pre-cutover data when it fired.
- Weather.build_grid_cache_row/4 now prefers persisted fields from the
ProfilesFile profile map (surface_refractivity, min_refractivity_gradient,
ducting_detected, duct_characteristics) over re-deriving from the raw
profile, matching the old DB-path semantics.
- NotifyListener: wrap per-band warm in try/rescue so one corrupt or
mid-rename .ntms file doesn't abort the remaining bands. Failures log
a warning with band + valid_time; successes still emit one info line.
- weather_grid_test: rewrite insert_hrrr_profile helper to write
ProfilesFile directly and drop the "only returns points on 0.125 grid"
test (filter was a property of the deleted DB fallback; Rust writes
only grid points by construction).
Column name now matches the type key used everywhere else:
:hrrr -> hrrr_status, :weather -> weather_status,
:terrain -> terrain_status, :mechanism -> mechanism_status
Drops the @status_column_overrides remap in BackfillEnqueueWorker
(status_column/1 is now a straight :"#{type}_status" derivation) and
the long-form references in Contact, ContactWeatherEnqueueWorker,
and MechanismClassifyWorker. Migration is a metadata-only RENAME
COLUMN + RENAME INDEX — safe to land in prod (no table rewrite,
brief catalog lock only).
Periodic 60s sweep re-warms ScoreCache from /data/scores whenever
{band, valid_time} pairs are on disk but absent in local ETS.
Covers the three silent-fail modes of the NOTIFY/LISTEN path:
dropped notifies on reconnect, missing listener, and stale NFS
reads. Each pod reconciles its own cache — no PubSub fan-out,
no coordination required. Rescue on File.Error handles the
rare window where an atomic-rename write races the reader.
docs/runbook_propagation_pipeline.md names every failure mode
in the Rust↔Elixir seam and what recovers it.
