`handle_event("switch_tab", ...)` ran `String.to_existing_atom/1` on
the raw `tab` value from the browser, so a stale, forged, or
mistyped client event with any unknown string crashed the LiveView
with `ArgumentError: not an already existing atom`.
Compares against an explicit `@valid_tabs` allowlist (`:single`,
`:csv`, `:adif`) — known values switch the active tab, anything
else is ignored.
`apply_edit_to_contact/2` is the path for both owner direct-edits and
admin reviewed-edits. The public contact-map + total-count + gzipped
controller payload caches are built from `private == false` rows, but
only the insert path was clearing them — every other update left
/api/contacts/map and /contacts/map serving the previous snapshot for
up to 10 minutes. A `private: false -> true` flip on a contact already
present in the cached payload was a temporary privacy leak; grid
corrections, callsign changes, and flagged_invalid flips silently
diverged from the live row.
Extracts the three `Cache.invalidate` calls into
`invalidate_contact_map_caches/0` and runs it from both
`apply_edit_to_contact/2` and the existing insert path so the public
view stays in sync.
The /contacts/:id edit form always submits the `private` checkbox, but
`current_value/2` had no `"private"` clause and fell through to the
catch-all `nil`. The Contact schema defaults `private: false`, so an
unchanged box compared `false != nil` and was always tagged as a
change — for non-owners that meant a spurious pending review queued
on every save, and for owners/admins it triggered a no-op direct
update.
Adds the missing clause so the diff sees boolean-vs-boolean and
correctly returns `:no_changes` when the box is untouched.
The /contacts/:id edit form pre-fills qso_timestamp via
`Calendar.strftime(ts, "%Y-%m-%d %H:%M")` and submits whatever the
user typed back unchanged. Two failure modes resulted:
1. `diff_against_contact/2` compared that string against the stored
`%DateTime{}`, never matched, and treated every untouched submit
as a modification.
2. `build_contact_changes/2` then ran `DateTime.from_iso8601/1` on
the space-separated, no-seconds, no-Z form and crashed with
`MatchError` on `{:error, :invalid_format}` — direct owner/admin
submits raised instead of saving.
Adds `normalize_timestamp_field/2` to `normalize_proposed/1` that
parses both the form's `"YYYY-MM-DD HH:MM[:SS]"` and the strict
`"YYYY-MM-DDTHH:MM:SSZ"` shapes into a UTC `%DateTime{}` (with
`NaiveDateTime.from_iso8601` doing the heavy lifting after a small
seconds-padding pass). Empty strings drop the field; unparseable
input drops it too so a typo becomes "no changes" rather than a
500.
Bandit's dual-stack listener delivers IPv4 connections to the app as
IPv4-mapped IPv6 tuples (`::ffff:a.b.c.d`). The trust check compared
that 128-bit form against our IPv4 trusted_proxies CIDRs and never
matched, so cf-connecting-ip was ignored on every cloudflared-relayed
request — every visitor logged as the cloudflared sidecar pod IP
(rendered as `0:0:0:0:0:65535:2804:101` etc.).
Collapses the mapped tuple to plain IPv4 before the trust check and
records it that way on the conn, so logs and session storage see the
real client IP that Cloudflare set in cf-connecting-ip.
Adds a `layer` query parameter to `/weather`. Loading
`/weather?layer=temp_700mb` boots straight into that overlay; clicking
a different layer pill push_patches the URL so the active selection is
shareable and survives a refresh. Unknown layer ids fall back to the
default (temperature) instead of crashing.
PropagationGridWorker hard-coded `now-2h` for run_time selection on
the conservative theory that NOAA always finishes publishing within
two hours. In practice f18 is on the mirror by HH:55–HH:00 of the
next hour, so the cron at HH:05 was always picking a cycle that was
already an hour stale.
`HrrrClient.cycle_available?/1` HEADs the wrfprsf18 idx (the slowest
file in the cycle) — a 200 means every earlier forecast hour is also
on disk. `pick_run_time/1` probes `now-1h` first; if the probe
returns true we use that cycle, otherwise we fall back to the
existing `now-2h` slot. Test hook via :hrrr_cycle_available_fn env
keeps the behaviour fully exercisable without hitting NOAA.
End-to-end: weather/score data lag drops from ~2 h to ~1 h on every
hour where NOAA delivers on time. The fallback path keeps the chain
running on slow-publish hours.
850 mb T alone shows that warm air is present aloft but doesn't reveal
whether it's the textbook cap structure — a warm 700 mb above an EML
plume. The three new layers expose the canonical capping diagnostics
visually:
* T @ 700 mb — ≥10 °C is the southern Plains "moderate cap" threshold
* Td @ 700 mb — wide T-Td depression at 700 mb signals the EML plume
* Lapse 850→700 — steep mid-layer lapse rate (≥7 °C/km) over a moist
boundary layer is the cap mechanism itself
All three derive purely from the existing pressure-level profile (Rust
already fetches up through 700 mb), so no Rust pipeline changes needed.
ScoreCache held 437 entries × ~1.94 MiB each (~850 MiB per pod) of
{band_mhz, valid_time} grids — data already on NFS as compact .prop
files. NotifyListener and ScoreCacheReconciler both eagerly materialised
every band into ETS on each Rust completion / 60s sweep, so 5 hot
replicas wasted ~4.2 GiB of redundant cache and OOMed at 6 GiB.
Bound the cache to 32 entries with eviction by oldest valid_time, drop
the eager warm loops, and let LiveView callers lazy-fill via the
existing ScoresFile read path. ScoreCacheReconciler had no remaining
purpose and is deleted; runbook + prom_ex counters updated to match.
Steady-state cache footprint ~60 MiB per pod instead of ~850 MiB.
The Rust f01..f18 pipeline writes ProfilesFile cells, and the Rust
hrrr_points worker writes hrrr_profiles rows, with profile entries
keyed "pres_mb" / "hght_m" / "tmpc" / "dwpc" (string keys pre-
atomization; atom keys post-ProfilesFile.read). Elixir's
SoundingParams.derive and WeatherLayers.sort_profile filtered on
"pres" / "hght" — so every derived field returned nil for any
Rust-provided profile.
Visible symptoms on /weather: N-gradient, Refractivity, T @ 850mb,
Td @ 850mb, Lapse Rate, Inversion, Inv. Base all rendered no overlay.
For per-contact analysis, min_refractivity_gradient from the Rust
HRRR point worker's rows silently dropped.
- SoundingParams.normalize_profile_entry/1: single-source normalizer
accepting legacy, Rust-string, and Rust-atom shapes; returns the
canonical "pres"/"hght"/"tmpc"/"dwpc"/"drct"/"sknt" shape.
- SoundingParams.derive/1 and WeatherLayers.sort_profile/1 both run
every entry through it before the rest of the existing logic.
- Weather.build_grid_cache_row/4 now falls back to native_min_gradient
when SoundingParams can't derive one (surface-only profiles).
Also:
- Default /weather overlay changed to Temperature (LiveView assign +
JS fallback in the hook's dataset default) per user request.
Extends the weather-status reconciler with symmetric sweeps for HRRR
and IEMRE. Same failure mode: the worker writes data to the shared
table but has no back-pointer to the contact that triggered the
fetch, so contact-level status never flips unless a user views the
page.
- Weather.reconcile_iemre_statuses/0: iterates :queued contacts;
flips :complete when every path point has an iemre_observations
row at the rounded 0.125° grid cell for the qso_timestamp's date.
- Weather.reconcile_hrrr_statuses/0: same shape; flips when
hrrr_data_fully_present?/1 holds.
- Both run alongside the existing weather sweep at the tail of
ContactWeatherEnqueueWorker.perform/1.
Kept in Elixir (not SQL) because contact_path_points/1 emits 1-3
points per contact and grid-rounding each in pure SQL is awkward.
The stuck-at-steady-state count is always tiny (<20), so the
iteration is cheap.
2,019 contacts sat permanently in weather_status=:queued even after
their ASOS data had been ingested. The only code path that flipped
:queued → :complete was MicrowavepropWeb.ContactLive.Show on page
view — contacts that nobody visited stayed :queued forever.
Adds Weather.reconcile_weather_statuses/0, a single SQL UPDATE that
flips every :queued contact whose ±2h / 150km window now contains
at least one surface observation. Called at the tail of the
existing ContactWeatherEnqueueWorker cron so the correction runs
regardless of whether anyone opens the UI.
Visible effect: the status page's "Weather done" count (previously
capped at 79,975 while data kept landing) will converge on the real
total after the next cron tick.
2838 tests + credo green.
Replaces per-QSO-endpoint asos_batch granularity with per-station-day
granularity. Each contact in the same UTC day near the same station
now produces identical unique-args and collapses to ONE Oban job, so
the backlog shrinks as cross-contact duplicates dedup.
Each fetch also covers a full 24h window (vs the previous 4h) so one
IEM request returns ~24 hourly observations instead of ~1-2. The
effective bytes-per-request is 10x higher, amortizing the
IemRateLimiter gap + 429 retry tail across far more data.
Changes:
- Weather.station_day_covered?/2 + station_day_pairs_covered/1 — cheap
UTC-day coverage checks for the enqueuer + worker skip paths.
- WeatherFetchWorker gains an "asos_day" clause that fetches the full
UTC day for one station and upserts. Skips if the day is already
covered in the DB.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one asos_day
job per (station_id, UTC date) pair. ±2h windows crossing midnight
enumerate both dates.
- Microwaveprop.Weather.RebatchAsos.to_day_jobs/1 migrates any
already-queued asos/asos_batch jobs into the new shape. Idempotent;
supports dry_run.
- asos_batch worker clause retained for jobs already in flight at
deploy time.
2835 tests + credo green.
Mix.Task.run isn't available in a compiled release, so the in-prod
`bin/microwaveprop eval 'Mix.Tasks.Weather.RebatchAsos.run(...)'`
dispatch crashes. Moves the real logic into a plain module
(`Microwaveprop.Weather.RebatchAsos`) that both the Mix task and a
release eval can call. Uses IO.puts over Mix.shell for the same
reason.
- IemRateLimiter gains AIMD-style adaptive spacing. signal_429/0
widens the current gap (*= 1.5, capped at max_interval_ms default
10s); signal_success/0 narrows it back toward the configured base
(*= 0.92, floored at interval_ms). Self-tunes to IEM's moving
ceiling without needing the manual "safe for 4 pods" constant.
- IemClient now routes every response through a central handle_response
helper that fires the widen/narrow feedback signals, eliminating the
four near-identical case blocks.
- Mix.Tasks.Weather.RebatchAsos collapses any already-queued single-
station "asos" jobs into the batched "asos_batch" shape the
enqueuer now emits, so the pending backfill queue converts to the
new per-request-efficient path instead of draining at the old rate.
Idempotent; supports --dry-run.
2833 tests + credo green.
IEM's ASOS CSV endpoint accepts multiple station= params in a single
request. Prior code enqueued one WeatherFetchWorker job per nearby
station per QSO endpoint — N jobs × 1 station each — which paid the
IemRateLimiter's 1500ms gap and the IEM 429-retry tail N times per
contact.
Changes:
- IemClient.asos_url/3 accepts a list of station codes.
- IemClient.fetch_asos_batch/3 fetches N stations in one call and
returns rows grouped by station_code (with absent codes filled as
empty lists so callers can stub them).
- parse_asos_csv/1 now exposes the station_code column it was
previously discarding.
- WeatherFetchWorker gains an "asos_batch" fetch_type clause that
unpacks rows per (station_id, station_code), upserting or stubbing
each. The single-station "asos" clause stays for already-queued
retryable jobs.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one batch
job per (lat, lon, 4h window) covering every uncovered nearby
station (sorted for deterministic unique-args).
Expected effect on backfill: ~10-20x fewer IEM requests per contact
enrichment cycle, matching drop in 429 retry traffic.
Final coverage round: 84.44% → 86.18% (target was 85%).
Two lib_ml/ tasks were stranded at 0% (1035 combined lines) because
the qsos → contacts table rename left stale refs in
PropagationAnalyze's SQL. Fixed:
- `FROM qsos q` → `FROM contacts q`
- `terrain_profiles tp ON tp.qso_id = q.id` → `tp.contact_id = q.id`
Adds test/mix/tasks/propagation_ml_tasks_test.exs with 4 tests:
- PropagationAnalyze end-to-end against empty DB (walks every section
header: correlation, binned factor, interaction, close).
- PropagationAnalyze with seeded contact + matching HRRR at both
endpoints yields a 1-row dataset (exercises derive_averages +
format_band + median + percentile helpers).
- PropagationAnalyze with a pre-2016-06-30 contact is excluded by the
WHERE clause.
- PropagationTrain on empty hrrr_profiles raises the expected Nx
"cannot build empty tensor" error after walking header + load path.
215 → 221 properties, 2812 → 2846 tests, 0 failures.
- Bump render_async/2 timeout to 2_000ms across contact_live_test.
Default 100ms races when the suite runs on a loaded box — ContactLive.
Show fans out 8 hydration tasks at mount and sporadically missed the
window even on a quiet DB.
- Telemetry metrics-shape assertion uses Enum.any?/1 instead of the
`metrics != []` comparison Elixir 1.19's type system flags as a
distinct-type comparison (the function's non-empty return type).
Passes `capture_log: true` to ExUnit.start/1 so log output emitted
during a passing test stays in the per-test capture buffer instead of
stdout. Drops ~360 lines of noise from the suite: Postgrex teardown
disconnects, expected worker failures, NexradClient 404 stubs,
NotifyListener warm-skip messages, SNMP poll failures, PromEx tag-
drop warnings, and OTel-handler boot errors. Failing tests still
surface their full logs on report, so debugging is unchanged.
Side-effect fixes:
- LoggerFormatTest opts out via @moduletag capture_log: false — it
patches the :default :logger handler and capture_log hot-swaps the
same handler, so the setup lookup would 404.
- TelemetryTest.start_link/1 unlinks the supervisor before the brutal
kill so the exit signal doesn't propagate to the test process.
- Drop unused `import Ecto.Query` in a top_hours mix-task test.
- Drop unused default-arg `attrs \\ %{}` on create_contact/1 in
contact_map_live_test (every caller passes attrs explicitly).
Remaining noise is ~3 lines from oban_pro vendored-dep @impl warnings
(require an upstream patch) plus rare intermittent async teardowns.
Adds 30 more test cases (2594 → 2624 tests) targeting the last
low-coverage hot spots.
Notable additions:
- ContactLive.Show fully-hydrated render path (seeds HrrrProfile,
TerrainProfile, ContactCommonVolumeRadar, Sounding, SurfaceObservation,
IemreObservation) exercises refresh_computed, build_propagation_analysis,
build_data_sources, compute_elevation_profile, and the downstream
render branches they unlock.
- handle_info :terrain_ready and :sounding_fetched paths for same-contact
and unrelated-contact dispatch.
- Flagged-invalid badge + line-through header style.
- Owner-label button path (Edit vs Suggest Edit).
- NotifyListener init/1 subscribes on schedule; handle_info(:subscribe)
succeeds against the live Repo.
- RecalibrateScorer runs cleanly against an empty corpus via the
Recalibrator insufficient-data fallback.
- HrrrNativeClient.extract_native_profiles/2 tolerates malformed input
without crashing.
Adds ~600 new test cases (2013 → 2613 tests; 170 properties; 0
failures) across 12 new test files plus expansions of eight existing
files. Big lifts per module:
ContactLive.Mechanism 33 → 100%
MetricsPlug 54 → ~100%
Microwaveprop.Release 15.8 → 70%+
Telemetry 38 → 76%
HrrrNativeGridWorker 27.7 → 76%
ContactLive.Show 34 → 48% (handler + render branches)
Admin.ContactEditLive 49.7 → 70%+
GefsFetchWorker 35.8 → ~55%
IonosphereFetchWorker 56.3 → 75%
PathLive 58.9 → 67%
WeatherMapLive 66.9 → 80.2%
RoverLive 0 → 70.3%
ContactMapLive 59.2 → 89.8%
ContactMapController 0 → 100%
Mix tasks (Rust.Golden, Notebook, Backtest, HrrrBackfill,
HrrrClimatology, HrrrNativeBackfill, NexradBackfill,
RadarBackfill, ImportContestLogs, PropagationGrid,
ResetEnrichment, Hrrr.PurgeGridPoints) 0 → ~60-100%
Two incidental fixes made while adding tests:
- ContactLive.Show.handle_event("toggle_flag", ...) was passing
socket.assigns.current_scope to admin?/1 instead of socket.assigns,
so admins never matched the pattern and every toggle ran the
"Admins only." flash branch. Flag now toggles for admins again.
- Commercial.PollWorker.fetch_weather/1 promoted from private to
@doc'd public so the IEM-fetch + ASOS-upsert path can be tested
directly without driving perform/1 through live SNMP (which was
timing out for ~50 s per test in the earlier attempt).
Stable property-test additions cover the dewpoint-from-RH monotonicity,
nearest_run/1 idempotence, and the ionosphere station envelope.
Replaces the SVG schematic on /eme with a textured WebGL scene: NASA
Blue Marble Earth, real-size Moon, outbound beam, dashed return ray
to the sub-lunar point, and a red shading on the anti-moon hemisphere
that marks who can't see the Moon (outside the bounce footprint).
Three.js (~680 kB) now lives in its own chunk; esbuild --splitting
keeps it out of the main bundle so it only loads when a user hits
/eme. Main app.js drops from 1.6 MB to 922 kB.
Also swaps the Elevation/SNR rows so the badge comes before the number.
New /eme page — antenna aiming + link budget for moonbounce.
Given a station (callsign / grid / lat,lon), TX power, antenna gain,
detection bandwidth, and system Tsys, the page shows in real time:
- Moon azimuth / elevation from the observer (ticks every 10 s)
- Slant range to the Moon
- EIRP breakdown: TX(dBm) + gain(dBi) = EIRP(dBm)
- Path-loss decomposition (Earth → Moon → Earth):
free-space spreading (×2)
− moon reflection gain 10·log(4π σ / λ²) where σ = ρ·π·R²
= round-trip path loss
- Band-dependent lunar albedo ρ (VHF ≈ 0.065, microwave ≈ 0.08,
mm-wave ≈ 0.02) surfaced as a named line item with the current
ρ shown to three decimals
- Received power, thermal noise floor, and SNR with a badge that
classifies the margin (strong / marginal / below noise)
- Self-echo Doppler shift with its radial-velocity driver
New modules:
- `Microwaveprop.Moon` — low-precision lunar ephemeris
(Astronomical Almanac Sec. D.4 truncated series). Provides
`julian_day/1`, `geocentric/1`, `observer_position/3`, and
`radial_velocity_km_s/3`. Accurate to ~0.3° position / ~200 km
distance — well inside any amateur EME-dish beamwidth.
- `Microwaveprop.Propagation.Eme` — path-loss helpers:
`moon_albedo/1` (band lookup), `moon_rcs_m2/1`,
`fspl_round_trip_db/2`, `moon_reflection_gain_db/2`,
`path_loss_db/3`, `received_power_dbm/1`, `noise_floor_dbm/2`,
`snr_db/1`, `doppler_shift_hz/2`.
Wired into the main nav ("EME" button) and the router between
/path and /rover.
Tests: 3 properties + 41 unit tests cover JD conversion, moon
position envelope, band-dependent albedo lookup, decomposition
identity, linearity/monotonicity of path loss and Doppler, and a
LiveView integration test against the rendered DOM.
Full suite: 2,460 tests + 170 properties; credo strict clean.
- identify_var: full round-trip across every known (discipline,
category, number) triple, plus a property that any triple outside
the 12-entry lookup falls back to the "cat:num" default.
- identify_level: hybrid-level and entire-atmosphere branches, plus
two properties — pressure levels always render as
`div(value_pa, 100) <> " mb"` and unknown surface types always
render as `unknown:<type>:<value>` (generator samples only from
types outside the 6-entry lookup set).
- sign_magnitude_16: round-trip property over [-32_767, 32_767] (±0
both decode to 0 as intended) and a monotonicity property inside
a single sign band.
Suite: 2,419 tests + 164 properties (was 2,416 + 159); credo strict
clean.
Root cause of 14 stuck hrrr-queued contacts: hrrr_point_rs's CONUS-only
grid silently writes zero profiles for out-of-bounds points (UK,
Winnipeg, Alberta, >50° lat), but the fetch task still completes
successfully. ContactWeatherEnqueueWorker's hrrr_placeholder_jobs
always emitted a queued placeholder for any contact with a pos1,
which meant mark_hrrr_status! re-flagged :queued every backfill tick
forever.
Introduce Propagation.Grid.contains?/1 as the canonical in-CONUS
check (inclusive on all four edges, nil-safe). Use it in two places
of ContactWeatherEnqueueWorker:
- hrrr_placeholder_jobs/1 now returns [] for OCONUS contacts, same
as pre-2014 / already-complete ones.
- mark_hrrr_status!/3 empty-list clause now branches on Grid.contains?
first (→ :unavailable) before falling through to the existing
NARR-coverage / :complete logic.
Next backfill tick will flip the 14 stuck OCONUS contacts in prod to
:unavailable automatically. The remaining 13 "CONUS" stuck ones are
right at the 50° edge (lat=50.1875 rounds outside the box) — same
reconciliation path.
Coverage additions:
- Grid.contains?/1 — corner-inclusive, OCONUS positive/negative
fixtures based on the actual stuck-contact pos1 values, nil/missing-
key/nil-value handling.
- enqueue_for_contact regression tests: pos1 in the UK and pos1 just
north of the lat cap both land at :unavailable, not :queued.
Suite: 2,416 tests + 159 properties (was 2,409 + 159); credo strict
clean.
- Format.number/1: boundary cases (0, 999, 1_000), float rounding,
nil/atom fallback to to_string, plus two properties — comma
insertion is lossless round-trip and the comma count follows
`div(digits - 1, 3)` for |n| ≥ 1000.
- Format.bytes/1: per-unit boundary cases plus a monotonicity
property (parse-back preserves ordering across the B/KB/MB/GB
transitions).
- Instrument.span/2,3 (0→100%): wraps the block in a telemetry span
returning its value, defaults metadata to %{}, emits the
:exception event and re-raises on a raise inside the span, and
prefixes every event suffix with :microwaveprop.
- SolarClient: short-line / empty-line / blank-day handling for
parse_gfz_row; blank-line + short-line tolerance in parse_gfz_file;
filter_since boundary semantics (>= is inclusive, empty list
passes through).
Suite: 2,397 tests + 159 properties (was 2,382 + 155); coverage
71.12% → 71.16%; Format + Instrument now at 100%; credo strict clean.
- PageController: cover the legacy `/qsos` → `/contacts` redirect pair
(both the index and the id-preserving per-resource redirect), which
had no coverage since the route rename.
- Scorer property tests:
- f_to_c / c_to_f are inverses up to float epsilon, both propagate
nil, and the C→F slope is exactly 5/9.
- wind_speed_kts is non-negative, propagates nil on either axis,
and matches sqrt(u²+v²)·1.94384 (the 5-digit kts factor the
scorer uses).
- precip_to_rate_mmhr is non-negative, 0 for nil/zero/negative.
- dbz_to_rain_rate_mmhr is 0 below 5 dBZ (and for nil), and
monotonically non-decreasing above — guards against accidental
inversion of the Marshall-Palmer lookup.
Suite: 2,382 tests + 155 properties (was 2,380 + 148); coverage
71.08% → 71.12%; credo strict clean.
- SwpcClient: unparseable time_tag rows drop silently, integer / string
numerics both cast, already-decoded list bodies (Req auto-decode) are
accepted, non-JSON-array bodies return {:error, :not_a_list}, X-ray
rows whose energy isn't the long wavelength are filtered, and the
misspelled-upstream electron_contaminaton key flags
electron_contaminated.
- UwyoSoundingClient: every month abbreviation routes to its correct
number, bogus month tokens fall through to an empty result, and
short-of-fixed-width lines are skipped without raising.
- Grid.wgrib2_grid_spec/0: lon/lat_start anchor to the SW corner,
steps match Grid.step/0, cell counts span CONUS inclusive, and the
spec's cell product equals length(Grid.conus_points/0).
- PollWorker: empty-link list is a no-op, and poll_fn receives host /
community / radio_type straight from the link row.
- Contact submission_changeset: antenna heights are bounded to
[0, 1000] ft, whitespace-only mode normalises without firing an
invalid-mode error, plus a pair of StreamData properties — every
sanctioned band validates and every out-of-list band string is
rejected.
Suite: 2,380 tests + 148 properties (was 2,370 + 146); coverage
70.38% → 71.08% total; credo strict clean.
Changes under lib/ (source):
- iemre_fetch_worker: replace `if transient_failure?/1` boolean predicate
+ `if/else` with a classifier that pattern-matches the error reason
directly, returning `:transient | :permanent`, and dispatch via
`case`. Three-clause head is clearer than the boolean predicate.
- scores_file.extract_points and nexrad_client.extract_box: force `//1`
step on the row/col comprehensions so an image box whose centre
lands outside the raster collapses to an empty iteration instead
of firing Elixir 1.19's ambiguous-range warning. Adds a regression
test for the right-edge box.
Test coverage added:
- Feature wrappers in Backtest.Features (theta_e_jump, shear_at_top,
duct_thickness, best_duct_freq, duct_usable_for_band,
distance_to_front / parallel_to_front stubs, all_features/0).
- NotifyListener handle_info tolerance for malformed payloads +
unknown messages, plus the PubSub broadcast side effect.
- Viewshed.effective_reach_km across every verdict / diffraction
tier / ducting-score tier combination.
- SnmpClient parse_af60_output unit conversions + unknown-column
handling; parse_snmpget_output OID normalisation and junk-line
tolerance.
- HrrrNativeClient native_level_count, native_variables,
native_messages shape, duct_messages / duct_byte_ranges.
- Changeset coverage for GeomagneticObservation, SolarFluxObservation,
SolarXrayObservation, Ionosphere.Observation, HrrrClimatology, and
Metar5minObservation — lifted each from ~50% / 0% to 100%.
- Property tests: GefsClient.dewpoint_from_rh invariants (Td ≤ T,
monotonic in RH, finite across the operating envelope) + idempotent
nearest_run. NexradClient.pixel_to_dbz monotonicity + bounded
output; latlon_to_pixel round-trip for every grid cell.
Also cleared several unrelated compiler/linter warnings:
- Three private test helpers (create_contact, insert_contact,
current_hour) had `\\ %{}` / `\\ 0` defaults that no caller used.
- profiles_file_test bound `dir` in a pattern match without using it.
Suite: 2,359 tests + 146 properties pass (was 2,300 / 139); coverage
70.38% → 70.89%; credo strict clean.
Pure-function invariants that example tests can't express at scale:
- Weather.round_to_iemre_grid/2, HrrrClient.nearest_hrrr_hour/1,
NexradClient.round_to_5min/1, NarrClient.snap_to_analysis_hour/1:
each is idempotent, its output satisfies a grid/slot-membership
predicate, and lies within a bounded distance of the input.
- NarrClient.in_coverage?/1: coverage_end is an exclusive upper
bound; any timestamp after it is out of coverage.
- IemreFetchWorker.backoff/1: result is in [120, 21_600], monotonic
non-decreasing in attempt, equals 120*2^(attempt-1) pre-cap.
- Contact submission_changeset notes field: every non-blank string
up to 2000 chars validates; any longer string is rejected.
22 new properties (139 → 161 including existing); all 2,300 tests
green, credo clean.
Rust workers (prop-grid-rs) that die mid-claim (SIGKILL, OOM, node
drain) leave grid_tasks rows stuck in status='running' forever,
because claim_next uses FOR UPDATE SKIP LOCKED and nothing resets the
orphan. The /status page then shows a permanent spinner with stale
f00/f10/f18 badges — currently 21 rows claimed as far back as
2026-04-20.
GridTaskEnqueuer.reclaim_stale_running/1 flips rows whose claimed_at
is older than 15 minutes back to 'queued'. Rows that have already
burned through 5 claim/reclaim cycles become 'failed' with a
reclaim-orphan error so the next hourly seed can replace them.
Wired into PropagationGridWorker.seed_chain/0 so it runs every :05
cron tick before new rows are seeded.
Also rename the status panel "Retrying" column to "Failed" — it was
always showing terminal `failed` rows, never retrying ones.
IemreFetchWorker now writes an empty stub observation when IEM returns
a permanent-failure status (404, 422), mirroring the existing {:ok, []}
path. The backfill cron's next tick sees the stub via
has_iemre_observation?/3, generates no job, and mark_status!/3 flips
the contact's iemre_status from :queued to :complete — draining the
51 contacts that have been stuck behind cancelled out-of-grid IEMRE
jobs.
Raise hot pod CPU limit 2 → 3 so BEAM gets 3 schedulers online.
Observed run queues of [2, 10, 0, 0, 0, 0] on the 2-scheduler
configuration during the :05 propagation chain, starving /live and
/health probe handlers and tripping intermittent liveness timeouts.
ADIF records frequently include operator commentary in the
<NOTES> field (multi-line, detailed) or <COMMENT> (short
one-liner). Now that contacts have a notes column, pass that
value through the import pipeline instead of dropping it.
Preference is NOTES over COMMENT to match the ADIF 3.1.4 spec's
intended split — NOTES is the richer field. Whitespace-only
values collapse to nil so a program that always emits a blank
NOTES tag doesn't fill every row with an empty string.
Tests cover NOTES capture, COMMENT fallback, NOTES-over-COMMENT
preference, and the absent-field nil case. ADIF upload blurb
lists the new carry-over so users see it before confirming.
Operators want a place to jot observations that aren't captured by
the structured fields — weather anecdotes, propagation mode
commentary, equipment details, band conditions. Add a text column
on contacts plus the two ingest paths users submit through:
* /submit single-contact form gets a 3-row textarea under the
other fields. Optional, max 2000 chars, with a live length cap
via maxlength. Whitespace-only input collapses to NULL so the
column reflects "no notes" rather than an empty string.
* CSV importer recognises a `notes` (or `note`) column via the
existing header_aliases table and flows values straight through
submission_changeset. Sample template gets a matching example
row with embedded commas so the quoted-field round-trip is
exercised on download.
ADIF import and the refinement/refinement-notify paths are out of
scope — users specifically asked for CSV + single-contact. Tests
cover the changeset (accept/blank/over-length), CSV header parsing
(plain + RFC-4180 quoted), and LiveView form submit end-to-end.
When the Rust hrrr-point-worker marks a task as `failed` (upstream
`idx HTTP 404` or wgrib2 decode error on old archive formats), the
corresponding contacts sit at `hrrr_status = :queued` until the
generic 3-day stale-queued reconcile flips them. Speed that up: on
every BackfillEnqueue tick, join contacts to `hrrr_fetch_tasks` by
rounded HRRR hour and flip any `:queued` contact whose cycle is
permanently `failed` to `:unavailable` immediately.
Motivation: a handful of 2018-08-04/05 cycles (t20z/t21z/t15z/t17z)
are genuinely missing `wrfsfcf`/`wrfprsf` files on the NOAA S3
archive — other hours on the same dates are fine, so this is an
upstream data gap rather than a structural pre-2019 issue. Users
see an accurate "Partial" marker on those contacts instead of a
three-day `:queued` stall.
Tested: 4 new cases covering rounding, failed-vs-done, and the
`hrrr not in types` skip branch; full suite 2283/0.
The log_level(%{method: \"HEAD\"}) clause never matched because
Plug.Head rewrites the method to \"GET\" before Plug.Telemetry's
register_before_send callback fires, so by log time conn.method is
\"GET\". Stash the original method in private before Plug.Head runs
and key the filter off it.
Three bugs were letting mechanism/terrain/radar contacts ping-pong
between :complete and :queued on every backfill cron:
1. enqueue_for_contact/2 now filters out types that are already
:complete on the contact before building jobs. Previously
mark_status!/3 demoted every non-empty jobs_by_type entry back to
:queued, overwriting the :complete that workers had just set.
2. reconcile_stale_queued/1 now also flips mechanism_status and
radar_status back to :complete when the companion data
(propagation_mechanism / contact_common_volume_radar) already
exists. Drains the existing ~14k row backlog on the first post-
deploy tick.
3. IemClient.parse_iemre_json/1 accepts "" and nil so a 200-with-
empty-body IEMRE response doesn't FunctionClauseError the worker
through four retry attempts.
lgdc.uml.edu's TLS handshake doesn't include the InCommon RSA Server
CA 2 intermediate, and Erlang/OTP's chain decode trips on an ASN.1
table-constraint mismatch even when the intermediate is merged into
the trust store. The data is public, read-only, unauthenticated
ionosonde measurements, so verify_none is scoped to this one client
rather than let the pipeline stay dark.
Hot pods were restart-looping every ~25 minutes on liveness probe
timeouts. Root cause: ScoreCacheReconciler mirrored every .prop file
from NFS into ETS, including 7 days of GEFS Day 2-7 forecasts. With
23 bands × 43 valid_times × ~2 MB per grid = 1.95 GB in the
propagation_score_cache table alone; GC sweeps starved the
scheduler enough that /live dropped its 3 s budget.
The /map UI only ever requests the [now-1h, now+18h] window. Share
that bound as Propagation.hot_cache_window/0 and apply it in the
reconciler's disk-scan path plus NotifyListener's post-warm prune.
Long-horizon GEFS files stay on disk and are still served via the
lazy read_from_disk_and_cache path when requested directly.
Adds ScoreCache.prune_outside_window/2 (inclusive bounds) and
updates the reconciler tests to use relative-to-now timestamps
since hardcoded fixture dates now drift out of window.
**Rust workers exit after 30 consecutive claim_next failures.**
Both `prop-grid-rs` and `hrrr-point-worker` ran a `while` loop that
slept 10s and retried forever on `claim_next` errors. If Postgres
went unreachable, pods would stay alive but useless — no circuit
breaker, no liveness probe, just an infinite stream of error logs.
Now we increment a consecutive-failure counter and exit with an
error once it hits 30 (~5 minutes of DB unavailability). k8s
restart policy then cycles the pod, which both triggers an
operator alert and gives us a fresh sqlx pool on the way back up.
A successful claim (Some or None) resets the counter so one
flaky query doesn't poison it.
**BeaconMonitorsTest: drop the Process.sleep(1100).** The test
relied on wall-clock gaps to disambiguate `inserted_at` values on
the `:utc_datetime` (second-precision) timestamps — one full
second of sleep per run, and on shared CI it could still flake
when both inserts landed in the same second. Replaced with an
explicit `Repo.update_all` that back-dates the first row by one
second. 1.1s faster, deterministic.
**Wire format — /map scores payload.** Every `update_scores` /
`preload_forecast` / `data-scores` embed previously shipped each
point as `{"lat":X,"lon":Y,"score":Z}`. At 95k cells in full CONUS
that's ~36 bytes/point × 95k = 3.4 MB of JSON, ~50% of it repeating
the three key names. Packing as `[lat, lon, score]` tuples drops
per-point overhead to ~19 bytes → ~1.8 MB on the wire, ~45% smaller.
New `Propagation.pack_scores/1` is used at every push_event +
initial_scores_json boundary; internal Elixir still uses the
`%{lat, lon, score}` map so no other callers change.
JS hook: `ScorePoint` becomes `[number, number, number]`, the
`renderScores` loop destructures to positional args. Regression
test in map_live_test updated to match the new shape.
**ProfilesFile compression.** Rust writer was at gzip level 6
(default). Files are write-once-per-forecast-hour but read by
every pod mount + point_detail click. Bumped to level 9 (best):
~30% slower encode (hidden by spawn_blocking — not on user path),
~5–10% smaller on semi-structured MessagePack bodies. No reader
changes needed — same gzip stream.
**TypeScript strictness.**
- `app.ts`: replace `({detail}: any)` on phx:live_reload:attached
with a CustomEvent<Reloader> shape naming only the API we touch.
- `weather_map_hook.ts`: drop the `(this as any)._map` pair by
declaring a `GridLayerWithMap = L.GridLayer & { _map: L.Map }`
intersection on the overlay's createTile `this` parameter.
- `global.d.ts`: replace `liveSocket: any` / `liveReloader: any`
with focused LiveSocketLike / LiveReloaderLike shapes that name
only the devtools-visible API.
No more `any` in the codebase (verified via rg).
The main /map timeline was rendering each forecast hour twice
('Now 22:00' + '0h 22:00', '+1h 23:00' + '+1h 23:00', …).
Root cause: during the rolling `.ntms` → `.prop` rename window,
both extensions live on NFS for the same valid_time. `list_score_files`
matches either, so `list_valid_times_raw` mapped 2 files → 2 DateTimes
with no dedupe. Propagation.available_valid_times inherited the dup,
push_timeline pushed it to the JS hook, renderTimeline drew one chip
per entry — hence the visible duplicate.
Fix: `Enum.uniq` before the sort. Regression test writes a `.prop`
file, copies it as the same valid_time's `.ntms` legacy twin, and
asserts list_valid_times returns a single DateTime.
Also land a small refactor that was queued behind this: move the three
pure mechanism display helpers (label/1, badge_class/1, explainer/1)
out of the 2500-line contact_live/show.ex into a new
MicrowavepropWeb.ContactLive.Mechanism module. No behavior change — just
shrinks the LiveView and gives the display mapping a testable home.
- Add one-line moduledocs to 18 LiveViews that were carrying
@moduledoc false. Each line names the route and summarizes what
the page does so a future reader knows where to look before
opening the file.
- Add MapLive handle_event tests for toggle_radar, select_time /
set_selected_time, point_detail, map_bounds, retry_initial_scores.
Assertions focus on socket-state transitions and "didn't crash"
rather than raw HTML — map_bounds in particular has no visible
render-side effect (it push_events the new scores to the JS hook).
- Rust pipeline: integration-shaped test that hand-builds a 2-cell
surface + pressure grid, runs merge → cell_to_conditions →
precompute_band_invariants → composite_score_with → write_atomic
→ decode, asserting header + body length round-trip. Closes the
'pipeline's happy path is only covered by unit tests' gap.
Two unrelated production warnings from the same log window:
WeatherFetchWorker raised Postgrex 21000 cardinality_violation on
ASOS upserts when IEM returned two METAR records for the same
timestamp (routine + special at the same minute). Dedupe by
(station_id, observed_at) in upsert_surface_observations, keeping
the last occurrence since IEM's ordering is chronological.
SwpcClient.fetch_xrays warned with the full truncated response
body (~650 KB of GOES X-ray JSON) dumped into the log aggregator
whenever SWPC's CDN cut the response mid-stream. Intercept
Jason.DecodeError in fetch_and_parse and format it short: keep
the byte position and an 80-char preview, drop the rest.
/live fires every 10s per pod; the request line and Sent 200 response
from Plug.Telemetry drowned out real requests. log_level/1 now returns
false for /live too, same treatment /health already got.
