Two user-visible crashes in MapLive seen on prod pods today:
1. `:preload_forecast` handle_info timed out a Task.async_stream subtask
after 10 s and the `Enum.map(fn {:ok, h} -> h end)` match cascaded
the `:exit, :timeout` into the LiveView process, terminating the
session. Adds `on_timeout: :kill_task` and flat-maps only :ok
elements — a slow NFS forecast hour now drops from the preload
payload instead of killing the LV.
2. `LiveStash.stash/1` crashed with `** (ArgumentError) errors were
found at the given arguments: 2nd argument: not a valid match
specification` inside `:ets.select_replace/2` on OTP 28. The dep
(live_stash 0.2.0) has a latent bug against the tightened OTP 28
match-spec validator. Both call sites (MapLive.select_band,
SubmitLive.switch_tab) now route through a new
`MicrowavepropWeb.LiveStashGuard.stash/1` wrapper that rescues
ArgumentError, logs a warning, and returns the socket unchanged.
Stash is a convenience for reconnect state — failing it does not
need to kill the user's session.
Drop the guard module once the upstream match-spec fix lands.
EmeLive.mount/3 dropped the {:ok, tref()} return from send_interval
on the floor. Elixir 1.19 + :unmatched_returns dialyzer flag catches
the discarded tuple; the if/else with explicit :ok arms keeps both
branches typed as :ok.
- Bump render_async/2 timeout to 2_000ms across contact_live_test.
Default 100ms races when the suite runs on a loaded box — ContactLive.
Show fans out 8 hydration tasks at mount and sporadically missed the
window even on a quiet DB.
- Telemetry metrics-shape assertion uses Enum.any?/1 instead of the
`metrics != []` comparison Elixir 1.19's type system flags as a
distinct-type comparison (the function's non-empty return type).
Passes `capture_log: true` to ExUnit.start/1 so log output emitted
during a passing test stays in the per-test capture buffer instead of
stdout. Drops ~360 lines of noise from the suite: Postgrex teardown
disconnects, expected worker failures, NexradClient 404 stubs,
NotifyListener warm-skip messages, SNMP poll failures, PromEx tag-
drop warnings, and OTel-handler boot errors. Failing tests still
surface their full logs on report, so debugging is unchanged.
Side-effect fixes:
- LoggerFormatTest opts out via @moduletag capture_log: false — it
patches the :default :logger handler and capture_log hot-swaps the
same handler, so the setup lookup would 404.
- TelemetryTest.start_link/1 unlinks the supervisor before the brutal
kill so the exit signal doesn't propagate to the test process.
- Drop unused `import Ecto.Query` in a top_hours mix-task test.
- Drop unused default-arg `attrs \\ %{}` on create_contact/1 in
contact_map_live_test (every caller passes attrs explicitly).
Remaining noise is ~3 lines from oban_pro vendored-dep @impl warnings
(require an upstream patch) plus rare intermittent async teardowns.
Removes the OTLP span exporter (tracing-opentelemetry, opentelemetry,
opentelemetry_sdk, opentelemetry-otlp) and its transitive tonic/prost/
h2 tree — 56 crates off the build graph, ~3-5 min off a cold build on
the forgejo runner.
The fmt layer (JSON to stdout via tracing-subscriber) is unchanged;
structured logs still reach the Loki collector through kubectl logs,
and /metrics on :9100 still serves Prometheus. Traces were a nice-to-
have that nothing on call was actually reading.
telemetry::init keeps its signature (service_name + TelemetryGuard)
so re-adding OTLP later is a single-module change. k8s manifests drop
the now-unused OTEL_EXPORTER_OTLP_ENDPOINT env var.
The main-1776894678-93b526c image's index manifest references a
platform-manifest digest (sha256:8850f92b...) that's no longer in the
Forgejo registry. Both `latest` and the versioned tag point at the same
broken digest, so every `docker pull` on talos5 has been 404ing for 14h.
Bumping the crate root forces build-grid-rs.yaml's path filter to fire
and push a fresh image.
Adds 30 more test cases (2594 → 2624 tests) targeting the last
low-coverage hot spots.
Notable additions:
- ContactLive.Show fully-hydrated render path (seeds HrrrProfile,
TerrainProfile, ContactCommonVolumeRadar, Sounding, SurfaceObservation,
IemreObservation) exercises refresh_computed, build_propagation_analysis,
build_data_sources, compute_elevation_profile, and the downstream
render branches they unlock.
- handle_info :terrain_ready and :sounding_fetched paths for same-contact
and unrelated-contact dispatch.
- Flagged-invalid badge + line-through header style.
- Owner-label button path (Edit vs Suggest Edit).
- NotifyListener init/1 subscribes on schedule; handle_info(:subscribe)
succeeds against the live Repo.
- RecalibrateScorer runs cleanly against an empty corpus via the
Recalibrator insufficient-data fallback.
- HrrrNativeClient.extract_native_profiles/2 tolerates malformed input
without crashing.
Adds ~600 new test cases (2013 → 2613 tests; 170 properties; 0
failures) across 12 new test files plus expansions of eight existing
files. Big lifts per module:
ContactLive.Mechanism 33 → 100%
MetricsPlug 54 → ~100%
Microwaveprop.Release 15.8 → 70%+
Telemetry 38 → 76%
HrrrNativeGridWorker 27.7 → 76%
ContactLive.Show 34 → 48% (handler + render branches)
Admin.ContactEditLive 49.7 → 70%+
GefsFetchWorker 35.8 → ~55%
IonosphereFetchWorker 56.3 → 75%
PathLive 58.9 → 67%
WeatherMapLive 66.9 → 80.2%
RoverLive 0 → 70.3%
ContactMapLive 59.2 → 89.8%
ContactMapController 0 → 100%
Mix tasks (Rust.Golden, Notebook, Backtest, HrrrBackfill,
HrrrClimatology, HrrrNativeBackfill, NexradBackfill,
RadarBackfill, ImportContestLogs, PropagationGrid,
ResetEnrichment, Hrrr.PurgeGridPoints) 0 → ~60-100%
Two incidental fixes made while adding tests:
- ContactLive.Show.handle_event("toggle_flag", ...) was passing
socket.assigns.current_scope to admin?/1 instead of socket.assigns,
so admins never matched the pattern and every toggle ran the
"Admins only." flash branch. Flag now toggles for admins again.
- Commercial.PollWorker.fetch_weather/1 promoted from private to
@doc'd public so the IEM-fetch + ASOS-upsert path can be tested
directly without driving perform/1 through live SNMP (which was
timing out for ~50 s per test in the earlier attempt).
Stable property-test additions cover the dewpoint-from-RH monotonicity,
nearest_run/1 idempotence, and the ionosphere station envelope.
Adds HorizontalPodAutoscaler resources for the three load-responsive
Deployments:
- prop (web): min 2, max 5, averageValue 800m CPU
- prop-grid-rs: min 1, max 4, averageValue 400m
- hrrr-point-rs: min 1, max 4, averageValue 400m
- prop-backfill is intentionally unmanaged (queue-driven, not CPU-bound)
The scale-up policy reacts in 30 s with up-to-2-pods or +100 % per
window; scale-down stabilises over 10 min so the hourly propagation
chain doesn't churn replicas right after the :05 cron.
Removes `spec.replicas` from the HPA-managed Deployments so Flux's
periodic reconcile stops fighting HPA's live scaling decisions.
Drops `podAntiAffinity` from prop, prop-grid-rs, and hrrr-point-rs
so HPA can actually reach its maxReplicas when one physical host has
capacity for two pods; the scheduler picks whatever fits.
Updates runbook + project memory to drop the `talos5` hostname —
Postgres is now addressed strictly by IP (10.0.15.30).
Replaces the SVG schematic on /eme with a textured WebGL scene: NASA
Blue Marble Earth, real-size Moon, outbound beam, dashed return ray
to the sub-lunar point, and a red shading on the anti-moon hemisphere
that marks who can't see the Moon (outside the bounce footprint).
Three.js (~680 kB) now lives in its own chunk; esbuild --splitting
keeps it out of the main bundle so it only loads when a user hits
/eme. Main app.js drops from 1.6 MB to 922 kB.
Also swaps the Elevation/SNR rows so the badge comes before the number.
New /eme page — antenna aiming + link budget for moonbounce.
Given a station (callsign / grid / lat,lon), TX power, antenna gain,
detection bandwidth, and system Tsys, the page shows in real time:
- Moon azimuth / elevation from the observer (ticks every 10 s)
- Slant range to the Moon
- EIRP breakdown: TX(dBm) + gain(dBi) = EIRP(dBm)
- Path-loss decomposition (Earth → Moon → Earth):
free-space spreading (×2)
− moon reflection gain 10·log(4π σ / λ²) where σ = ρ·π·R²
= round-trip path loss
- Band-dependent lunar albedo ρ (VHF ≈ 0.065, microwave ≈ 0.08,
mm-wave ≈ 0.02) surfaced as a named line item with the current
ρ shown to three decimals
- Received power, thermal noise floor, and SNR with a badge that
classifies the margin (strong / marginal / below noise)
- Self-echo Doppler shift with its radial-velocity driver
New modules:
- `Microwaveprop.Moon` — low-precision lunar ephemeris
(Astronomical Almanac Sec. D.4 truncated series). Provides
`julian_day/1`, `geocentric/1`, `observer_position/3`, and
`radial_velocity_km_s/3`. Accurate to ~0.3° position / ~200 km
distance — well inside any amateur EME-dish beamwidth.
- `Microwaveprop.Propagation.Eme` — path-loss helpers:
`moon_albedo/1` (band lookup), `moon_rcs_m2/1`,
`fspl_round_trip_db/2`, `moon_reflection_gain_db/2`,
`path_loss_db/3`, `received_power_dbm/1`, `noise_floor_dbm/2`,
`snr_db/1`, `doppler_shift_hz/2`.
Wired into the main nav ("EME" button) and the router between
/path and /rover.
Tests: 3 properties + 41 unit tests cover JD conversion, moon
position envelope, band-dependent albedo lookup, decomposition
identity, linearity/monotonicity of path loss and Doppler, and a
LiveView integration test against the rendered DOM.
Full suite: 2,460 tests + 170 properties; credo strict clean.
- identify_var: full round-trip across every known (discipline,
category, number) triple, plus a property that any triple outside
the 12-entry lookup falls back to the "cat:num" default.
- identify_level: hybrid-level and entire-atmosphere branches, plus
two properties — pressure levels always render as
`div(value_pa, 100) <> " mb"` and unknown surface types always
render as `unknown:<type>:<value>` (generator samples only from
types outside the 6-entry lookup set).
- sign_magnitude_16: round-trip property over [-32_767, 32_767] (±0
both decode to 0 as intended) and a monotonicity property inside
a single sign band.
Suite: 2,419 tests + 164 properties (was 2,416 + 159); credo strict
clean.
Root cause of 14 stuck hrrr-queued contacts: hrrr_point_rs's CONUS-only
grid silently writes zero profiles for out-of-bounds points (UK,
Winnipeg, Alberta, >50° lat), but the fetch task still completes
successfully. ContactWeatherEnqueueWorker's hrrr_placeholder_jobs
always emitted a queued placeholder for any contact with a pos1,
which meant mark_hrrr_status! re-flagged :queued every backfill tick
forever.
Introduce Propagation.Grid.contains?/1 as the canonical in-CONUS
check (inclusive on all four edges, nil-safe). Use it in two places
of ContactWeatherEnqueueWorker:
- hrrr_placeholder_jobs/1 now returns [] for OCONUS contacts, same
as pre-2014 / already-complete ones.
- mark_hrrr_status!/3 empty-list clause now branches on Grid.contains?
first (→ :unavailable) before falling through to the existing
NARR-coverage / :complete logic.
Next backfill tick will flip the 14 stuck OCONUS contacts in prod to
:unavailable automatically. The remaining 13 "CONUS" stuck ones are
right at the 50° edge (lat=50.1875 rounds outside the box) — same
reconciliation path.
Coverage additions:
- Grid.contains?/1 — corner-inclusive, OCONUS positive/negative
fixtures based on the actual stuck-contact pos1 values, nil/missing-
key/nil-value handling.
- enqueue_for_contact regression tests: pos1 in the UK and pos1 just
north of the lat cap both land at :unavailable, not :queued.
Suite: 2,416 tests + 159 properties (was 2,409 + 159); credo strict
clean.
- Format.number/1: boundary cases (0, 999, 1_000), float rounding,
nil/atom fallback to to_string, plus two properties — comma
insertion is lossless round-trip and the comma count follows
`div(digits - 1, 3)` for |n| ≥ 1000.
- Format.bytes/1: per-unit boundary cases plus a monotonicity
property (parse-back preserves ordering across the B/KB/MB/GB
transitions).
- Instrument.span/2,3 (0→100%): wraps the block in a telemetry span
returning its value, defaults metadata to %{}, emits the
:exception event and re-raises on a raise inside the span, and
prefixes every event suffix with :microwaveprop.
- SolarClient: short-line / empty-line / blank-day handling for
parse_gfz_row; blank-line + short-line tolerance in parse_gfz_file;
filter_since boundary semantics (>= is inclusive, empty list
passes through).
Suite: 2,397 tests + 159 properties (was 2,382 + 155); coverage
71.12% → 71.16%; Format + Instrument now at 100%; credo strict clean.
- PageController: cover the legacy `/qsos` → `/contacts` redirect pair
(both the index and the id-preserving per-resource redirect), which
had no coverage since the route rename.
- Scorer property tests:
- f_to_c / c_to_f are inverses up to float epsilon, both propagate
nil, and the C→F slope is exactly 5/9.
- wind_speed_kts is non-negative, propagates nil on either axis,
and matches sqrt(u²+v²)·1.94384 (the 5-digit kts factor the
scorer uses).
- precip_to_rate_mmhr is non-negative, 0 for nil/zero/negative.
- dbz_to_rain_rate_mmhr is 0 below 5 dBZ (and for nil), and
monotonically non-decreasing above — guards against accidental
inversion of the Marshall-Palmer lookup.
Suite: 2,382 tests + 155 properties (was 2,380 + 148); coverage
71.08% → 71.12%; credo strict clean.
scripts/sync_prod_to_local.sh pulls only rows whose updated_at is
newer than the corresponding local row, versus restore_prod_to_local
which does a full pg_dump + drop + restore. For day-to-day refreshes
the incremental path is dramatically cheaper — the full dump is ~12
GB, while an hourly incremental is typically a few thousand rows.
Approach:
- Discover every public table that carries `updated_at` (skips the
oban_* runtime tables and schema_migrations by default).
- For each, intersect the column list across local and prod so schema
drift (a new column on prod not yet migrated locally, or vice-versa)
doesn't break the sync — we transfer the columns that exist on
both sides and leave the rest alone.
- Resolve the ON CONFLICT target: primary key first, else the
shortest non-partial unique index (important for partitioned tables
like hrrr_profiles which have no top-level PK).
- Stream the changed rows via `\COPY ... TO STDOUT` into a per-table
temp file, then load into a LIKE-cloned temp table and UPSERT
across `SET session_replication_role = replica` so FK ordering
doesn't block the dev-only run.
- Partitioned parents (relkind='p', relispartition=false) are kept in
the discovery list; partition children (relkind='r',
relispartition=true) are excluded so INSERTs route through the
parent.
Flags: `--dry-run` counts without writing; a positional argument
limits the run to a single table. `SKIP_TABLES` env var overrides
the exclusion list.
Also added a pointer from restore_prod_to_local.sh's header comment
to the new script.
- SwpcClient: unparseable time_tag rows drop silently, integer / string
numerics both cast, already-decoded list bodies (Req auto-decode) are
accepted, non-JSON-array bodies return {:error, :not_a_list}, X-ray
rows whose energy isn't the long wavelength are filtered, and the
misspelled-upstream electron_contaminaton key flags
electron_contaminated.
- UwyoSoundingClient: every month abbreviation routes to its correct
number, bogus month tokens fall through to an empty result, and
short-of-fixed-width lines are skipped without raising.
- Grid.wgrib2_grid_spec/0: lon/lat_start anchor to the SW corner,
steps match Grid.step/0, cell counts span CONUS inclusive, and the
spec's cell product equals length(Grid.conus_points/0).
- PollWorker: empty-link list is a no-op, and poll_fn receives host /
community / radio_type straight from the link row.
- Contact submission_changeset: antenna heights are bounded to
[0, 1000] ft, whitespace-only mode normalises without firing an
invalid-mode error, plus a pair of StreamData properties — every
sanctioned band validates and every out-of-list band string is
rejected.
Suite: 2,380 tests + 148 properties (was 2,370 + 146); coverage
70.38% → 71.08% total; credo strict clean.
Two breaking API changes in 0.2:
- `stash_assigns/2` is gone. Keys to persist are now declared on the
`use LiveStash, stored_keys: [...]` macro and `stash/1` picks them
up automatically.
- The TTL unit switched to seconds — this codebase never called the
TTL setter, so no config change is needed.
Updated both callers (MapLive persists selected_band + selected_time;
SubmitLive persists active_tab). `recover_state/1` signature is
unchanged. Full LiveView suite passes (67 tests).
Changes under lib/ (source):
- iemre_fetch_worker: replace `if transient_failure?/1` boolean predicate
+ `if/else` with a classifier that pattern-matches the error reason
directly, returning `:transient | :permanent`, and dispatch via
`case`. Three-clause head is clearer than the boolean predicate.
- scores_file.extract_points and nexrad_client.extract_box: force `//1`
step on the row/col comprehensions so an image box whose centre
lands outside the raster collapses to an empty iteration instead
of firing Elixir 1.19's ambiguous-range warning. Adds a regression
test for the right-edge box.
Test coverage added:
- Feature wrappers in Backtest.Features (theta_e_jump, shear_at_top,
duct_thickness, best_duct_freq, duct_usable_for_band,
distance_to_front / parallel_to_front stubs, all_features/0).
- NotifyListener handle_info tolerance for malformed payloads +
unknown messages, plus the PubSub broadcast side effect.
- Viewshed.effective_reach_km across every verdict / diffraction
tier / ducting-score tier combination.
- SnmpClient parse_af60_output unit conversions + unknown-column
handling; parse_snmpget_output OID normalisation and junk-line
tolerance.
- HrrrNativeClient native_level_count, native_variables,
native_messages shape, duct_messages / duct_byte_ranges.
- Changeset coverage for GeomagneticObservation, SolarFluxObservation,
SolarXrayObservation, Ionosphere.Observation, HrrrClimatology, and
Metar5minObservation — lifted each from ~50% / 0% to 100%.
- Property tests: GefsClient.dewpoint_from_rh invariants (Td ≤ T,
monotonic in RH, finite across the operating envelope) + idempotent
nearest_run. NexradClient.pixel_to_dbz monotonicity + bounded
output; latlon_to_pixel round-trip for every grid cell.
Also cleared several unrelated compiler/linter warnings:
- Three private test helpers (create_contact, insert_contact,
current_hour) had `\\ %{}` / `\\ 0` defaults that no caller used.
- profiles_file_test bound `dir` in a pattern match without using it.
Suite: 2,359 tests + 146 properties pass (was 2,300 / 139); coverage
70.38% → 70.89%; credo strict clean.
Pure-function invariants that example tests can't express at scale:
- Weather.round_to_iemre_grid/2, HrrrClient.nearest_hrrr_hour/1,
NexradClient.round_to_5min/1, NarrClient.snap_to_analysis_hour/1:
each is idempotent, its output satisfies a grid/slot-membership
predicate, and lies within a bounded distance of the input.
- NarrClient.in_coverage?/1: coverage_end is an exclusive upper
bound; any timestamp after it is out of coverage.
- IemreFetchWorker.backoff/1: result is in [120, 21_600], monotonic
non-decreasing in attempt, equals 120*2^(attempt-1) pre-cap.
- Contact submission_changeset notes field: every non-blank string
up to 2000 chars validates; any longer string is rejected.
22 new properties (139 → 161 including existing); all 2,300 tests
green, credo clean.
Rust workers (prop-grid-rs) that die mid-claim (SIGKILL, OOM, node
drain) leave grid_tasks rows stuck in status='running' forever,
because claim_next uses FOR UPDATE SKIP LOCKED and nothing resets the
orphan. The /status page then shows a permanent spinner with stale
f00/f10/f18 badges — currently 21 rows claimed as far back as
2026-04-20.
GridTaskEnqueuer.reclaim_stale_running/1 flips rows whose claimed_at
is older than 15 minutes back to 'queued'. Rows that have already
burned through 5 claim/reclaim cycles become 'failed' with a
reclaim-orphan error so the next hourly seed can replace them.
Wired into PropagationGridWorker.seed_chain/0 so it runs every :05
cron tick before new rows are seeded.
Also rename the status panel "Retrying" column to "Failed" — it was
always showing terminal `failed` rows, never retrying ones.
IemreFetchWorker now writes an empty stub observation when IEM returns
a permanent-failure status (404, 422), mirroring the existing {:ok, []}
path. The backfill cron's next tick sees the stub via
has_iemre_observation?/3, generates no job, and mark_status!/3 flips
the contact's iemre_status from :queued to :complete — draining the
51 contacts that have been stuck behind cancelled out-of-grid IEMRE
jobs.
Raise hot pod CPU limit 2 → 3 so BEAM gets 3 schedulers online.
Observed run queues of [2, 10, 0, 0, 0, 0] on the 2-scheduler
configuration during the :05 propagation chain, starving /live and
/health probe handlers and tripping intermittent liveness timeouts.
ADIF records frequently include operator commentary in the
<NOTES> field (multi-line, detailed) or <COMMENT> (short
one-liner). Now that contacts have a notes column, pass that
value through the import pipeline instead of dropping it.
Preference is NOTES over COMMENT to match the ADIF 3.1.4 spec's
intended split — NOTES is the richer field. Whitespace-only
values collapse to nil so a program that always emits a blank
NOTES tag doesn't fill every row with an empty string.
Tests cover NOTES capture, COMMENT fallback, NOTES-over-COMMENT
preference, and the absent-field nil case. ADIF upload blurb
lists the new carry-over so users see it before confirming.
Operators want a place to jot observations that aren't captured by
the structured fields — weather anecdotes, propagation mode
commentary, equipment details, band conditions. Add a text column
on contacts plus the two ingest paths users submit through:
* /submit single-contact form gets a 3-row textarea under the
other fields. Optional, max 2000 chars, with a live length cap
via maxlength. Whitespace-only input collapses to NULL so the
column reflects "no notes" rather than an empty string.
* CSV importer recognises a `notes` (or `note`) column via the
existing header_aliases table and flows values straight through
submission_changeset. Sample template gets a matching example
row with embedded commas so the quoted-field round-trip is
exercised on download.
ADIF import and the refinement/refinement-notify paths are out of
scope — users specifically asked for CSV + single-contact. Tests
cover the changeset (accept/blank/over-length), CSV header parsing
(plain + RFC-4180 quoted), and LiveView form submit end-to-end.
