Skewed isotherms, dry adiabats, and mixing-ratio lines were drifting
outside the frame at the corners — most visibly the warm-side
isotherms exiting the right edge near the top of the diagram. The
existing `clip_polyline/1` only filtered with a ±200 px slack, so
segments crossing the boundary still drew past the frame.
Wrap every grid layer + the T/Td/parcel polylines in a
`<g clip-path="url(#skewt-plot-clip)">`. The clipPath defines a
rectangle from (left, top) to (right, bottom). Frame outline and
axis labels stay outside the clip group so they keep rendering at
the box edge unchanged. The hover-cursor crosshair also lives
outside the clip so its readout box can sit flush against the plot
corner without being trimmed.
mix test: 2,902 / 2,902 + 221 properties green. mix credo --strict
clean.
Two changes that bring /skewt visually and analytically into parity
with the NOAA SPC sounding viewer
(https://www.spc.noaa.gov/exper/soundings/):
NOAA-style line set
-------------------
Updated the SkewtSvg style block to match the public SPC chart:
* Temperature trace ........ pure red (#ff0000) at 2.5 px stroke
* Dewpoint trace ........... pure green (#00aa00) at 2.5 px stroke
* Isotherms ................ dashed teal (#14b8a6)
* Dry adiabats ............. solid orange (#f59e0b)
* Moist (saturated) adiabats dashed sky-blue (#38bdf8)
* Mixing-ratio lines ....... dotted darker green (#16a34a), with
labels at 700 mb showing the g/kg
value, drawn only below 600 mb (matches
the SPC chart's lower-band number row)
* Parcel ascent ............ new dashed grey (#475569) line, drawn
whenever SkewtParams.derive returns a
non-empty parcel_trace
The mixing-ratio set is the SPC standard (0.4 / 0.7 / 1 / 2 / 3 / 5 /
8 / 12 / 16 / 20 g/kg) instead of the prior arbitrary set.
SkewtParams: SPC parcel-theory + indices module
-----------------------------------------------
New `Microwaveprop.Weather.SkewtParams` derives the parameter set that
ships in the SPC `*.txt` companion file:
* LCL pressure / temperature / height (Bolton 1980 eq 22)
* LFC pressure / height
* EL pressure / height
* SBCAPE, SBCIN (J/kg, virtual-temperature integration with
surface parcel)
* 3 km CAPE
* Lifted Index (parcel ↔ environment T at 500 mb)
* Freezing level (T = 0 °C height) and Wet-Bulb Zero
* DCAPE (downdraft CAPE — picks the min-θₑ source level in
400–700 mb, descends moist-adiabatically to surface)
* Layer lapse rates: sfc–3 km, 700–500 mb, 850–500 mb
* `parcel_trace` — pressure/temperature trajectory used by
SkewtSvg to draw the dashed parcel-ascent overlay
Wind-derived indices (SRH, BWD, Bunkers motion, SCP, STP, SHIP) are
deliberately *not* produced and the LiveView surfaces a one-line note
explaining why: HRRR persists wind only at 10 m AGL, so per-pressure-
level shear can't be computed from this data source.
LiveView panel restructured into grouped sections (Surface, Convective
parcel, Levels, Lapse rates, Moisture/stability, Refractivity) so the
layout matches the SPC viewer's order. The Levels rows show pressure
*and* height ("872 mb · 1,140 m") for quick cross-reference between
the diagram axis and the readout.
Tests: 8/8 new SkewtParamsTest cases (LCL Bolton check, saturated-air
edge case, full field-set presence, summer profile produces SBCAPE > 0
and LI < 0, capped profile produces SBCAPE = 0 and LI > 0, freezing
level interpolated correctly, sfc–3 km lapse rate, empty profile no
crash). 25/25 across all skewt-related test files. mix test green
(2,902 / 2,902 + 221 properties), mix credo --strict clean, mix
assets.build clean.
Two follow-ups to the earlier /skewt work:
1. Interactive crosshair (matches the SPC-viewer feel from the
reference screenshot). Mousing over the diagram now drops a
horizontal pressure cursor with a top-left readout box showing the
pressure at the cursor's altitude (mb), the height in m and ft,
and the linearly interpolated temperature and dewpoint at that
level. Two coloured dots track the T and Td traces along the
cursor so the user can read the spread visually.
Mechanics: SkewtSvg.geometry/0 exposes the plot geometry (left,
right, top, bottom, p_bottom, p_top, t_min, t_max, skew) as JSON;
SkewtSvg emits an initially-hidden <g class="skewt-cursor"> layer.
A new Skewt hook in assets/js/app.ts inverts pressure_y to recover
pressure from the cursor's viewBox-Y, walks the (descending-pres-
sorted) profile to interp T/Td/height, and updates the elements.
The hook also handles mouseleave to hide the cursor and uses the
SVG's screen-CTM so the math stays correct under any container
aspect-ratio resize.
2. DB fallback for `available_valid_times`/`load_profile`. When the
on-disk `ProfilesFile` has nothing for the location-and-window
(true on dev, transient on prod between chain runs), SkewtLive
now falls back to `HrrrProfileLookup.{list_valid_times_near,
read_point_near}/3` against the `hrrr_profiles` Postgres table.
The lookup uses the same ±0.07° spatial tolerance and the
`(lat, lon, valid_time)` unique index that `Weather.find_nearest_
hrrr/3` already relies on, so the queries are index-only.
Returned shape matches `ProfilesFile.read_point/3` so SkewtLive
reads from either source transparently. NaiveDateTime → UTC
normalisation is centralised in `ensure_utc/1` so the LiveView
sees `%DateTime{time_zone: "Etc/UTC"}` regardless of which side
produced the value.
Tests: 6/6 new HrrrProfileLookupTest cases (list/limit/tolerance,
read nearest cell, nil for far cells, picks closest among matches),
3/3 SkewtLiveTest still green, 2,894 total Elixir tests + 221
properties green via mix test (one pre-existing flaky test in
admin/contact_edit_live re-runs clean). mix credo --strict clean.
mix assets.build clean.
The /skewt page was empty in prod because Phase 2 of the Rust cutover
moved f01..f18 grid scoring to Rust but left profile-file writing only
on the analysis (f00) path. Result: /data/scores/profiles/ accumulated
one file per chain run instead of nineteen, so SkewtLive only ever saw
the analysis hour — and that hour falls outside the 1 h past cutoff
within an hour of the chain finishing, leaving the page empty most of
the time.
Two surgical fixes:
* `pipeline::run_chain_step` now builds `profile_entries` alongside
`prepared` from the merged grid in a single iter pass, then spawns
`profiles_file::write_atomic` on the blocking pool to overlap the
NFS write with the score-band scoring/write fan-out — same pattern
`run_analysis_step` already uses. `ChainStepStats` gains
`profile_cells_written` so the worker log line is symmetric with
the analysis step's existing field and a future failure mode that
drops the profile would show up as a divergence.
* `SkewtLive.available_valid_times/0` widens the past cutoff from
1 h to 3 h. The chain's analysis hour can be ~70 min old by the
time the next chain finishes (and ~2 h on a missed cycle), so the
1 h cutoff was leaving the page blank during normal transients.
18 h forward window unchanged.
Cosmetic: cargo fmt picked up trivial rewrites in decoder/fetcher/
native_duct/hrrr_point_worker that had drifted; rolled in.
Verification: cargo build --release, cargo clippy --lib --bins -D
warnings, cargo test --lib (134/134), mix test
test/microwaveprop_web/live/skewt_live_test.exs (3/3) all green.
Both retired in light of the 2026-04-25 revision report
(`docs/algo-reports/2026-04-25-algo-revisions.md`):
* HPBL boundary-layer multiplier in `Scorer.score_refractivity/{3,4,5}`
is gone. On the n=47,418 10 GHz HRRR-matched corpus rho_hpbl is
+0.004; binned distance is flat to within ±5 % across 200–2,000 m.
The previously reported 2.3× ratio between shallow and deep HPBL
bins was a small-corpus artefact that disappeared once the matched
corpus passed ~5,000 contacts.
* Native-profile 1.15× duct boost is gone. At 10 GHz on n=52,341
matched contacts, cells where `best_duct_band_ghz` ≥ band ran
*shorter* than no-duct cells (198 km vs 211 km, n=173 vs 44,658).
The Bulk Richardson gate that existed only to suppress that boost
in turbulent conditions is also retired; arity preserved on every
`score_refractivity` overload so callers compile unchanged.
Same retirements applied in the Rust port (`rust/prop_grid_rs/src/scorer.rs`)
to keep the Elixir/Rust parity tests green.
algo.md updated end-to-end:
* Finding 4 (HPBL) rewritten to "no usable signal" with the
n=47,418 bin table.
* Finding 5 (gradient) tightened: load-bearing only at 24 GHz,
set band-specific gradient weight to 0 outside [10, 47] GHz on
the next derive_band_weights run.
* Finding 8 (time-of-day) augmented with the robust hour-bucketed
amplitude index (40 / 82 / 101 % at 10 / 24 / 47 GHz) and a
selection-bias caveat for VHF/UHF (where 129–148 % amplitude
reflects evening contest scheduling, not propagation).
* Finding 9 (sounding) refreshed against the n=27,058 corpus.
* Part 2c "Native-profile duct boost" rewritten as
"retired 2026-04-25" with the falsifying join.
* Part 2c "Commercial-link inverse sensor" promoted from deferred
with the 22-day, 38k-sample SNMP corpus, the +0.61 / −0.61 PWAT
and pressure correlations against the 37-hour HRRR overlap, and
the 04–05 / 10–12 local two-peak morning fade window.
* Part 2b "Pressure" annotated with the U-shape at 10 GHz and the
next-iteration target.
* Part 2d "What stayed, what left" + "Open items" updated to
reflect the four code/doc moves.
119 Elixir scorer tests green, 134 Rust scorer tests green, 2,888
total Elixir tests + 221 properties green via `mix precommit`.
Single search bar accepts an address, Maidenhead grid square, or
callsign; resolves it to lat/lon via Geocoder / Maidenhead /
CallsignLocation (cheapest classification first), snaps to the HRRR
grid via the existing ProfilesFile reader, and renders an SVG
Skew-T-Log-P with isobars, skewed isotherms, dry/moist adiabats, and
mixing-ratio lines. A button row picks between every available
forecast hour (now → f18); default is the most recent analysis.
Right pane lists derived stability and refractivity stats from
SoundingParams.derive (PWAT, BL depth, K-index, lifted index, min
dN/dh, ducting status + per-duct base/top/ΔM).
Renderer is server-side SVG so the page works without JS and serializes
into the LiveView payload as a single tag. Wind barbs are deliberately
omitted — HRRR's persisted profile carries wind only at 10 m AGL.
Each fix is covered by a regression test that fails on `main` and
passes on this commit.
Round 1 (initial review):
* propagation: thread `latitude` into the conditions map so
`score_season/4` actually picks up regional multipliers
* hrrr_client / fetcher.rs: `nearest_hrrr_hour` rounds DOWN, never at
a future cycle that NOAA hasn't published yet
* radio: spherical-vector great-circle midpoint replaces the
arithmetic mean — anti-meridian paths no longer fold to Greenwich
* weather: `reconcile_weather_statuses` scales the longitude band by
`1 / cos(lat)` so the bbox stays ~150 km wide at every latitude
* radio/maidenhead: clamp 90°/180° below the field-bucket overflow so
`from_latlon` never emits invalid characters like 'S'
* prop_grid_rs/pipeline: merge HRRR + NEXRAD-derived rain rates and
read `best_duct_freq_ghz` into `best_duct_band_ghz` so the Native
Duct Boost actually fires
* propagation/region (Elixir + Rust): inclusive upper bounds so points
exactly at lat_max get the regional multiplier
* weather/sounding_params (Elixir + Rust): drop the 10 m gradient
floor so HRRR's thin near-surface layers stop hiding sharp ducts
* weather/sounding_params: when the profile ends inside a duct,
finalize it with the highest sample as the top instead of throwing
it away (Rust port already correct)
Round 2 (post-fix sweep):
* radio + commercial: single canonical haversine in Radio (atan2
form); Commercial delegates instead of carrying a second copy that
could disagree at threshold distances
* prop_grid_rs/profiles_file: `snap_coords` matches Elixir's
step-aware snap (`round(coord/0.125) * 0.125`, then 3-dp round) so
Rust-keyed and Elixir-keyed profile maps land on the same cell
* weather/grib2/wgrib2: `parse_lon_val_segment` uses `Float.parse`
uniformly — wgrib2 dropping the trailing `.0` from a longitude no
longer crashes the whole chain step
`handle_event("switch_tab", ...)` ran `String.to_existing_atom/1` on
the raw `tab` value from the browser, so a stale, forged, or
mistyped client event with any unknown string crashed the LiveView
with `ArgumentError: not an already existing atom`.
Compares against an explicit `@valid_tabs` allowlist (`:single`,
`:csv`, `:adif`) — known values switch the active tab, anything
else is ignored.
`apply_edit_to_contact/2` is the path for both owner direct-edits and
admin reviewed-edits. The public contact-map + total-count + gzipped
controller payload caches are built from `private == false` rows, but
only the insert path was clearing them — every other update left
/api/contacts/map and /contacts/map serving the previous snapshot for
up to 10 minutes. A `private: false -> true` flip on a contact already
present in the cached payload was a temporary privacy leak; grid
corrections, callsign changes, and flagged_invalid flips silently
diverged from the live row.
Extracts the three `Cache.invalidate` calls into
`invalidate_contact_map_caches/0` and runs it from both
`apply_edit_to_contact/2` and the existing insert path so the public
view stays in sync.
The /contacts/:id edit form always submits the `private` checkbox, but
`current_value/2` had no `"private"` clause and fell through to the
catch-all `nil`. The Contact schema defaults `private: false`, so an
unchanged box compared `false != nil` and was always tagged as a
change — for non-owners that meant a spurious pending review queued
on every save, and for owners/admins it triggered a no-op direct
update.
Adds the missing clause so the diff sees boolean-vs-boolean and
correctly returns `:no_changes` when the box is untouched.
The /contacts/:id edit form pre-fills qso_timestamp via
`Calendar.strftime(ts, "%Y-%m-%d %H:%M")` and submits whatever the
user typed back unchanged. Two failure modes resulted:
1. `diff_against_contact/2` compared that string against the stored
`%DateTime{}`, never matched, and treated every untouched submit
as a modification.
2. `build_contact_changes/2` then ran `DateTime.from_iso8601/1` on
the space-separated, no-seconds, no-Z form and crashed with
`MatchError` on `{:error, :invalid_format}` — direct owner/admin
submits raised instead of saving.
Adds `normalize_timestamp_field/2` to `normalize_proposed/1` that
parses both the form's `"YYYY-MM-DD HH:MM[:SS]"` and the strict
`"YYYY-MM-DDTHH:MM:SSZ"` shapes into a UTC `%DateTime{}` (with
`NaiveDateTime.from_iso8601` doing the heavy lifting after a small
seconds-padding pass). Empty strings drop the field; unparseable
input drops it too so a typo becomes "no changes" rather than a
500.
Bandit's dual-stack listener delivers IPv4 connections to the app as
IPv4-mapped IPv6 tuples (`::ffff:a.b.c.d`). The trust check compared
that 128-bit form against our IPv4 trusted_proxies CIDRs and never
matched, so cf-connecting-ip was ignored on every cloudflared-relayed
request — every visitor logged as the cloudflared sidecar pod IP
(rendered as `0:0:0:0:0:65535:2804:101` etc.).
Collapses the mapped tuple to plain IPv4 before the trust check and
records it that way on the conn, so logs and session storage see the
real client IP that Cloudflare set in cf-connecting-ip.
The top navbar in the root layout already linked to /eme, but the
in-page side sidebars on /map and /weather (mobile + desktop) skipped
straight from Path Calculator to Beacons. Adds the EME entry between
them in all four panels so the Earth-Moon-Earth calculator is reachable
without bouncing through the navbar.
Also demotes the LiveStashGuard "stash skipped" warning to debug. It's
the known OTP 28 / LiveStash 0.2.0 ArgumentError that fires on every
event triggering a stash; the guard already handles it, the noise was
adding nothing.
Adds a `layer` query parameter to `/weather`. Loading
`/weather?layer=temp_700mb` boots straight into that overlay; clicking
a different layer pill push_patches the URL so the active selection is
shareable and survives a refresh. Unknown layer ids fall back to the
default (temperature) instead of crashing.
/weather mount called Weather.latest_weather_grid/1 directly, which
runs WeatherLayers.derive over 92k cells and Jason-encodes the
result. Production HTTP logs showed dead renders consistently
landing at 9.8–10.2 s — right at LiveView's socket join deadline.
The browser would render half the page, the WS would fail to mount,
and the user saw "attempting to reconnect".
The JS hook already fires map_bounds via requestAnimationFrame
immediately after mount, and the existing handle_event("map_bounds")
path returns viewport-scoped weather data via the update_weather
push event. So the dead-render data fetch is redundant — drop it.
Mount now returns initial_data_json: "[]"; the layer paints
~50–200 ms after WS connect via the bounds-driven flow.
PropagationGridWorker hard-coded `now-2h` for run_time selection on
the conservative theory that NOAA always finishes publishing within
two hours. In practice f18 is on the mirror by HH:55–HH:00 of the
next hour, so the cron at HH:05 was always picking a cycle that was
already an hour stale.
`HrrrClient.cycle_available?/1` HEADs the wrfprsf18 idx (the slowest
file in the cycle) — a 200 means every earlier forecast hour is also
on disk. `pick_run_time/1` probes `now-1h` first; if the probe
returns true we use that cycle, otherwise we fall back to the
existing `now-2h` slot. Test hook via :hrrr_cycle_available_fn env
keeps the behaviour fully exercisable without hitting NOAA.
End-to-end: weather/score data lag drops from ~2 h to ~1 h on every
hour where NOAA delivers on time. The fallback path keeps the chain
running on slow-publish hours.
850 mb T alone shows that warm air is present aloft but doesn't reveal
whether it's the textbook cap structure — a warm 700 mb above an EML
plume. The three new layers expose the canonical capping diagnostics
visually:
* T @ 700 mb — ≥10 °C is the southern Plains "moderate cap" threshold
* Td @ 700 mb — wide T-Td depression at 700 mb signals the EML plume
* Lapse 850→700 — steep mid-layer lapse rate (≥7 °C/km) over a moist
boundary layer is the cap mechanism itself
All three derive purely from the existing pressure-level profile (Rust
already fetches up through 700 mb), so no Rust pipeline changes needed.
ScoreCache held 437 entries × ~1.94 MiB each (~850 MiB per pod) of
{band_mhz, valid_time} grids — data already on NFS as compact .prop
files. NotifyListener and ScoreCacheReconciler both eagerly materialised
every band into ETS on each Rust completion / 60s sweep, so 5 hot
replicas wasted ~4.2 GiB of redundant cache and OOMed at 6 GiB.
Bound the cache to 32 entries with eviction by oldest valid_time, drop
the eager warm loops, and let LiveView callers lazy-fill via the
existing ScoresFile read path. ScoreCacheReconciler had no remaining
purpose and is deleted; runbook + prom_ex counters updated to match.
Steady-state cache footprint ~60 MiB per pod instead of ~850 MiB.
pg_stat_user_tables tracks two timestamps — `last_autoanalyze`
is only bumped by the autovacuum daemon, while `last_analyze` is
bumped by manual ANALYZE statements (including the ones our Task
issues). The previous query only filtered on `last_autoanalyze`,
so on a re-run within the skip_recent window every manually-
analyzed table came back for another ANALYZE pass.
Uses `GREATEST(last_analyze, last_autoanalyze)` so fresh manual
analyses count as recent.
Running ANALYZE serially against the 13M-row HRRR monthly partitions
took longer than the kubectl exec idle timeout (~300s per partition
on the Turing Pi 2), so the one-shot RPC kept getting killed before
the big partitions ran. Moves the loop into a supervised Task on
the caller's node so the RPC returns immediately; the BEAM keeps
running ANALYZE regardless of kubectl connection state.
Also sorts tables ascending by n_live_tup so small tables finish
first and their fresh stats are visible to the planner right away.
Progress is logged per table via `Logger.info`.
Two issues spotted in pg_stat_user_tables:
1. Most `hrrr_profiles` partitions have `last_autoanalyze IS NULL`.
The partitions are append-only from the hourly grid worker, so
autovacuum never crosses the analyze threshold — but without
stats the planner falls back to estimates and picks bad joins
on per-QSO lookups.
2. `contacts`, `grid_tasks`, `hrrr_fetch_tasks`, and
`contact_common_volume_radar` see high UPDATE volumes
(enrichment status flips, queue claim/complete cycles, mechanism
classifier writes) but only accrue ~5% dead before the default
20% / 10% autovacuum triggers fire. BackfillEnqueueWorker's
status-priority ORDER BY + the Rust workers' FOR UPDATE SKIP
LOCKED claim scan both want fresher stats than that.
Changes:
- Migration `20260424204656_tune_autovacuum_for_high_churn` sets
per-table `autovacuum_*_scale_factor` + `_threshold = 50` on the
four tables above. Aggressive enough to keep stats current;
tame enough that vacuum doesn't thrash the Turing Pi 2 Postgres
node. Reversible.
- `Weather.analyze_all/1` walks `pg_stat_user_tables`, runs
`ANALYZE` on every table not auto-analyzed in the last 6 h,
reports counts. Meant to be invoked once via
`bin/microwaveprop rpc 'Microwaveprop.Weather.analyze_all()'`
after deploy to seed statistics on the cold partitions;
idempotent on re-run.
The three tables on /status (Job Queue, Rust Workers, Table Sizes)
were collapsing to fit the viewport on narrow phones, wrapping
header text ("Siz…") and splitting values like "160 GB" into two
lines ("160." / "GB"). The overflow-x-auto wrappers were already
present but column auto-sizing still squeezed width below content.
Adds `whitespace-nowrap` to each `<table>` so cells never wrap, and
`-mx-2 px-2` on the overflow wrapper so horizontal scroll extends
edge-to-edge of the mobile viewport instead of being inset by the
layout's outer padding. Tables now scroll horizontally as intended
on mobile; desktop layout unchanged.
Root cause of the "81,980 / 81,994" display that wouldn't advance:
14 contacts had pos1 inside the HRRR CONUS grid but pos2 or the
great-circle midpoint landed outside it (50°N+ into Canada, mid-
Atlantic, Pacific, Caribbean, Alaska). The Rust hrrr-point-worker
silently returns no profile for OCONUS points, so
`hrrr_data_fully_present?/1` never evaluated true and the contact
sat `:queued` forever.
The existing ContactWeatherEnqueueWorker.mark_hrrr_status!/3 already
flips to `:unavailable` on empty job lists via NarrClient.in_coverage?
+ Grid.contains?(pos1), but those checks only considered pos1 — not
the full contact_path_points list. When pos1 was in-grid but a
downstream point was not, the contact entered `:queued` and the
reconciler couldn't rescue it.
`reconcile_hrrr_statuses/0` now sweeps `:queued` contacts into two
buckets:
- ANY path point OCONUS → `:unavailable` (matches what the enqueuer
would emit had it known about the downstream points);
- else, all points present → `:complete`.
Applied once in prod to 14 stuck contacts; cron path picks up any
new occurrences going forward.
The Rust hrrr_points worker now persists surface_refractivity /
min_refractivity_gradient / ducting_detected at upsert time (previous
commit), but the ~80k rows written before that fix have the scalar
columns NULL even though the pressure-level `profile` JSONB is
populated. Re-fetching them would re-pull tens of GB of HRRR grids
we already parsed.
Weather.backfill_hrrr_scalars/1 streams rows with
`surface_refractivity IS NULL`, runs the existing
SoundingParams.derive/1 against the stored profile, and writes the
scalars back — no network, no GRIB decode. Batched (500 rows per
tick, 1000 tick cap) to keep transactions short on the Turing Pi 2
PG node.
Meant to be run once via `bin/microwaveprop rpc` after deploy.
Idempotent; the WHERE clause excludes already-backfilled rows.
Found by auditing the ProfilesFile, hrrr_profiles, and weather-layer
contracts against what the Rust writers actually emit.
Bug 1 (HIGH) — Rust `hrrr_points` worker upserted hrrr_profiles rows
with NULL `surface_refractivity`, `min_refractivity_gradient`, and
`ducting_detected`. The Elixir HrrrClient path always wrote them
via SoundingParams.derive. Per-QSO pages that rely on these
scalars ("N:", "dN/dh:", duct badge) showed "—" for any contact
enriched through Stream C. Fixed by porting
`surface_refractivity` + `ducting_detected` into Rust's
`sounding_params.rs` and deriving at upsert time.
Bug 2 (HIGH) — Rust f01..f18 `cell_to_profile_entry` never emitted
wind_u / wind_v / cloud_cover_pct / precip_mm into ProfilesFile
cells. `Propagation.factors_for` recomputes factor scores from the
profile cell on /map click, and these four fields are read
(wind_speed_kts, sky_cover_pct, precip_to_rate_mmhr). Result: every
f01..f18 point-detail popup showed 0 for wind/sky/rain factors.
Fixed by extracting the grib values at cell_to_profile_entry time
and whitelisting the keys for atomization in
profiles_file.ex `@mp_atom_keys`.
Bug 4 (MEDIUM) — `WeatherLayers.duct_field/2` only knew the
`SoundingParams.detect_ducts` shape (`d["base"]` / `d["strength"]`).
The Rust/HrrrNativeClient shape uses `:base_m` / `:thickness_m`
atom keys (post-ProfilesFile.read atomization). On /weather, the
Duct Base / Duct Strength layers returned nil for every Rust-origin
cell. Tolerant lookup now reads either shape; uses `thickness_m`
as the "strength" proxy for the native shape (thicker duct traps
a wider band).
Skipped (verified but lower impact):
- Rust ProfilesFile aggregates ducts as scalars, not per-layer — the
`ducts` array in the popup is always empty. Requires larger
DuctMetrics refactor in Rust; deferred.
- `complete_hrrr_task` emits no NOTIFY (vs `complete` which does).
Elixir has no matching listener yet, so adding NOTIFY alone is a
no-op; the existing cron reconciler catches it.
- `:best_duct_band_ghz` fallback in propagation.ex is dead code;
cleanup, not a bug.
133 Rust tests + 2842 Elixir tests + credo green.
The Rust f01..f18 pipeline writes ProfilesFile cells, and the Rust
hrrr_points worker writes hrrr_profiles rows, with profile entries
keyed "pres_mb" / "hght_m" / "tmpc" / "dwpc" (string keys pre-
atomization; atom keys post-ProfilesFile.read). Elixir's
SoundingParams.derive and WeatherLayers.sort_profile filtered on
"pres" / "hght" — so every derived field returned nil for any
Rust-provided profile.
Visible symptoms on /weather: N-gradient, Refractivity, T @ 850mb,
Td @ 850mb, Lapse Rate, Inversion, Inv. Base all rendered no overlay.
For per-contact analysis, min_refractivity_gradient from the Rust
HRRR point worker's rows silently dropped.
- SoundingParams.normalize_profile_entry/1: single-source normalizer
accepting legacy, Rust-string, and Rust-atom shapes; returns the
canonical "pres"/"hght"/"tmpc"/"dwpc"/"drct"/"sknt" shape.
- SoundingParams.derive/1 and WeatherLayers.sort_profile/1 both run
every entry through it before the rest of the existing logic.
- Weather.build_grid_cache_row/4 now falls back to native_min_gradient
when SoundingParams can't derive one (surface-only profiles).
Also:
- Default /weather overlay changed to Temperature (LiveView assign +
JS fallback in the hook's dataset default) per user request.
Extends the weather-status reconciler with symmetric sweeps for HRRR
and IEMRE. Same failure mode: the worker writes data to the shared
table but has no back-pointer to the contact that triggered the
fetch, so contact-level status never flips unless a user views the
page.
- Weather.reconcile_iemre_statuses/0: iterates :queued contacts;
flips :complete when every path point has an iemre_observations
row at the rounded 0.125° grid cell for the qso_timestamp's date.
- Weather.reconcile_hrrr_statuses/0: same shape; flips when
hrrr_data_fully_present?/1 holds.
- Both run alongside the existing weather sweep at the tail of
ContactWeatherEnqueueWorker.perform/1.
Kept in Elixir (not SQL) because contact_path_points/1 emits 1-3
points per contact and grid-rounding each in pure SQL is awkward.
The stuck-at-steady-state count is always tiny (<20), so the
iteration is cheap.
2,019 contacts sat permanently in weather_status=:queued even after
their ASOS data had been ingested. The only code path that flipped
:queued → :complete was MicrowavepropWeb.ContactLive.Show on page
view — contacts that nobody visited stayed :queued forever.
Adds Weather.reconcile_weather_statuses/0, a single SQL UPDATE that
flips every :queued contact whose ±2h / 150km window now contains
at least one surface observation. Called at the tail of the
existing ContactWeatherEnqueueWorker cron so the correction runs
regardless of whether anyone opens the UI.
Visible effect: the status page's "Weather done" count (previously
capped at 79,975 while data kept landing) will converge on the real
total after the next cron tick.
2838 tests + credo green.
Replaces per-QSO-endpoint asos_batch granularity with per-station-day
granularity. Each contact in the same UTC day near the same station
now produces identical unique-args and collapses to ONE Oban job, so
the backlog shrinks as cross-contact duplicates dedup.
Each fetch also covers a full 24h window (vs the previous 4h) so one
IEM request returns ~24 hourly observations instead of ~1-2. The
effective bytes-per-request is 10x higher, amortizing the
IemRateLimiter gap + 429 retry tail across far more data.
Changes:
- Weather.station_day_covered?/2 + station_day_pairs_covered/1 — cheap
UTC-day coverage checks for the enqueuer + worker skip paths.
- WeatherFetchWorker gains an "asos_day" clause that fetches the full
UTC day for one station and upserts. Skips if the day is already
covered in the DB.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one asos_day
job per (station_id, UTC date) pair. ±2h windows crossing midnight
enumerate both dates.
- Microwaveprop.Weather.RebatchAsos.to_day_jobs/1 migrates any
already-queued asos/asos_batch jobs into the new shape. Idempotent;
supports dry_run.
- asos_batch worker clause retained for jobs already in flight at
deploy time.
2835 tests + credo green.
Mix.Task.run isn't available in a compiled release, so the in-prod
`bin/microwaveprop eval 'Mix.Tasks.Weather.RebatchAsos.run(...)'`
dispatch crashes. Moves the real logic into a plain module
(`Microwaveprop.Weather.RebatchAsos`) that both the Mix task and a
release eval can call. Uses IO.puts over Mix.shell for the same
reason.
- IemRateLimiter gains AIMD-style adaptive spacing. signal_429/0
widens the current gap (*= 1.5, capped at max_interval_ms default
10s); signal_success/0 narrows it back toward the configured base
(*= 0.92, floored at interval_ms). Self-tunes to IEM's moving
ceiling without needing the manual "safe for 4 pods" constant.
- IemClient now routes every response through a central handle_response
helper that fires the widen/narrow feedback signals, eliminating the
four near-identical case blocks.
- Mix.Tasks.Weather.RebatchAsos collapses any already-queued single-
station "asos" jobs into the batched "asos_batch" shape the
enqueuer now emits, so the pending backfill queue converts to the
new per-request-efficient path instead of draining at the old rate.
Idempotent; supports --dry-run.
2833 tests + credo green.
IEM's ASOS CSV endpoint accepts multiple station= params in a single
request. Prior code enqueued one WeatherFetchWorker job per nearby
station per QSO endpoint — N jobs × 1 station each — which paid the
IemRateLimiter's 1500ms gap and the IEM 429-retry tail N times per
contact.
Changes:
- IemClient.asos_url/3 accepts a list of station codes.
- IemClient.fetch_asos_batch/3 fetches N stations in one call and
returns rows grouped by station_code (with absent codes filled as
empty lists so callers can stub them).
- parse_asos_csv/1 now exposes the station_code column it was
previously discarding.
- WeatherFetchWorker gains an "asos_batch" fetch_type clause that
unpacks rows per (station_id, station_code), upserting or stubbing
each. The single-station "asos" clause stays for already-queued
retryable jobs.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one batch
job per (lat, lon, 4h window) covering every uncovered nearby
station (sorted for deterministic unique-args).
Expected effect on backfill: ~10-20x fewer IEM requests per contact
enrichment cycle, matching drop in 429 retry traffic.
Removes the six OTel packages (opentelemetry, _api, _exporter,
_phoenix, _oban, _bandit) and their 9 transitive deps. Tracing was
only wired for Phoenix/Bandit/Oban auto-instrumentation — PromEx
already covers the metrics we actually use on the status page +
Grafana. Also drops the OTEL_EXPORTER_OTLP_ENDPOINT env var and
its comment block from both deployment manifests.
No translations are actually maintained and no UI is shipped in a
non-English locale. Replaced the handful of gettext(...) calls in
core_components + layouts with their literal English strings, and
rewrote translate_error/1 as a small %{key} interpolator over the
Ecto changeset error tuple.
Removes gettext + expo (compile dep). sutra_ui stays — it's a
transitive dep of live_table's TableComponent.
Two user-visible crashes in MapLive seen on prod pods today:
1. `:preload_forecast` handle_info timed out a Task.async_stream subtask
after 10 s and the `Enum.map(fn {:ok, h} -> h end)` match cascaded
the `:exit, :timeout` into the LiveView process, terminating the
session. Adds `on_timeout: :kill_task` and flat-maps only :ok
elements — a slow NFS forecast hour now drops from the preload
payload instead of killing the LV.
2. `LiveStash.stash/1` crashed with `** (ArgumentError) errors were
found at the given arguments: 2nd argument: not a valid match
specification` inside `:ets.select_replace/2` on OTP 28. The dep
(live_stash 0.2.0) has a latent bug against the tightened OTP 28
match-spec validator. Both call sites (MapLive.select_band,
SubmitLive.switch_tab) now route through a new
`MicrowavepropWeb.LiveStashGuard.stash/1` wrapper that rescues
ArgumentError, logs a warning, and returns the socket unchanged.
Stash is a convenience for reconnect state — failing it does not
need to kill the user's session.
Drop the guard module once the upstream match-spec fix lands.
EmeLive.mount/3 dropped the {:ok, tref()} return from send_interval
on the floor. Elixir 1.19 + :unmatched_returns dialyzer flag catches
the discarded tuple; the if/else with explicit :ok arms keeps both
branches typed as :ok.
Adds ~600 new test cases (2013 → 2613 tests; 170 properties; 0
failures) across 12 new test files plus expansions of eight existing
files. Big lifts per module:
ContactLive.Mechanism 33 → 100%
MetricsPlug 54 → ~100%
Microwaveprop.Release 15.8 → 70%+
Telemetry 38 → 76%
HrrrNativeGridWorker 27.7 → 76%
ContactLive.Show 34 → 48% (handler + render branches)
Admin.ContactEditLive 49.7 → 70%+
GefsFetchWorker 35.8 → ~55%
IonosphereFetchWorker 56.3 → 75%
PathLive 58.9 → 67%
WeatherMapLive 66.9 → 80.2%
RoverLive 0 → 70.3%
ContactMapLive 59.2 → 89.8%
ContactMapController 0 → 100%
Mix tasks (Rust.Golden, Notebook, Backtest, HrrrBackfill,
HrrrClimatology, HrrrNativeBackfill, NexradBackfill,
RadarBackfill, ImportContestLogs, PropagationGrid,
ResetEnrichment, Hrrr.PurgeGridPoints) 0 → ~60-100%
Two incidental fixes made while adding tests:
- ContactLive.Show.handle_event("toggle_flag", ...) was passing
socket.assigns.current_scope to admin?/1 instead of socket.assigns,
so admins never matched the pattern and every toggle ran the
"Admins only." flash branch. Flag now toggles for admins again.
- Commercial.PollWorker.fetch_weather/1 promoted from private to
@doc'd public so the IEM-fetch + ASOS-upsert path can be tested
directly without driving perform/1 through live SNMP (which was
timing out for ~50 s per test in the earlier attempt).
Stable property-test additions cover the dewpoint-from-RH monotonicity,
nearest_run/1 idempotence, and the ionosphere station envelope.
Replaces the SVG schematic on /eme with a textured WebGL scene: NASA
Blue Marble Earth, real-size Moon, outbound beam, dashed return ray
to the sub-lunar point, and a red shading on the anti-moon hemisphere
that marks who can't see the Moon (outside the bounce footprint).
Three.js (~680 kB) now lives in its own chunk; esbuild --splitting
keeps it out of the main bundle so it only loads when a user hits
/eme. Main app.js drops from 1.6 MB to 922 kB.
Also swaps the Elevation/SNR rows so the badge comes before the number.
New /eme page — antenna aiming + link budget for moonbounce.
Given a station (callsign / grid / lat,lon), TX power, antenna gain,
detection bandwidth, and system Tsys, the page shows in real time:
- Moon azimuth / elevation from the observer (ticks every 10 s)
- Slant range to the Moon
- EIRP breakdown: TX(dBm) + gain(dBi) = EIRP(dBm)
- Path-loss decomposition (Earth → Moon → Earth):
free-space spreading (×2)
− moon reflection gain 10·log(4π σ / λ²) where σ = ρ·π·R²
= round-trip path loss
- Band-dependent lunar albedo ρ (VHF ≈ 0.065, microwave ≈ 0.08,
mm-wave ≈ 0.02) surfaced as a named line item with the current
ρ shown to three decimals
- Received power, thermal noise floor, and SNR with a badge that
classifies the margin (strong / marginal / below noise)
- Self-echo Doppler shift with its radial-velocity driver
New modules:
- `Microwaveprop.Moon` — low-precision lunar ephemeris
(Astronomical Almanac Sec. D.4 truncated series). Provides
`julian_day/1`, `geocentric/1`, `observer_position/3`, and
`radial_velocity_km_s/3`. Accurate to ~0.3° position / ~200 km
distance — well inside any amateur EME-dish beamwidth.
- `Microwaveprop.Propagation.Eme` — path-loss helpers:
`moon_albedo/1` (band lookup), `moon_rcs_m2/1`,
`fspl_round_trip_db/2`, `moon_reflection_gain_db/2`,
`path_loss_db/3`, `received_power_dbm/1`, `noise_floor_dbm/2`,
`snr_db/1`, `doppler_shift_hz/2`.
Wired into the main nav ("EME" button) and the router between
/path and /rover.
Tests: 3 properties + 41 unit tests cover JD conversion, moon
position envelope, band-dependent albedo lookup, decomposition
identity, linearity/monotonicity of path loss and Doppler, and a
LiveView integration test against the rendered DOM.
Full suite: 2,460 tests + 170 properties; credo strict clean.
Root cause of 14 stuck hrrr-queued contacts: hrrr_point_rs's CONUS-only
grid silently writes zero profiles for out-of-bounds points (UK,
Winnipeg, Alberta, >50° lat), but the fetch task still completes
successfully. ContactWeatherEnqueueWorker's hrrr_placeholder_jobs
always emitted a queued placeholder for any contact with a pos1,
which meant mark_hrrr_status! re-flagged :queued every backfill tick
forever.
Introduce Propagation.Grid.contains?/1 as the canonical in-CONUS
check (inclusive on all four edges, nil-safe). Use it in two places
of ContactWeatherEnqueueWorker:
- hrrr_placeholder_jobs/1 now returns [] for OCONUS contacts, same
as pre-2014 / already-complete ones.
- mark_hrrr_status!/3 empty-list clause now branches on Grid.contains?
first (→ :unavailable) before falling through to the existing
NARR-coverage / :complete logic.
Next backfill tick will flip the 14 stuck OCONUS contacts in prod to
:unavailable automatically. The remaining 13 "CONUS" stuck ones are
right at the 50° edge (lat=50.1875 rounds outside the box) — same
reconciliation path.
Coverage additions:
- Grid.contains?/1 — corner-inclusive, OCONUS positive/negative
fixtures based on the actual stuck-contact pos1 values, nil/missing-
key/nil-value handling.
- enqueue_for_contact regression tests: pos1 in the UK and pos1 just
north of the lat cap both land at :unavailable, not :queued.
Suite: 2,416 tests + 159 properties (was 2,409 + 159); credo strict
clean.
Two breaking API changes in 0.2:
- `stash_assigns/2` is gone. Keys to persist are now declared on the
`use LiveStash, stored_keys: [...]` macro and `stash/1` picks them
up automatically.
- The TTL unit switched to seconds — this codebase never called the
TTL setter, so no config change is needed.
Updated both callers (MapLive persists selected_band + selected_time;
SubmitLive persists active_tab). `recover_state/1` signature is
unchanged. Full LiveView suite passes (67 tests).
Changes under lib/ (source):
- iemre_fetch_worker: replace `if transient_failure?/1` boolean predicate
+ `if/else` with a classifier that pattern-matches the error reason
directly, returning `:transient | :permanent`, and dispatch via
`case`. Three-clause head is clearer than the boolean predicate.
- scores_file.extract_points and nexrad_client.extract_box: force `//1`
step on the row/col comprehensions so an image box whose centre
lands outside the raster collapses to an empty iteration instead
of firing Elixir 1.19's ambiguous-range warning. Adds a regression
test for the right-edge box.
Test coverage added:
- Feature wrappers in Backtest.Features (theta_e_jump, shear_at_top,
duct_thickness, best_duct_freq, duct_usable_for_band,
distance_to_front / parallel_to_front stubs, all_features/0).
- NotifyListener handle_info tolerance for malformed payloads +
unknown messages, plus the PubSub broadcast side effect.
- Viewshed.effective_reach_km across every verdict / diffraction
tier / ducting-score tier combination.
- SnmpClient parse_af60_output unit conversions + unknown-column
handling; parse_snmpget_output OID normalisation and junk-line
tolerance.
- HrrrNativeClient native_level_count, native_variables,
native_messages shape, duct_messages / duct_byte_ranges.
- Changeset coverage for GeomagneticObservation, SolarFluxObservation,
SolarXrayObservation, Ionosphere.Observation, HrrrClimatology, and
Metar5minObservation — lifted each from ~50% / 0% to 100%.
- Property tests: GefsClient.dewpoint_from_rh invariants (Td ≤ T,
monotonic in RH, finite across the operating envelope) + idempotent
nearest_run. NexradClient.pixel_to_dbz monotonicity + bounded
output; latlon_to_pixel round-trip for every grid cell.
Also cleared several unrelated compiler/linter warnings:
- Three private test helpers (create_contact, insert_contact,
current_hour) had `\\ %{}` / `\\ 0` defaults that no caller used.
- profiles_file_test bound `dir` in a pattern match without using it.
Suite: 2,359 tests + 146 properties pass (was 2,300 / 139); coverage
70.38% → 70.89%; credo strict clean.
Rust workers (prop-grid-rs) that die mid-claim (SIGKILL, OOM, node
drain) leave grid_tasks rows stuck in status='running' forever,
because claim_next uses FOR UPDATE SKIP LOCKED and nothing resets the
orphan. The /status page then shows a permanent spinner with stale
f00/f10/f18 badges — currently 21 rows claimed as far back as
2026-04-20.
GridTaskEnqueuer.reclaim_stale_running/1 flips rows whose claimed_at
is older than 15 minutes back to 'queued'. Rows that have already
burned through 5 claim/reclaim cycles become 'failed' with a
reclaim-orphan error so the next hourly seed can replace them.
Wired into PropagationGridWorker.seed_chain/0 so it runs every :05
cron tick before new rows are seeded.
Also rename the status panel "Retrying" column to "Failed" — it was
always showing terminal `failed` rows, never retrying ones.
IemreFetchWorker now writes an empty stub observation when IEM returns
a permanent-failure status (404, 422), mirroring the existing {:ok, []}
path. The backfill cron's next tick sees the stub via
has_iemre_observation?/3, generates no job, and mark_status!/3 flips
the contact's iemre_status from :queued to :complete — draining the
51 contacts that have been stuck behind cancelled out-of-grid IEMRE
jobs.
Raise hot pod CPU limit 2 → 3 so BEAM gets 3 schedulers online.
Observed run queues of [2, 10, 0, 0, 0, 0] on the 2-scheduler
configuration during the :05 propagation chain, starving /live and
/health probe handlers and tripping intermittent liveness timeouts.
ADIF records frequently include operator commentary in the
<NOTES> field (multi-line, detailed) or <COMMENT> (short
one-liner). Now that contacts have a notes column, pass that
value through the import pipeline instead of dropping it.
Preference is NOTES over COMMENT to match the ADIF 3.1.4 spec's
intended split — NOTES is the richer field. Whitespace-only
values collapse to nil so a program that always emits a blank
NOTES tag doesn't fill every row with an empty string.
Tests cover NOTES capture, COMMENT fallback, NOTES-over-COMMENT
preference, and the absent-field nil case. ADIF upload blurb
lists the new carry-over so users see it before confirming.
Operators want a place to jot observations that aren't captured by
the structured fields — weather anecdotes, propagation mode
commentary, equipment details, band conditions. Add a text column
on contacts plus the two ingest paths users submit through:
* /submit single-contact form gets a 3-row textarea under the
other fields. Optional, max 2000 chars, with a live length cap
via maxlength. Whitespace-only input collapses to NULL so the
column reflects "no notes" rather than an empty string.
* CSV importer recognises a `notes` (or `note`) column via the
existing header_aliases table and flows values straight through
submission_changeset. Sample template gets a matching example
row with embedded commas so the quoted-field round-trip is
exercised on download.
ADIF import and the refinement/refinement-notify paths are out of
scope — users specifically asked for CSV + single-contact. Tests
cover the changeset (accept/blank/over-length), CSV header parsing
(plain + RFC-4180 quoted), and LiveView form submit end-to-end.
When the Rust hrrr-point-worker marks a task as `failed` (upstream
`idx HTTP 404` or wgrib2 decode error on old archive formats), the
corresponding contacts sit at `hrrr_status = :queued` until the
generic 3-day stale-queued reconcile flips them. Speed that up: on
every BackfillEnqueue tick, join contacts to `hrrr_fetch_tasks` by
rounded HRRR hour and flip any `:queued` contact whose cycle is
permanently `failed` to `:unavailable` immediately.
Motivation: a handful of 2018-08-04/05 cycles (t20z/t21z/t15z/t17z)
are genuinely missing `wrfsfcf`/`wrfprsf` files on the NOAA S3
archive — other hours on the same dates are fine, so this is an
upstream data gap rather than a structural pre-2019 issue. Users
see an accurate "Partial" marker on those contacts instead of a
three-day `:queued` stall.
Tested: 4 new cases covering rounding, failed-vs-done, and the
`hrrr not in types` skip branch; full suite 2283/0.
The log_level(%{method: \"HEAD\"}) clause never matched because
Plug.Head rewrites the method to \"GET\" before Plug.Telemetry's
register_before_send callback fires, so by log time conn.method is
\"GET\". Stash the original method in private before Plug.Head runs
and key the filter off it.
