The /skewt page was empty in prod because Phase 2 of the Rust cutover
moved f01..f18 grid scoring to Rust but left profile-file writing only
on the analysis (f00) path. Result: /data/scores/profiles/ accumulated
one file per chain run instead of nineteen, so SkewtLive only ever saw
the analysis hour — and that hour falls outside the 1 h past cutoff
within an hour of the chain finishing, leaving the page empty most of
the time.
Two surgical fixes:
* `pipeline::run_chain_step` now builds `profile_entries` alongside
`prepared` from the merged grid in a single iter pass, then spawns
`profiles_file::write_atomic` on the blocking pool to overlap the
NFS write with the score-band scoring/write fan-out — same pattern
`run_analysis_step` already uses. `ChainStepStats` gains
`profile_cells_written` so the worker log line is symmetric with
the analysis step's existing field and a future failure mode that
drops the profile would show up as a divergence.
* `SkewtLive.available_valid_times/0` widens the past cutoff from
1 h to 3 h. The chain's analysis hour can be ~70 min old by the
time the next chain finishes (and ~2 h on a missed cycle), so the
1 h cutoff was leaving the page blank during normal transients.
18 h forward window unchanged.
Cosmetic: cargo fmt picked up trivial rewrites in decoder/fetcher/
native_duct/hrrr_point_worker that had drifted; rolled in.
Verification: cargo build --release, cargo clippy --lib --bins -D
warnings, cargo test --lib (134/134), mix test
test/microwaveprop_web/live/skewt_live_test.exs (3/3) all green.
Both retired in light of the 2026-04-25 revision report
(`docs/algo-reports/2026-04-25-algo-revisions.md`):
* HPBL boundary-layer multiplier in `Scorer.score_refractivity/{3,4,5}`
is gone. On the n=47,418 10 GHz HRRR-matched corpus rho_hpbl is
+0.004; binned distance is flat to within ±5 % across 200–2,000 m.
The previously reported 2.3× ratio between shallow and deep HPBL
bins was a small-corpus artefact that disappeared once the matched
corpus passed ~5,000 contacts.
* Native-profile 1.15× duct boost is gone. At 10 GHz on n=52,341
matched contacts, cells where `best_duct_band_ghz` ≥ band ran
*shorter* than no-duct cells (198 km vs 211 km, n=173 vs 44,658).
The Bulk Richardson gate that existed only to suppress that boost
in turbulent conditions is also retired; arity preserved on every
`score_refractivity` overload so callers compile unchanged.
Same retirements applied in the Rust port (`rust/prop_grid_rs/src/scorer.rs`)
to keep the Elixir/Rust parity tests green.
algo.md updated end-to-end:
* Finding 4 (HPBL) rewritten to "no usable signal" with the
n=47,418 bin table.
* Finding 5 (gradient) tightened: load-bearing only at 24 GHz,
set band-specific gradient weight to 0 outside [10, 47] GHz on
the next derive_band_weights run.
* Finding 8 (time-of-day) augmented with the robust hour-bucketed
amplitude index (40 / 82 / 101 % at 10 / 24 / 47 GHz) and a
selection-bias caveat for VHF/UHF (where 129–148 % amplitude
reflects evening contest scheduling, not propagation).
* Finding 9 (sounding) refreshed against the n=27,058 corpus.
* Part 2c "Native-profile duct boost" rewritten as
"retired 2026-04-25" with the falsifying join.
* Part 2c "Commercial-link inverse sensor" promoted from deferred
with the 22-day, 38k-sample SNMP corpus, the +0.61 / −0.61 PWAT
and pressure correlations against the 37-hour HRRR overlap, and
the 04–05 / 10–12 local two-peak morning fade window.
* Part 2b "Pressure" annotated with the U-shape at 10 GHz and the
next-iteration target.
* Part 2d "What stayed, what left" + "Open items" updated to
reflect the four code/doc moves.
119 Elixir scorer tests green, 134 Rust scorer tests green, 2,888
total Elixir tests + 221 properties green via `mix precommit`.
The "Data gaps" section was a flat key/value dump that buried what was
actually broken (hrrr_climatology=0, rtma=0, metar5=0) under metrics
that are merely informational (pre2014_contacts=207). Operators reading
the report had to know each metric's healthy range by heart.
This commit:
* Classifies every gap metric BROKEN / WARN / OK via DATA_GAP_RULES,
with a one-line remediation hint per non-OK row (e.g. "run
`mix hrrr_climatology`").
* Prints BROKEN items to stderr at runtime so a CI/cron run surfaces
them without anyone opening the markdown file.
* Adds NARR-coverage-by-year and HRRR-status-by-year tables. With
only 358 NARR rows, the year breakdown shows whether the backfill
ever started; with 36k pending HRRR contacts, the year breakdown
shows whether the backlog is a stalled historical backfill or a
stuck live queue — different remediations.
* Reports HRRR enrichment completeness as a percentage instead of
forcing the reader to do the division.
Single search bar accepts an address, Maidenhead grid square, or
callsign; resolves it to lat/lon via Geocoder / Maidenhead /
CallsignLocation (cheapest classification first), snaps to the HRRR
grid via the existing ProfilesFile reader, and renders an SVG
Skew-T-Log-P with isobars, skewed isotherms, dry/moist adiabats, and
mixing-ratio lines. A button row picks between every available
forecast hour (now → f18); default is the most recent analysis.
Right pane lists derived stability and refractivity stats from
SoundingParams.derive (PWAT, BL depth, K-index, lifted index, min
dN/dh, ducting status + per-duct base/top/ΔM).
Renderer is server-side SVG so the page works without JS and serializes
into the LiveView payload as a single tag. Wind barbs are deliberately
omitted — HRRR's persisted profile carries wind only at 10 m AGL.
Each fix is covered by a regression test that fails on `main` and
passes on this commit.
Round 1 (initial review):
* propagation: thread `latitude` into the conditions map so
`score_season/4` actually picks up regional multipliers
* hrrr_client / fetcher.rs: `nearest_hrrr_hour` rounds DOWN, never at
a future cycle that NOAA hasn't published yet
* radio: spherical-vector great-circle midpoint replaces the
arithmetic mean — anti-meridian paths no longer fold to Greenwich
* weather: `reconcile_weather_statuses` scales the longitude band by
`1 / cos(lat)` so the bbox stays ~150 km wide at every latitude
* radio/maidenhead: clamp 90°/180° below the field-bucket overflow so
`from_latlon` never emits invalid characters like 'S'
* prop_grid_rs/pipeline: merge HRRR + NEXRAD-derived rain rates and
read `best_duct_freq_ghz` into `best_duct_band_ghz` so the Native
Duct Boost actually fires
* propagation/region (Elixir + Rust): inclusive upper bounds so points
exactly at lat_max get the regional multiplier
* weather/sounding_params (Elixir + Rust): drop the 10 m gradient
floor so HRRR's thin near-surface layers stop hiding sharp ducts
* weather/sounding_params: when the profile ends inside a duct,
finalize it with the highest sample as the top instead of throwing
it away (Rust port already correct)
Round 2 (post-fix sweep):
* radio + commercial: single canonical haversine in Radio (atan2
form); Commercial delegates instead of carrying a second copy that
could disagree at threshold distances
* prop_grid_rs/profiles_file: `snap_coords` matches Elixir's
step-aware snap (`round(coord/0.125) * 0.125`, then 3-dp round) so
Rust-keyed and Elixir-keyed profile maps land on the same cell
* weather/grib2/wgrib2: `parse_lon_val_segment` uses `Float.parse`
uniformly — wgrib2 dropping the trailing `.0` from a longitude no
longer crashes the whole chain step
`handle_event("switch_tab", ...)` ran `String.to_existing_atom/1` on
the raw `tab` value from the browser, so a stale, forged, or
mistyped client event with any unknown string crashed the LiveView
with `ArgumentError: not an already existing atom`.
Compares against an explicit `@valid_tabs` allowlist (`:single`,
`:csv`, `:adif`) — known values switch the active tab, anything
else is ignored.
`apply_edit_to_contact/2` is the path for both owner direct-edits and
admin reviewed-edits. The public contact-map + total-count + gzipped
controller payload caches are built from `private == false` rows, but
only the insert path was clearing them — every other update left
/api/contacts/map and /contacts/map serving the previous snapshot for
up to 10 minutes. A `private: false -> true` flip on a contact already
present in the cached payload was a temporary privacy leak; grid
corrections, callsign changes, and flagged_invalid flips silently
diverged from the live row.
Extracts the three `Cache.invalidate` calls into
`invalidate_contact_map_caches/0` and runs it from both
`apply_edit_to_contact/2` and the existing insert path so the public
view stays in sync.
The /contacts/:id edit form always submits the `private` checkbox, but
`current_value/2` had no `"private"` clause and fell through to the
catch-all `nil`. The Contact schema defaults `private: false`, so an
unchanged box compared `false != nil` and was always tagged as a
change — for non-owners that meant a spurious pending review queued
on every save, and for owners/admins it triggered a no-op direct
update.
Adds the missing clause so the diff sees boolean-vs-boolean and
correctly returns `:no_changes` when the box is untouched.
The /contacts/:id edit form pre-fills qso_timestamp via
`Calendar.strftime(ts, "%Y-%m-%d %H:%M")` and submits whatever the
user typed back unchanged. Two failure modes resulted:
1. `diff_against_contact/2` compared that string against the stored
`%DateTime{}`, never matched, and treated every untouched submit
as a modification.
2. `build_contact_changes/2` then ran `DateTime.from_iso8601/1` on
the space-separated, no-seconds, no-Z form and crashed with
`MatchError` on `{:error, :invalid_format}` — direct owner/admin
submits raised instead of saving.
Adds `normalize_timestamp_field/2` to `normalize_proposed/1` that
parses both the form's `"YYYY-MM-DD HH:MM[:SS]"` and the strict
`"YYYY-MM-DDTHH:MM:SSZ"` shapes into a UTC `%DateTime{}` (with
`NaiveDateTime.from_iso8601` doing the heavy lifting after a small
seconds-padding pass). Empty strings drop the field; unparseable
input drops it too so a typo becomes "no changes" rather than a
500.
Bandit's dual-stack listener delivers IPv4 connections to the app as
IPv4-mapped IPv6 tuples (`::ffff:a.b.c.d`). The trust check compared
that 128-bit form against our IPv4 trusted_proxies CIDRs and never
matched, so cf-connecting-ip was ignored on every cloudflared-relayed
request — every visitor logged as the cloudflared sidecar pod IP
(rendered as `0:0:0:0:0:65535:2804:101` etc.).
Collapses the mapped tuple to plain IPv4 before the trust check and
records it that way on the conn, so logs and session storage see the
real client IP that Cloudflare set in cf-connecting-ip.
The top navbar in the root layout already linked to /eme, but the
in-page side sidebars on /map and /weather (mobile + desktop) skipped
straight from Path Calculator to Beacons. Adds the EME entry between
them in all four panels so the Earth-Moon-Earth calculator is reachable
without bouncing through the navbar.
Also demotes the LiveStashGuard "stash skipped" warning to debug. It's
the known OTP 28 / LiveStash 0.2.0 ArgumentError that fires on every
event triggering a stash; the guard already handles it, the noise was
adding nothing.
Adds a `layer` query parameter to `/weather`. Loading
`/weather?layer=temp_700mb` boots straight into that overlay; clicking
a different layer pill push_patches the URL so the active selection is
shareable and survives a refresh. Unknown layer ids fall back to the
default (temperature) instead of crashing.
NOAA publish latency for HRRR pressure-level f18 straddles ~50-65 min
after cycle hour. The HH:05 cron frequently probes a (now-1h) cycle
that's not yet on S3 and falls back to (now-2h). Until today the next
re-attempt was a full hour later, leaving weather visibly stale.
Quadrupling the schedule lets pick_run_time/1 upgrade to the freshest
cycle within 15 min of it publishing. seed_with_analysis is idempotent
(on_conflict: :nothing), so re-fires against an already-seeded run_time
are a single cheap INSERT batch.
/weather mount called Weather.latest_weather_grid/1 directly, which
runs WeatherLayers.derive over 92k cells and Jason-encodes the
result. Production HTTP logs showed dead renders consistently
landing at 9.8–10.2 s — right at LiveView's socket join deadline.
The browser would render half the page, the WS would fail to mount,
and the user saw "attempting to reconnect".
The JS hook already fires map_bounds via requestAnimationFrame
immediately after mount, and the existing handle_event("map_bounds")
path returns viewport-scoped weather data via the update_weather
push event. So the dead-render data fetch is redundant — drop it.
Mount now returns initial_data_json: "[]"; the layer paints
~50–200 ms after WS connect via the bounds-driven flow.
PropagationGridWorker hard-coded `now-2h` for run_time selection on
the conservative theory that NOAA always finishes publishing within
two hours. In practice f18 is on the mirror by HH:55–HH:00 of the
next hour, so the cron at HH:05 was always picking a cycle that was
already an hour stale.
`HrrrClient.cycle_available?/1` HEADs the wrfprsf18 idx (the slowest
file in the cycle) — a 200 means every earlier forecast hour is also
on disk. `pick_run_time/1` probes `now-1h` first; if the probe
returns true we use that cycle, otherwise we fall back to the
existing `now-2h` slot. Test hook via :hrrr_cycle_available_fn env
keeps the behaviour fully exercisable without hitting NOAA.
End-to-end: weather/score data lag drops from ~2 h to ~1 h on every
hour where NOAA delivers on time. The fallback path keeps the chain
running on slow-publish hours.
850 mb T alone shows that warm air is present aloft but doesn't reveal
whether it's the textbook cap structure — a warm 700 mb above an EML
plume. The three new layers expose the canonical capping diagnostics
visually:
* T @ 700 mb — ≥10 °C is the southern Plains "moderate cap" threshold
* Td @ 700 mb — wide T-Td depression at 700 mb signals the EML plume
* Lapse 850→700 — steep mid-layer lapse rate (≥7 °C/km) over a moist
boundary layer is the cap mechanism itself
All three derive purely from the existing pressure-level profile (Rust
already fetches up through 700 mb), so no Rust pipeline changes needed.
ScoreCache held 437 entries × ~1.94 MiB each (~850 MiB per pod) of
{band_mhz, valid_time} grids — data already on NFS as compact .prop
files. NotifyListener and ScoreCacheReconciler both eagerly materialised
every band into ETS on each Rust completion / 60s sweep, so 5 hot
replicas wasted ~4.2 GiB of redundant cache and OOMed at 6 GiB.
Bound the cache to 32 entries with eviction by oldest valid_time, drop
the eager warm loops, and let LiveView callers lazy-fill via the
existing ScoresFile read path. ScoreCacheReconciler had no remaining
purpose and is deleted; runbook + prom_ex counters updated to match.
Steady-state cache footprint ~60 MiB per pod instead of ~850 MiB.
pg_stat_user_tables tracks two timestamps — `last_autoanalyze`
is only bumped by the autovacuum daemon, while `last_analyze` is
bumped by manual ANALYZE statements (including the ones our Task
issues). The previous query only filtered on `last_autoanalyze`,
so on a re-run within the skip_recent window every manually-
analyzed table came back for another ANALYZE pass.
Uses `GREATEST(last_analyze, last_autoanalyze)` so fresh manual
analyses count as recent.
Running ANALYZE serially against the 13M-row HRRR monthly partitions
took longer than the kubectl exec idle timeout (~300s per partition
on the Turing Pi 2), so the one-shot RPC kept getting killed before
the big partitions ran. Moves the loop into a supervised Task on
the caller's node so the RPC returns immediately; the BEAM keeps
running ANALYZE regardless of kubectl connection state.
Also sorts tables ascending by n_live_tup so small tables finish
first and their fresh stats are visible to the planner right away.
Progress is logged per table via `Logger.info`.
Two issues spotted in pg_stat_user_tables:
1. Most `hrrr_profiles` partitions have `last_autoanalyze IS NULL`.
The partitions are append-only from the hourly grid worker, so
autovacuum never crosses the analyze threshold — but without
stats the planner falls back to estimates and picks bad joins
on per-QSO lookups.
2. `contacts`, `grid_tasks`, `hrrr_fetch_tasks`, and
`contact_common_volume_radar` see high UPDATE volumes
(enrichment status flips, queue claim/complete cycles, mechanism
classifier writes) but only accrue ~5% dead before the default
20% / 10% autovacuum triggers fire. BackfillEnqueueWorker's
status-priority ORDER BY + the Rust workers' FOR UPDATE SKIP
LOCKED claim scan both want fresher stats than that.
Changes:
- Migration `20260424204656_tune_autovacuum_for_high_churn` sets
per-table `autovacuum_*_scale_factor` + `_threshold = 50` on the
four tables above. Aggressive enough to keep stats current;
tame enough that vacuum doesn't thrash the Turing Pi 2 Postgres
node. Reversible.
- `Weather.analyze_all/1` walks `pg_stat_user_tables`, runs
`ANALYZE` on every table not auto-analyzed in the last 6 h,
reports counts. Meant to be invoked once via
`bin/microwaveprop rpc 'Microwaveprop.Weather.analyze_all()'`
after deploy to seed statistics on the cold partitions;
idempotent on re-run.
The three tables on /status (Job Queue, Rust Workers, Table Sizes)
were collapsing to fit the viewport on narrow phones, wrapping
header text ("Siz…") and splitting values like "160 GB" into two
lines ("160." / "GB"). The overflow-x-auto wrappers were already
present but column auto-sizing still squeezed width below content.
Adds `whitespace-nowrap` to each `<table>` so cells never wrap, and
`-mx-2 px-2` on the overflow wrapper so horizontal scroll extends
edge-to-edge of the mobile viewport instead of being inset by the
layout's outer padding. Tables now scroll horizontally as intended
on mobile; desktop layout unchanged.
Root cause of the "81,980 / 81,994" display that wouldn't advance:
14 contacts had pos1 inside the HRRR CONUS grid but pos2 or the
great-circle midpoint landed outside it (50°N+ into Canada, mid-
Atlantic, Pacific, Caribbean, Alaska). The Rust hrrr-point-worker
silently returns no profile for OCONUS points, so
`hrrr_data_fully_present?/1` never evaluated true and the contact
sat `:queued` forever.
The existing ContactWeatherEnqueueWorker.mark_hrrr_status!/3 already
flips to `:unavailable` on empty job lists via NarrClient.in_coverage?
+ Grid.contains?(pos1), but those checks only considered pos1 — not
the full contact_path_points list. When pos1 was in-grid but a
downstream point was not, the contact entered `:queued` and the
reconciler couldn't rescue it.
`reconcile_hrrr_statuses/0` now sweeps `:queued` contacts into two
buckets:
- ANY path point OCONUS → `:unavailable` (matches what the enqueuer
would emit had it known about the downstream points);
- else, all points present → `:complete`.
Applied once in prod to 14 stuck contacts; cron path picks up any
new occurrences going forward.
The Rust hrrr_points worker now persists surface_refractivity /
min_refractivity_gradient / ducting_detected at upsert time (previous
commit), but the ~80k rows written before that fix have the scalar
columns NULL even though the pressure-level `profile` JSONB is
populated. Re-fetching them would re-pull tens of GB of HRRR grids
we already parsed.
Weather.backfill_hrrr_scalars/1 streams rows with
`surface_refractivity IS NULL`, runs the existing
SoundingParams.derive/1 against the stored profile, and writes the
scalars back — no network, no GRIB decode. Batched (500 rows per
tick, 1000 tick cap) to keep transactions short on the Turing Pi 2
PG node.
Meant to be run once via `bin/microwaveprop rpc` after deploy.
Idempotent; the WHERE clause excludes already-backfilled rows.
Found by auditing the ProfilesFile, hrrr_profiles, and weather-layer
contracts against what the Rust writers actually emit.
Bug 1 (HIGH) — Rust `hrrr_points` worker upserted hrrr_profiles rows
with NULL `surface_refractivity`, `min_refractivity_gradient`, and
`ducting_detected`. The Elixir HrrrClient path always wrote them
via SoundingParams.derive. Per-QSO pages that rely on these
scalars ("N:", "dN/dh:", duct badge) showed "—" for any contact
enriched through Stream C. Fixed by porting
`surface_refractivity` + `ducting_detected` into Rust's
`sounding_params.rs` and deriving at upsert time.
Bug 2 (HIGH) — Rust f01..f18 `cell_to_profile_entry` never emitted
wind_u / wind_v / cloud_cover_pct / precip_mm into ProfilesFile
cells. `Propagation.factors_for` recomputes factor scores from the
profile cell on /map click, and these four fields are read
(wind_speed_kts, sky_cover_pct, precip_to_rate_mmhr). Result: every
f01..f18 point-detail popup showed 0 for wind/sky/rain factors.
Fixed by extracting the grib values at cell_to_profile_entry time
and whitelisting the keys for atomization in
profiles_file.ex `@mp_atom_keys`.
Bug 4 (MEDIUM) — `WeatherLayers.duct_field/2` only knew the
`SoundingParams.detect_ducts` shape (`d["base"]` / `d["strength"]`).
The Rust/HrrrNativeClient shape uses `:base_m` / `:thickness_m`
atom keys (post-ProfilesFile.read atomization). On /weather, the
Duct Base / Duct Strength layers returned nil for every Rust-origin
cell. Tolerant lookup now reads either shape; uses `thickness_m`
as the "strength" proxy for the native shape (thicker duct traps
a wider band).
Skipped (verified but lower impact):
- Rust ProfilesFile aggregates ducts as scalars, not per-layer — the
`ducts` array in the popup is always empty. Requires larger
DuctMetrics refactor in Rust; deferred.
- `complete_hrrr_task` emits no NOTIFY (vs `complete` which does).
Elixir has no matching listener yet, so adding NOTIFY alone is a
no-op; the existing cron reconciler catches it.
- `:best_duct_band_ghz` fallback in propagation.ex is dead code;
cleanup, not a bug.
133 Rust tests + 2842 Elixir tests + credo green.
The Rust f01..f18 pipeline writes ProfilesFile cells, and the Rust
hrrr_points worker writes hrrr_profiles rows, with profile entries
keyed "pres_mb" / "hght_m" / "tmpc" / "dwpc" (string keys pre-
atomization; atom keys post-ProfilesFile.read). Elixir's
SoundingParams.derive and WeatherLayers.sort_profile filtered on
"pres" / "hght" — so every derived field returned nil for any
Rust-provided profile.
Visible symptoms on /weather: N-gradient, Refractivity, T @ 850mb,
Td @ 850mb, Lapse Rate, Inversion, Inv. Base all rendered no overlay.
For per-contact analysis, min_refractivity_gradient from the Rust
HRRR point worker's rows silently dropped.
- SoundingParams.normalize_profile_entry/1: single-source normalizer
accepting legacy, Rust-string, and Rust-atom shapes; returns the
canonical "pres"/"hght"/"tmpc"/"dwpc"/"drct"/"sknt" shape.
- SoundingParams.derive/1 and WeatherLayers.sort_profile/1 both run
every entry through it before the rest of the existing logic.
- Weather.build_grid_cache_row/4 now falls back to native_min_gradient
when SoundingParams can't derive one (surface-only profiles).
Also:
- Default /weather overlay changed to Temperature (LiveView assign +
JS fallback in the hook's dataset default) per user request.
Extends the weather-status reconciler with symmetric sweeps for HRRR
and IEMRE. Same failure mode: the worker writes data to the shared
table but has no back-pointer to the contact that triggered the
fetch, so contact-level status never flips unless a user views the
page.
- Weather.reconcile_iemre_statuses/0: iterates :queued contacts;
flips :complete when every path point has an iemre_observations
row at the rounded 0.125° grid cell for the qso_timestamp's date.
- Weather.reconcile_hrrr_statuses/0: same shape; flips when
hrrr_data_fully_present?/1 holds.
- Both run alongside the existing weather sweep at the tail of
ContactWeatherEnqueueWorker.perform/1.
Kept in Elixir (not SQL) because contact_path_points/1 emits 1-3
points per contact and grid-rounding each in pure SQL is awkward.
The stuck-at-steady-state count is always tiny (<20), so the
iteration is cheap.
