The top navbar in the root layout already linked to /eme, but the
in-page side sidebars on /map and /weather (mobile + desktop) skipped
straight from Path Calculator to Beacons. Adds the EME entry between
them in all four panels so the Earth-Moon-Earth calculator is reachable
without bouncing through the navbar.
Also demotes the LiveStashGuard "stash skipped" warning to debug. It's
the known OTP 28 / LiveStash 0.2.0 ArgumentError that fires on every
event triggering a stash; the guard already handles it, the noise was
adding nothing.
Adds a `layer` query parameter to `/weather`. Loading
`/weather?layer=temp_700mb` boots straight into that overlay; clicking
a different layer pill push_patches the URL so the active selection is
shareable and survives a refresh. Unknown layer ids fall back to the
default (temperature) instead of crashing.
NOAA publish latency for HRRR pressure-level f18 straddles ~50-65 min
after cycle hour. The HH:05 cron frequently probes a (now-1h) cycle
that's not yet on S3 and falls back to (now-2h). Until today the next
re-attempt was a full hour later, leaving weather visibly stale.
Quadrupling the schedule lets pick_run_time/1 upgrade to the freshest
cycle within 15 min of it publishing. seed_with_analysis is idempotent
(on_conflict: :nothing), so re-fires against an already-seeded run_time
are a single cheap INSERT batch.
/weather mount called Weather.latest_weather_grid/1 directly, which
runs WeatherLayers.derive over 92k cells and Jason-encodes the
result. Production HTTP logs showed dead renders consistently
landing at 9.8–10.2 s — right at LiveView's socket join deadline.
The browser would render half the page, the WS would fail to mount,
and the user saw "attempting to reconnect".
The JS hook already fires map_bounds via requestAnimationFrame
immediately after mount, and the existing handle_event("map_bounds")
path returns viewport-scoped weather data via the update_weather
push event. So the dead-render data fetch is redundant — drop it.
Mount now returns initial_data_json: "[]"; the layer paints
~50–200 ms after WS connect via the bounds-driven flow.
PropagationGridWorker hard-coded `now-2h` for run_time selection on
the conservative theory that NOAA always finishes publishing within
two hours. In practice f18 is on the mirror by HH:55–HH:00 of the
next hour, so the cron at HH:05 was always picking a cycle that was
already an hour stale.
`HrrrClient.cycle_available?/1` HEADs the wrfprsf18 idx (the slowest
file in the cycle) — a 200 means every earlier forecast hour is also
on disk. `pick_run_time/1` probes `now-1h` first; if the probe
returns true we use that cycle, otherwise we fall back to the
existing `now-2h` slot. Test hook via :hrrr_cycle_available_fn env
keeps the behaviour fully exercisable without hitting NOAA.
End-to-end: weather/score data lag drops from ~2 h to ~1 h on every
hour where NOAA delivers on time. The fallback path keeps the chain
running on slow-publish hours.
850 mb T alone shows that warm air is present aloft but doesn't reveal
whether it's the textbook cap structure — a warm 700 mb above an EML
plume. The three new layers expose the canonical capping diagnostics
visually:
* T @ 700 mb — ≥10 °C is the southern Plains "moderate cap" threshold
* Td @ 700 mb — wide T-Td depression at 700 mb signals the EML plume
* Lapse 850→700 — steep mid-layer lapse rate (≥7 °C/km) over a moist
boundary layer is the cap mechanism itself
All three derive purely from the existing pressure-level profile (Rust
already fetches up through 700 mb), so no Rust pipeline changes needed.
ScoreCache held 437 entries × ~1.94 MiB each (~850 MiB per pod) of
{band_mhz, valid_time} grids — data already on NFS as compact .prop
files. NotifyListener and ScoreCacheReconciler both eagerly materialised
every band into ETS on each Rust completion / 60s sweep, so 5 hot
replicas wasted ~4.2 GiB of redundant cache and OOMed at 6 GiB.
Bound the cache to 32 entries with eviction by oldest valid_time, drop
the eager warm loops, and let LiveView callers lazy-fill via the
existing ScoresFile read path. ScoreCacheReconciler had no remaining
purpose and is deleted; runbook + prom_ex counters updated to match.
Steady-state cache footprint ~60 MiB per pod instead of ~850 MiB.
pg_stat_user_tables tracks two timestamps — `last_autoanalyze`
is only bumped by the autovacuum daemon, while `last_analyze` is
bumped by manual ANALYZE statements (including the ones our Task
issues). The previous query only filtered on `last_autoanalyze`,
so on a re-run within the skip_recent window every manually-
analyzed table came back for another ANALYZE pass.
Uses `GREATEST(last_analyze, last_autoanalyze)` so fresh manual
analyses count as recent.
Running ANALYZE serially against the 13M-row HRRR monthly partitions
took longer than the kubectl exec idle timeout (~300s per partition
on the Turing Pi 2), so the one-shot RPC kept getting killed before
the big partitions ran. Moves the loop into a supervised Task on
the caller's node so the RPC returns immediately; the BEAM keeps
running ANALYZE regardless of kubectl connection state.
Also sorts tables ascending by n_live_tup so small tables finish
first and their fresh stats are visible to the planner right away.
Progress is logged per table via `Logger.info`.
Two issues spotted in pg_stat_user_tables:
1. Most `hrrr_profiles` partitions have `last_autoanalyze IS NULL`.
The partitions are append-only from the hourly grid worker, so
autovacuum never crosses the analyze threshold — but without
stats the planner falls back to estimates and picks bad joins
on per-QSO lookups.
2. `contacts`, `grid_tasks`, `hrrr_fetch_tasks`, and
`contact_common_volume_radar` see high UPDATE volumes
(enrichment status flips, queue claim/complete cycles, mechanism
classifier writes) but only accrue ~5% dead before the default
20% / 10% autovacuum triggers fire. BackfillEnqueueWorker's
status-priority ORDER BY + the Rust workers' FOR UPDATE SKIP
LOCKED claim scan both want fresher stats than that.
Changes:
- Migration `20260424204656_tune_autovacuum_for_high_churn` sets
per-table `autovacuum_*_scale_factor` + `_threshold = 50` on the
four tables above. Aggressive enough to keep stats current;
tame enough that vacuum doesn't thrash the Turing Pi 2 Postgres
node. Reversible.
- `Weather.analyze_all/1` walks `pg_stat_user_tables`, runs
`ANALYZE` on every table not auto-analyzed in the last 6 h,
reports counts. Meant to be invoked once via
`bin/microwaveprop rpc 'Microwaveprop.Weather.analyze_all()'`
after deploy to seed statistics on the cold partitions;
idempotent on re-run.
The three tables on /status (Job Queue, Rust Workers, Table Sizes)
were collapsing to fit the viewport on narrow phones, wrapping
header text ("Siz…") and splitting values like "160 GB" into two
lines ("160." / "GB"). The overflow-x-auto wrappers were already
present but column auto-sizing still squeezed width below content.
Adds `whitespace-nowrap` to each `<table>` so cells never wrap, and
`-mx-2 px-2` on the overflow wrapper so horizontal scroll extends
edge-to-edge of the mobile viewport instead of being inset by the
layout's outer padding. Tables now scroll horizontally as intended
on mobile; desktop layout unchanged.
Root cause of the "81,980 / 81,994" display that wouldn't advance:
14 contacts had pos1 inside the HRRR CONUS grid but pos2 or the
great-circle midpoint landed outside it (50°N+ into Canada, mid-
Atlantic, Pacific, Caribbean, Alaska). The Rust hrrr-point-worker
silently returns no profile for OCONUS points, so
`hrrr_data_fully_present?/1` never evaluated true and the contact
sat `:queued` forever.
The existing ContactWeatherEnqueueWorker.mark_hrrr_status!/3 already
flips to `:unavailable` on empty job lists via NarrClient.in_coverage?
+ Grid.contains?(pos1), but those checks only considered pos1 — not
the full contact_path_points list. When pos1 was in-grid but a
downstream point was not, the contact entered `:queued` and the
reconciler couldn't rescue it.
`reconcile_hrrr_statuses/0` now sweeps `:queued` contacts into two
buckets:
- ANY path point OCONUS → `:unavailable` (matches what the enqueuer
would emit had it known about the downstream points);
- else, all points present → `:complete`.
Applied once in prod to 14 stuck contacts; cron path picks up any
new occurrences going forward.
The Rust hrrr_points worker now persists surface_refractivity /
min_refractivity_gradient / ducting_detected at upsert time (previous
commit), but the ~80k rows written before that fix have the scalar
columns NULL even though the pressure-level `profile` JSONB is
populated. Re-fetching them would re-pull tens of GB of HRRR grids
we already parsed.
Weather.backfill_hrrr_scalars/1 streams rows with
`surface_refractivity IS NULL`, runs the existing
SoundingParams.derive/1 against the stored profile, and writes the
scalars back — no network, no GRIB decode. Batched (500 rows per
tick, 1000 tick cap) to keep transactions short on the Turing Pi 2
PG node.
Meant to be run once via `bin/microwaveprop rpc` after deploy.
Idempotent; the WHERE clause excludes already-backfilled rows.
Found by auditing the ProfilesFile, hrrr_profiles, and weather-layer
contracts against what the Rust writers actually emit.
Bug 1 (HIGH) — Rust `hrrr_points` worker upserted hrrr_profiles rows
with NULL `surface_refractivity`, `min_refractivity_gradient`, and
`ducting_detected`. The Elixir HrrrClient path always wrote them
via SoundingParams.derive. Per-QSO pages that rely on these
scalars ("N:", "dN/dh:", duct badge) showed "—" for any contact
enriched through Stream C. Fixed by porting
`surface_refractivity` + `ducting_detected` into Rust's
`sounding_params.rs` and deriving at upsert time.
Bug 2 (HIGH) — Rust f01..f18 `cell_to_profile_entry` never emitted
wind_u / wind_v / cloud_cover_pct / precip_mm into ProfilesFile
cells. `Propagation.factors_for` recomputes factor scores from the
profile cell on /map click, and these four fields are read
(wind_speed_kts, sky_cover_pct, precip_to_rate_mmhr). Result: every
f01..f18 point-detail popup showed 0 for wind/sky/rain factors.
Fixed by extracting the grib values at cell_to_profile_entry time
and whitelisting the keys for atomization in
profiles_file.ex `@mp_atom_keys`.
Bug 4 (MEDIUM) — `WeatherLayers.duct_field/2` only knew the
`SoundingParams.detect_ducts` shape (`d["base"]` / `d["strength"]`).
The Rust/HrrrNativeClient shape uses `:base_m` / `:thickness_m`
atom keys (post-ProfilesFile.read atomization). On /weather, the
Duct Base / Duct Strength layers returned nil for every Rust-origin
cell. Tolerant lookup now reads either shape; uses `thickness_m`
as the "strength" proxy for the native shape (thicker duct traps
a wider band).
Skipped (verified but lower impact):
- Rust ProfilesFile aggregates ducts as scalars, not per-layer — the
`ducts` array in the popup is always empty. Requires larger
DuctMetrics refactor in Rust; deferred.
- `complete_hrrr_task` emits no NOTIFY (vs `complete` which does).
Elixir has no matching listener yet, so adding NOTIFY alone is a
no-op; the existing cron reconciler catches it.
- `:best_duct_band_ghz` fallback in propagation.ex is dead code;
cleanup, not a bug.
133 Rust tests + 2842 Elixir tests + credo green.
The Rust f01..f18 pipeline writes ProfilesFile cells, and the Rust
hrrr_points worker writes hrrr_profiles rows, with profile entries
keyed "pres_mb" / "hght_m" / "tmpc" / "dwpc" (string keys pre-
atomization; atom keys post-ProfilesFile.read). Elixir's
SoundingParams.derive and WeatherLayers.sort_profile filtered on
"pres" / "hght" — so every derived field returned nil for any
Rust-provided profile.
Visible symptoms on /weather: N-gradient, Refractivity, T @ 850mb,
Td @ 850mb, Lapse Rate, Inversion, Inv. Base all rendered no overlay.
For per-contact analysis, min_refractivity_gradient from the Rust
HRRR point worker's rows silently dropped.
- SoundingParams.normalize_profile_entry/1: single-source normalizer
accepting legacy, Rust-string, and Rust-atom shapes; returns the
canonical "pres"/"hght"/"tmpc"/"dwpc"/"drct"/"sknt" shape.
- SoundingParams.derive/1 and WeatherLayers.sort_profile/1 both run
every entry through it before the rest of the existing logic.
- Weather.build_grid_cache_row/4 now falls back to native_min_gradient
when SoundingParams can't derive one (surface-only profiles).
Also:
- Default /weather overlay changed to Temperature (LiveView assign +
JS fallback in the hook's dataset default) per user request.
Extends the weather-status reconciler with symmetric sweeps for HRRR
and IEMRE. Same failure mode: the worker writes data to the shared
table but has no back-pointer to the contact that triggered the
fetch, so contact-level status never flips unless a user views the
page.
- Weather.reconcile_iemre_statuses/0: iterates :queued contacts;
flips :complete when every path point has an iemre_observations
row at the rounded 0.125° grid cell for the qso_timestamp's date.
- Weather.reconcile_hrrr_statuses/0: same shape; flips when
hrrr_data_fully_present?/1 holds.
- Both run alongside the existing weather sweep at the tail of
ContactWeatherEnqueueWorker.perform/1.
Kept in Elixir (not SQL) because contact_path_points/1 emits 1-3
points per contact and grid-rounding each in pure SQL is awkward.
The stuck-at-steady-state count is always tiny (<20), so the
iteration is cheap.
2,019 contacts sat permanently in weather_status=:queued even after
their ASOS data had been ingested. The only code path that flipped
:queued → :complete was MicrowavepropWeb.ContactLive.Show on page
view — contacts that nobody visited stayed :queued forever.
Adds Weather.reconcile_weather_statuses/0, a single SQL UPDATE that
flips every :queued contact whose ±2h / 150km window now contains
at least one surface observation. Called at the tail of the
existing ContactWeatherEnqueueWorker cron so the correction runs
regardless of whether anyone opens the UI.
Visible effect: the status page's "Weather done" count (previously
capped at 79,975 while data kept landing) will converge on the real
total after the next cron tick.
2838 tests + credo green.
Replaces per-QSO-endpoint asos_batch granularity with per-station-day
granularity. Each contact in the same UTC day near the same station
now produces identical unique-args and collapses to ONE Oban job, so
the backlog shrinks as cross-contact duplicates dedup.
Each fetch also covers a full 24h window (vs the previous 4h) so one
IEM request returns ~24 hourly observations instead of ~1-2. The
effective bytes-per-request is 10x higher, amortizing the
IemRateLimiter gap + 429 retry tail across far more data.
Changes:
- Weather.station_day_covered?/2 + station_day_pairs_covered/1 — cheap
UTC-day coverage checks for the enqueuer + worker skip paths.
- WeatherFetchWorker gains an "asos_day" clause that fetches the full
UTC day for one station and upserts. Skips if the day is already
covered in the DB.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one asos_day
job per (station_id, UTC date) pair. ±2h windows crossing midnight
enumerate both dates.
- Microwaveprop.Weather.RebatchAsos.to_day_jobs/1 migrates any
already-queued asos/asos_batch jobs into the new shape. Idempotent;
supports dry_run.
- asos_batch worker clause retained for jobs already in flight at
deploy time.
2835 tests + credo green.
Mix.Task.run isn't available in a compiled release, so the in-prod
`bin/microwaveprop eval 'Mix.Tasks.Weather.RebatchAsos.run(...)'`
dispatch crashes. Moves the real logic into a plain module
(`Microwaveprop.Weather.RebatchAsos`) that both the Mix task and a
release eval can call. Uses IO.puts over Mix.shell for the same
reason.
- IemRateLimiter gains AIMD-style adaptive spacing. signal_429/0
widens the current gap (*= 1.5, capped at max_interval_ms default
10s); signal_success/0 narrows it back toward the configured base
(*= 0.92, floored at interval_ms). Self-tunes to IEM's moving
ceiling without needing the manual "safe for 4 pods" constant.
- IemClient now routes every response through a central handle_response
helper that fires the widen/narrow feedback signals, eliminating the
four near-identical case blocks.
- Mix.Tasks.Weather.RebatchAsos collapses any already-queued single-
station "asos" jobs into the batched "asos_batch" shape the
enqueuer now emits, so the pending backfill queue converts to the
new per-request-efficient path instead of draining at the old rate.
Idempotent; supports --dry-run.
2833 tests + credo green.
IEM's ASOS CSV endpoint accepts multiple station= params in a single
request. Prior code enqueued one WeatherFetchWorker job per nearby
station per QSO endpoint — N jobs × 1 station each — which paid the
IemRateLimiter's 1500ms gap and the IEM 429-retry tail N times per
contact.
Changes:
- IemClient.asos_url/3 accepts a list of station codes.
- IemClient.fetch_asos_batch/3 fetches N stations in one call and
returns rows grouped by station_code (with absent codes filled as
empty lists so callers can stub them).
- parse_asos_csv/1 now exposes the station_code column it was
previously discarding.
- WeatherFetchWorker gains an "asos_batch" fetch_type clause that
unpacks rows per (station_id, station_code), upserting or stubbing
each. The single-station "asos" clause stays for already-queued
retryable jobs.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one batch
job per (lat, lon, 4h window) covering every uncovered nearby
station (sorted for deterministic unique-args).
Expected effect on backfill: ~10-20x fewer IEM requests per contact
enrichment cycle, matching drop in 429 retry traffic.
Removes the six OTel packages (opentelemetry, _api, _exporter,
_phoenix, _oban, _bandit) and their 9 transitive deps. Tracing was
only wired for Phoenix/Bandit/Oban auto-instrumentation — PromEx
already covers the metrics we actually use on the status page +
Grafana. Also drops the OTEL_EXPORTER_OTLP_ENDPOINT env var and
its comment block from both deployment manifests.
No translations are actually maintained and no UI is shipped in a
non-English locale. Replaced the handful of gettext(...) calls in
core_components + layouts with their literal English strings, and
rewrote translate_error/1 as a small %{key} interpolator over the
Ecto changeset error tuple.
Removes gettext + expo (compile dep). sutra_ui stays — it's a
transitive dep of live_table's TableComponent.
- chrono: drop "serde" (no DateTime serde serialization in code)
- uuid: drop "serde" (no Uuid serde serialization in code)
- sqlx: drop "macros" (no compile-time sqlx::query! macros in use)
