The /path calculator showed "0 / 9 HRRR points" in production despite
the on-disk profile store being current. Root cause: profile_from_cell/2
treated the cell's :profile key as a wrapper sub-map and called
Map.put_new on it — but the :profile key actually holds the vertical
pressure-level LIST. Every point sample crashed with BadMapError, the
crash propagated as {:exit, _} through Task.async_stream, and the
consumer silently dropped all 9 results.
Fix: stop wrapping. Cells are already flat HrrrProfile-shaped maps;
just stamp lat/lon (from the caller, since cells don't carry their
own coords — those are the map key) and valid_time onto the cell.
Audit + log every other async error path so the next silent failure
isn't invisible:
- PathLive HRRR point lookup
- Propagation.point_forecast per-hour reads
- Viewshed ray crashes
- IemClient ASOS network fetches
- RtmaClient range-download tasks
- Recalibrator factor-vector batches (positive + negative samples)
- MapLive forecast preload tasks
- RoverLive station resolution
LiveViews already had handle_async/3 exit clauses with logging. The
gap was always in Task.async_stream consumers that wrote {:exit, _} -> []
without surfacing the reason.
Add the rule to CLAUDE.md and project memory so this never repeats.
Also fix a pre-existing skewt_svg.ex compiler warning where
@critical_label_min_dy was used before being defined.
The 18-hour forecast chart on /path now shows the same weighted-
criteria panel the /map popup builds when the user clicks a grid
cell. Hovering a dot fires a server roundtrip that fetches
Propagation.point_detail at the path's midpoint for that valid_time;
results are cached client-side so re-hovering is instant.
`Propagation.point_detail/4` now returns a `:profile_source` tag —
`:exact`, `{:fallback, fallback_valid_time}`, or `:unavailable` —
distinguishing real per-hour breakdowns (Rust pipeline writes a
profile file for every f00..f18 step) from rare cycle-miss fallbacks
where we approximate from a recent analysis profile. Stale comment
about "only f00 persists profiles" updated.
Map utilities (FACTOR_META, FACTOR_ORDER, factorBar,
factorExplanation, scoreTier, FactorMeta, ScoreTier) are now exported
from propagation_map_hook so the new path hook reuses the exact same
look + copy.
The grid-cached profile is trimmed to 13 levels (1000→700 mb) for
memory reasons, so the skew-T plot showed nothing above 700 mb.
HrrrClient.fetch_profile/4 now accepts a forecast_hour opt; SkewtLive
issues a per-point fetch against the same cycle the cached row came
from to get the full 1000→100 mb set, falling back to the cached
truncated cell on any failure.
Time selection no longer round-trips through the URL — `select_time`
updates state in-process and triggers a focused :load_skewt_time
async, so refreshing or sharing a /skewt?q=... link always lands on
the most recent analysis hour.
Three additions that bring the diagram closer to the SPC sounding
viewer's full-information chart:
* Wet-bulb temperature trace, plotted as a thinner blue line under
the dewpoint trace. Comes from a Stipanuk-style bracket between
the level's dewpoint and dry-bulb temperatures (already used by
`SkewtParams` for the wet-bulb-zero level — now public so the
renderer can call it per level).
* Environment virtual-temperature trace and parcel virtual-T
trajectory, both as thin pinkish dashed lines. The parcel
virtual-T is the actual buoyancy curve CAPE integrates against,
so showing it next to the dry-bulb parcel ascent makes the
moisture correction visible. New `SkewtParams.virtual_temp_c/2`
and `/3` helpers, plus `SkewtParams.mixing_ratio/2`.
* Critical-level rail on the right edge. Each of LCL / LFC / EL /
0 °C / WBZ that resolves on the current profile gets a short
horizontal tick at the right of the plot and a `<label> <p> mb`
tag just outside it — matches the SPC chart's `143 mb (mix)` /
`158 mb (mix)` annotation column. SkewtLive computes the
pressures (and converts the height-keyed 0 °C and WBZ from
SkewtParams back to pressure via `pressure_at_height/2`) before
handing them to `SkewtSvg.render/2`.
Wind-derived overlays (hodograph, wind barbs) remain off — HRRR
profile data is wind-free above 10 m AGL.
mix test: 27 / 27 across the four /skewt-related test files,
2,902 / 2,902 + 221 properties on the full suite. mix credo --strict
clean (one cyclomatic-complexity refactor extracted along the way).
When the on-disk ProfilesFile is empty SkewtLive falls back to the
hrrr_profiles Postgres table, but per-QSO HrrrFetchWorker rows
sometimes land with a partial profile (4–12 levels) when individual
byte-range fetches fail. The fallback was happily picking those rows,
which rendered a stubby Skew-T missing the upper troposphere.
HrrrProfileLookup.{list_valid_times_near, read_point_near}/3 now
filter on `coalesce(cardinality(profile), 0) >= min_profile_levels`,
defaulting to 13 (HRRR's standard pressure-level set: 1000/925/850/
700/500/400/300/250/200/150/100 mb plus the two boundary surfaces).
Pass `min_profile_levels: 0` to keep the partial rows for diagnostic
queries.
Confirmed against the dev mirror: the recent table has ~3.1 M rows
at 13 levels and ~250 k at 4–12 levels; the new default skips the
partial slice cleanly.
Tests: 8/8 HrrrProfileLookupTest cases (added two new ones for the
threshold). mix credo --strict clean.
Two changes that bring /skewt visually and analytically into parity
with the NOAA SPC sounding viewer
(https://www.spc.noaa.gov/exper/soundings/):
NOAA-style line set
-------------------
Updated the SkewtSvg style block to match the public SPC chart:
* Temperature trace ........ pure red (#ff0000) at 2.5 px stroke
* Dewpoint trace ........... pure green (#00aa00) at 2.5 px stroke
* Isotherms ................ dashed teal (#14b8a6)
* Dry adiabats ............. solid orange (#f59e0b)
* Moist (saturated) adiabats dashed sky-blue (#38bdf8)
* Mixing-ratio lines ....... dotted darker green (#16a34a), with
labels at 700 mb showing the g/kg
value, drawn only below 600 mb (matches
the SPC chart's lower-band number row)
* Parcel ascent ............ new dashed grey (#475569) line, drawn
whenever SkewtParams.derive returns a
non-empty parcel_trace
The mixing-ratio set is the SPC standard (0.4 / 0.7 / 1 / 2 / 3 / 5 /
8 / 12 / 16 / 20 g/kg) instead of the prior arbitrary set.
SkewtParams: SPC parcel-theory + indices module
-----------------------------------------------
New `Microwaveprop.Weather.SkewtParams` derives the parameter set that
ships in the SPC `*.txt` companion file:
* LCL pressure / temperature / height (Bolton 1980 eq 22)
* LFC pressure / height
* EL pressure / height
* SBCAPE, SBCIN (J/kg, virtual-temperature integration with
surface parcel)
* 3 km CAPE
* Lifted Index (parcel ↔ environment T at 500 mb)
* Freezing level (T = 0 °C height) and Wet-Bulb Zero
* DCAPE (downdraft CAPE — picks the min-θₑ source level in
400–700 mb, descends moist-adiabatically to surface)
* Layer lapse rates: sfc–3 km, 700–500 mb, 850–500 mb
* `parcel_trace` — pressure/temperature trajectory used by
SkewtSvg to draw the dashed parcel-ascent overlay
Wind-derived indices (SRH, BWD, Bunkers motion, SCP, STP, SHIP) are
deliberately *not* produced and the LiveView surfaces a one-line note
explaining why: HRRR persists wind only at 10 m AGL, so per-pressure-
level shear can't be computed from this data source.
LiveView panel restructured into grouped sections (Surface, Convective
parcel, Levels, Lapse rates, Moisture/stability, Refractivity) so the
layout matches the SPC viewer's order. The Levels rows show pressure
*and* height ("872 mb · 1,140 m") for quick cross-reference between
the diagram axis and the readout.
Tests: 8/8 new SkewtParamsTest cases (LCL Bolton check, saturated-air
edge case, full field-set presence, summer profile produces SBCAPE > 0
and LI < 0, capped profile produces SBCAPE = 0 and LI > 0, freezing
level interpolated correctly, sfc–3 km lapse rate, empty profile no
crash). 25/25 across all skewt-related test files. mix test green
(2,902 / 2,902 + 221 properties), mix credo --strict clean, mix
assets.build clean.
Two follow-ups to the earlier /skewt work:
1. Interactive crosshair (matches the SPC-viewer feel from the
reference screenshot). Mousing over the diagram now drops a
horizontal pressure cursor with a top-left readout box showing the
pressure at the cursor's altitude (mb), the height in m and ft,
and the linearly interpolated temperature and dewpoint at that
level. Two coloured dots track the T and Td traces along the
cursor so the user can read the spread visually.
Mechanics: SkewtSvg.geometry/0 exposes the plot geometry (left,
right, top, bottom, p_bottom, p_top, t_min, t_max, skew) as JSON;
SkewtSvg emits an initially-hidden <g class="skewt-cursor"> layer.
A new Skewt hook in assets/js/app.ts inverts pressure_y to recover
pressure from the cursor's viewBox-Y, walks the (descending-pres-
sorted) profile to interp T/Td/height, and updates the elements.
The hook also handles mouseleave to hide the cursor and uses the
SVG's screen-CTM so the math stays correct under any container
aspect-ratio resize.
2. DB fallback for `available_valid_times`/`load_profile`. When the
on-disk `ProfilesFile` has nothing for the location-and-window
(true on dev, transient on prod between chain runs), SkewtLive
now falls back to `HrrrProfileLookup.{list_valid_times_near,
read_point_near}/3` against the `hrrr_profiles` Postgres table.
The lookup uses the same ±0.07° spatial tolerance and the
`(lat, lon, valid_time)` unique index that `Weather.find_nearest_
hrrr/3` already relies on, so the queries are index-only.
Returned shape matches `ProfilesFile.read_point/3` so SkewtLive
reads from either source transparently. NaiveDateTime → UTC
normalisation is centralised in `ensure_utc/1` so the LiveView
sees `%DateTime{time_zone: "Etc/UTC"}` regardless of which side
produced the value.
Tests: 6/6 new HrrrProfileLookupTest cases (list/limit/tolerance,
read nearest cell, nil for far cells, picks closest among matches),
3/3 SkewtLiveTest still green, 2,894 total Elixir tests + 221
properties green via mix test (one pre-existing flaky test in
admin/contact_edit_live re-runs clean). mix credo --strict clean.
mix assets.build clean.
Both retired in light of the 2026-04-25 revision report
(`docs/algo-reports/2026-04-25-algo-revisions.md`):
* HPBL boundary-layer multiplier in `Scorer.score_refractivity/{3,4,5}`
is gone. On the n=47,418 10 GHz HRRR-matched corpus rho_hpbl is
+0.004; binned distance is flat to within ±5 % across 200–2,000 m.
The previously reported 2.3× ratio between shallow and deep HPBL
bins was a small-corpus artefact that disappeared once the matched
corpus passed ~5,000 contacts.
* Native-profile 1.15× duct boost is gone. At 10 GHz on n=52,341
matched contacts, cells where `best_duct_band_ghz` ≥ band ran
*shorter* than no-duct cells (198 km vs 211 km, n=173 vs 44,658).
The Bulk Richardson gate that existed only to suppress that boost
in turbulent conditions is also retired; arity preserved on every
`score_refractivity` overload so callers compile unchanged.
Same retirements applied in the Rust port (`rust/prop_grid_rs/src/scorer.rs`)
to keep the Elixir/Rust parity tests green.
algo.md updated end-to-end:
* Finding 4 (HPBL) rewritten to "no usable signal" with the
n=47,418 bin table.
* Finding 5 (gradient) tightened: load-bearing only at 24 GHz,
set band-specific gradient weight to 0 outside [10, 47] GHz on
the next derive_band_weights run.
* Finding 8 (time-of-day) augmented with the robust hour-bucketed
amplitude index (40 / 82 / 101 % at 10 / 24 / 47 GHz) and a
selection-bias caveat for VHF/UHF (where 129–148 % amplitude
reflects evening contest scheduling, not propagation).
* Finding 9 (sounding) refreshed against the n=27,058 corpus.
* Part 2c "Native-profile duct boost" rewritten as
"retired 2026-04-25" with the falsifying join.
* Part 2c "Commercial-link inverse sensor" promoted from deferred
with the 22-day, 38k-sample SNMP corpus, the +0.61 / −0.61 PWAT
and pressure correlations against the 37-hour HRRR overlap, and
the 04–05 / 10–12 local two-peak morning fade window.
* Part 2b "Pressure" annotated with the U-shape at 10 GHz and the
next-iteration target.
* Part 2d "What stayed, what left" + "Open items" updated to
reflect the four code/doc moves.
119 Elixir scorer tests green, 134 Rust scorer tests green, 2,888
total Elixir tests + 221 properties green via `mix precommit`.
Each fix is covered by a regression test that fails on `main` and
passes on this commit.
Round 1 (initial review):
* propagation: thread `latitude` into the conditions map so
`score_season/4` actually picks up regional multipliers
* hrrr_client / fetcher.rs: `nearest_hrrr_hour` rounds DOWN, never at
a future cycle that NOAA hasn't published yet
* radio: spherical-vector great-circle midpoint replaces the
arithmetic mean — anti-meridian paths no longer fold to Greenwich
* weather: `reconcile_weather_statuses` scales the longitude band by
`1 / cos(lat)` so the bbox stays ~150 km wide at every latitude
* radio/maidenhead: clamp 90°/180° below the field-bucket overflow so
`from_latlon` never emits invalid characters like 'S'
* prop_grid_rs/pipeline: merge HRRR + NEXRAD-derived rain rates and
read `best_duct_freq_ghz` into `best_duct_band_ghz` so the Native
Duct Boost actually fires
* propagation/region (Elixir + Rust): inclusive upper bounds so points
exactly at lat_max get the regional multiplier
* weather/sounding_params (Elixir + Rust): drop the 10 m gradient
floor so HRRR's thin near-surface layers stop hiding sharp ducts
* weather/sounding_params: when the profile ends inside a duct,
finalize it with the highest sample as the top instead of throwing
it away (Rust port already correct)
Round 2 (post-fix sweep):
* radio + commercial: single canonical haversine in Radio (atan2
form); Commercial delegates instead of carrying a second copy that
could disagree at threshold distances
* prop_grid_rs/profiles_file: `snap_coords` matches Elixir's
step-aware snap (`round(coord/0.125) * 0.125`, then 3-dp round) so
Rust-keyed and Elixir-keyed profile maps land on the same cell
* weather/grib2/wgrib2: `parse_lon_val_segment` uses `Float.parse`
uniformly — wgrib2 dropping the trailing `.0` from a longitude no
longer crashes the whole chain step
`apply_edit_to_contact/2` is the path for both owner direct-edits and
admin reviewed-edits. The public contact-map + total-count + gzipped
controller payload caches are built from `private == false` rows, but
only the insert path was clearing them — every other update left
/api/contacts/map and /contacts/map serving the previous snapshot for
up to 10 minutes. A `private: false -> true` flip on a contact already
present in the cached payload was a temporary privacy leak; grid
corrections, callsign changes, and flagged_invalid flips silently
diverged from the live row.
Extracts the three `Cache.invalidate` calls into
`invalidate_contact_map_caches/0` and runs it from both
`apply_edit_to_contact/2` and the existing insert path so the public
view stays in sync.
The /contacts/:id edit form always submits the `private` checkbox, but
`current_value/2` had no `"private"` clause and fell through to the
catch-all `nil`. The Contact schema defaults `private: false`, so an
unchanged box compared `false != nil` and was always tagged as a
change — for non-owners that meant a spurious pending review queued
on every save, and for owners/admins it triggered a no-op direct
update.
Adds the missing clause so the diff sees boolean-vs-boolean and
correctly returns `:no_changes` when the box is untouched.
The /contacts/:id edit form pre-fills qso_timestamp via
`Calendar.strftime(ts, "%Y-%m-%d %H:%M")` and submits whatever the
user typed back unchanged. Two failure modes resulted:
1. `diff_against_contact/2` compared that string against the stored
`%DateTime{}`, never matched, and treated every untouched submit
as a modification.
2. `build_contact_changes/2` then ran `DateTime.from_iso8601/1` on
the space-separated, no-seconds, no-Z form and crashed with
`MatchError` on `{:error, :invalid_format}` — direct owner/admin
submits raised instead of saving.
Adds `normalize_timestamp_field/2` to `normalize_proposed/1` that
parses both the form's `"YYYY-MM-DD HH:MM[:SS]"` and the strict
`"YYYY-MM-DDTHH:MM:SSZ"` shapes into a UTC `%DateTime{}` (with
`NaiveDateTime.from_iso8601` doing the heavy lifting after a small
seconds-padding pass). Empty strings drop the field; unparseable
input drops it too so a typo becomes "no changes" rather than a
500.
PropagationGridWorker hard-coded `now-2h` for run_time selection on
the conservative theory that NOAA always finishes publishing within
two hours. In practice f18 is on the mirror by HH:55–HH:00 of the
next hour, so the cron at HH:05 was always picking a cycle that was
already an hour stale.
`HrrrClient.cycle_available?/1` HEADs the wrfprsf18 idx (the slowest
file in the cycle) — a 200 means every earlier forecast hour is also
on disk. `pick_run_time/1` probes `now-1h` first; if the probe
returns true we use that cycle, otherwise we fall back to the
existing `now-2h` slot. Test hook via :hrrr_cycle_available_fn env
keeps the behaviour fully exercisable without hitting NOAA.
End-to-end: weather/score data lag drops from ~2 h to ~1 h on every
hour where NOAA delivers on time. The fallback path keeps the chain
running on slow-publish hours.
850 mb T alone shows that warm air is present aloft but doesn't reveal
whether it's the textbook cap structure — a warm 700 mb above an EML
plume. The three new layers expose the canonical capping diagnostics
visually:
* T @ 700 mb — ≥10 °C is the southern Plains "moderate cap" threshold
* Td @ 700 mb — wide T-Td depression at 700 mb signals the EML plume
* Lapse 850→700 — steep mid-layer lapse rate (≥7 °C/km) over a moist
boundary layer is the cap mechanism itself
All three derive purely from the existing pressure-level profile (Rust
already fetches up through 700 mb), so no Rust pipeline changes needed.
ScoreCache held 437 entries × ~1.94 MiB each (~850 MiB per pod) of
{band_mhz, valid_time} grids — data already on NFS as compact .prop
files. NotifyListener and ScoreCacheReconciler both eagerly materialised
every band into ETS on each Rust completion / 60s sweep, so 5 hot
replicas wasted ~4.2 GiB of redundant cache and OOMed at 6 GiB.
Bound the cache to 32 entries with eviction by oldest valid_time, drop
the eager warm loops, and let LiveView callers lazy-fill via the
existing ScoresFile read path. ScoreCacheReconciler had no remaining
purpose and is deleted; runbook + prom_ex counters updated to match.
Steady-state cache footprint ~60 MiB per pod instead of ~850 MiB.
pg_stat_user_tables tracks two timestamps — `last_autoanalyze`
is only bumped by the autovacuum daemon, while `last_analyze` is
bumped by manual ANALYZE statements (including the ones our Task
issues). The previous query only filtered on `last_autoanalyze`,
so on a re-run within the skip_recent window every manually-
analyzed table came back for another ANALYZE pass.
Uses `GREATEST(last_analyze, last_autoanalyze)` so fresh manual
analyses count as recent.
Running ANALYZE serially against the 13M-row HRRR monthly partitions
took longer than the kubectl exec idle timeout (~300s per partition
on the Turing Pi 2), so the one-shot RPC kept getting killed before
the big partitions ran. Moves the loop into a supervised Task on
the caller's node so the RPC returns immediately; the BEAM keeps
running ANALYZE regardless of kubectl connection state.
Also sorts tables ascending by n_live_tup so small tables finish
first and their fresh stats are visible to the planner right away.
Progress is logged per table via `Logger.info`.
Two issues spotted in pg_stat_user_tables:
1. Most `hrrr_profiles` partitions have `last_autoanalyze IS NULL`.
The partitions are append-only from the hourly grid worker, so
autovacuum never crosses the analyze threshold — but without
stats the planner falls back to estimates and picks bad joins
on per-QSO lookups.
2. `contacts`, `grid_tasks`, `hrrr_fetch_tasks`, and
`contact_common_volume_radar` see high UPDATE volumes
(enrichment status flips, queue claim/complete cycles, mechanism
classifier writes) but only accrue ~5% dead before the default
20% / 10% autovacuum triggers fire. BackfillEnqueueWorker's
status-priority ORDER BY + the Rust workers' FOR UPDATE SKIP
LOCKED claim scan both want fresher stats than that.
Changes:
- Migration `20260424204656_tune_autovacuum_for_high_churn` sets
per-table `autovacuum_*_scale_factor` + `_threshold = 50` on the
four tables above. Aggressive enough to keep stats current;
tame enough that vacuum doesn't thrash the Turing Pi 2 Postgres
node. Reversible.
- `Weather.analyze_all/1` walks `pg_stat_user_tables`, runs
`ANALYZE` on every table not auto-analyzed in the last 6 h,
reports counts. Meant to be invoked once via
`bin/microwaveprop rpc 'Microwaveprop.Weather.analyze_all()'`
after deploy to seed statistics on the cold partitions;
idempotent on re-run.
Root cause of the "81,980 / 81,994" display that wouldn't advance:
14 contacts had pos1 inside the HRRR CONUS grid but pos2 or the
great-circle midpoint landed outside it (50°N+ into Canada, mid-
Atlantic, Pacific, Caribbean, Alaska). The Rust hrrr-point-worker
silently returns no profile for OCONUS points, so
`hrrr_data_fully_present?/1` never evaluated true and the contact
sat `:queued` forever.
The existing ContactWeatherEnqueueWorker.mark_hrrr_status!/3 already
flips to `:unavailable` on empty job lists via NarrClient.in_coverage?
+ Grid.contains?(pos1), but those checks only considered pos1 — not
the full contact_path_points list. When pos1 was in-grid but a
downstream point was not, the contact entered `:queued` and the
reconciler couldn't rescue it.
`reconcile_hrrr_statuses/0` now sweeps `:queued` contacts into two
buckets:
- ANY path point OCONUS → `:unavailable` (matches what the enqueuer
would emit had it known about the downstream points);
- else, all points present → `:complete`.
Applied once in prod to 14 stuck contacts; cron path picks up any
new occurrences going forward.
The Rust hrrr_points worker now persists surface_refractivity /
min_refractivity_gradient / ducting_detected at upsert time (previous
commit), but the ~80k rows written before that fix have the scalar
columns NULL even though the pressure-level `profile` JSONB is
populated. Re-fetching them would re-pull tens of GB of HRRR grids
we already parsed.
Weather.backfill_hrrr_scalars/1 streams rows with
`surface_refractivity IS NULL`, runs the existing
SoundingParams.derive/1 against the stored profile, and writes the
scalars back — no network, no GRIB decode. Batched (500 rows per
tick, 1000 tick cap) to keep transactions short on the Turing Pi 2
PG node.
Meant to be run once via `bin/microwaveprop rpc` after deploy.
Idempotent; the WHERE clause excludes already-backfilled rows.
Found by auditing the ProfilesFile, hrrr_profiles, and weather-layer
contracts against what the Rust writers actually emit.
Bug 1 (HIGH) — Rust `hrrr_points` worker upserted hrrr_profiles rows
with NULL `surface_refractivity`, `min_refractivity_gradient`, and
`ducting_detected`. The Elixir HrrrClient path always wrote them
via SoundingParams.derive. Per-QSO pages that rely on these
scalars ("N:", "dN/dh:", duct badge) showed "—" for any contact
enriched through Stream C. Fixed by porting
`surface_refractivity` + `ducting_detected` into Rust's
`sounding_params.rs` and deriving at upsert time.
Bug 2 (HIGH) — Rust f01..f18 `cell_to_profile_entry` never emitted
wind_u / wind_v / cloud_cover_pct / precip_mm into ProfilesFile
cells. `Propagation.factors_for` recomputes factor scores from the
profile cell on /map click, and these four fields are read
(wind_speed_kts, sky_cover_pct, precip_to_rate_mmhr). Result: every
f01..f18 point-detail popup showed 0 for wind/sky/rain factors.
Fixed by extracting the grib values at cell_to_profile_entry time
and whitelisting the keys for atomization in
profiles_file.ex `@mp_atom_keys`.
Bug 4 (MEDIUM) — `WeatherLayers.duct_field/2` only knew the
`SoundingParams.detect_ducts` shape (`d["base"]` / `d["strength"]`).
The Rust/HrrrNativeClient shape uses `:base_m` / `:thickness_m`
atom keys (post-ProfilesFile.read atomization). On /weather, the
Duct Base / Duct Strength layers returned nil for every Rust-origin
cell. Tolerant lookup now reads either shape; uses `thickness_m`
as the "strength" proxy for the native shape (thicker duct traps
a wider band).
Skipped (verified but lower impact):
- Rust ProfilesFile aggregates ducts as scalars, not per-layer — the
`ducts` array in the popup is always empty. Requires larger
DuctMetrics refactor in Rust; deferred.
- `complete_hrrr_task` emits no NOTIFY (vs `complete` which does).
Elixir has no matching listener yet, so adding NOTIFY alone is a
no-op; the existing cron reconciler catches it.
- `:best_duct_band_ghz` fallback in propagation.ex is dead code;
cleanup, not a bug.
133 Rust tests + 2842 Elixir tests + credo green.
The Rust f01..f18 pipeline writes ProfilesFile cells, and the Rust
hrrr_points worker writes hrrr_profiles rows, with profile entries
keyed "pres_mb" / "hght_m" / "tmpc" / "dwpc" (string keys pre-
atomization; atom keys post-ProfilesFile.read). Elixir's
SoundingParams.derive and WeatherLayers.sort_profile filtered on
"pres" / "hght" — so every derived field returned nil for any
Rust-provided profile.
Visible symptoms on /weather: N-gradient, Refractivity, T @ 850mb,
Td @ 850mb, Lapse Rate, Inversion, Inv. Base all rendered no overlay.
For per-contact analysis, min_refractivity_gradient from the Rust
HRRR point worker's rows silently dropped.
- SoundingParams.normalize_profile_entry/1: single-source normalizer
accepting legacy, Rust-string, and Rust-atom shapes; returns the
canonical "pres"/"hght"/"tmpc"/"dwpc"/"drct"/"sknt" shape.
- SoundingParams.derive/1 and WeatherLayers.sort_profile/1 both run
every entry through it before the rest of the existing logic.
- Weather.build_grid_cache_row/4 now falls back to native_min_gradient
when SoundingParams can't derive one (surface-only profiles).
Also:
- Default /weather overlay changed to Temperature (LiveView assign +
JS fallback in the hook's dataset default) per user request.
Extends the weather-status reconciler with symmetric sweeps for HRRR
and IEMRE. Same failure mode: the worker writes data to the shared
table but has no back-pointer to the contact that triggered the
fetch, so contact-level status never flips unless a user views the
page.
- Weather.reconcile_iemre_statuses/0: iterates :queued contacts;
flips :complete when every path point has an iemre_observations
row at the rounded 0.125° grid cell for the qso_timestamp's date.
- Weather.reconcile_hrrr_statuses/0: same shape; flips when
hrrr_data_fully_present?/1 holds.
- Both run alongside the existing weather sweep at the tail of
ContactWeatherEnqueueWorker.perform/1.
Kept in Elixir (not SQL) because contact_path_points/1 emits 1-3
points per contact and grid-rounding each in pure SQL is awkward.
The stuck-at-steady-state count is always tiny (<20), so the
iteration is cheap.
2,019 contacts sat permanently in weather_status=:queued even after
their ASOS data had been ingested. The only code path that flipped
:queued → :complete was MicrowavepropWeb.ContactLive.Show on page
view — contacts that nobody visited stayed :queued forever.
Adds Weather.reconcile_weather_statuses/0, a single SQL UPDATE that
flips every :queued contact whose ±2h / 150km window now contains
at least one surface observation. Called at the tail of the
existing ContactWeatherEnqueueWorker cron so the correction runs
regardless of whether anyone opens the UI.
Visible effect: the status page's "Weather done" count (previously
capped at 79,975 while data kept landing) will converge on the real
total after the next cron tick.
2838 tests + credo green.
Replaces per-QSO-endpoint asos_batch granularity with per-station-day
granularity. Each contact in the same UTC day near the same station
now produces identical unique-args and collapses to ONE Oban job, so
the backlog shrinks as cross-contact duplicates dedup.
Each fetch also covers a full 24h window (vs the previous 4h) so one
IEM request returns ~24 hourly observations instead of ~1-2. The
effective bytes-per-request is 10x higher, amortizing the
IemRateLimiter gap + 429 retry tail across far more data.
Changes:
- Weather.station_day_covered?/2 + station_day_pairs_covered/1 — cheap
UTC-day coverage checks for the enqueuer + worker skip paths.
- WeatherFetchWorker gains an "asos_day" clause that fetches the full
UTC day for one station and upserts. Skips if the day is already
covered in the DB.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one asos_day
job per (station_id, UTC date) pair. ±2h windows crossing midnight
enumerate both dates.
- Microwaveprop.Weather.RebatchAsos.to_day_jobs/1 migrates any
already-queued asos/asos_batch jobs into the new shape. Idempotent;
supports dry_run.
- asos_batch worker clause retained for jobs already in flight at
deploy time.
2835 tests + credo green.
Mix.Task.run isn't available in a compiled release, so the in-prod
`bin/microwaveprop eval 'Mix.Tasks.Weather.RebatchAsos.run(...)'`
dispatch crashes. Moves the real logic into a plain module
(`Microwaveprop.Weather.RebatchAsos`) that both the Mix task and a
release eval can call. Uses IO.puts over Mix.shell for the same
reason.
- IemRateLimiter gains AIMD-style adaptive spacing. signal_429/0
widens the current gap (*= 1.5, capped at max_interval_ms default
10s); signal_success/0 narrows it back toward the configured base
(*= 0.92, floored at interval_ms). Self-tunes to IEM's moving
ceiling without needing the manual "safe for 4 pods" constant.
- IemClient now routes every response through a central handle_response
helper that fires the widen/narrow feedback signals, eliminating the
four near-identical case blocks.
- Mix.Tasks.Weather.RebatchAsos collapses any already-queued single-
station "asos" jobs into the batched "asos_batch" shape the
enqueuer now emits, so the pending backfill queue converts to the
new per-request-efficient path instead of draining at the old rate.
Idempotent; supports --dry-run.
2833 tests + credo green.
IEM's ASOS CSV endpoint accepts multiple station= params in a single
request. Prior code enqueued one WeatherFetchWorker job per nearby
station per QSO endpoint — N jobs × 1 station each — which paid the
IemRateLimiter's 1500ms gap and the IEM 429-retry tail N times per
contact.
Changes:
- IemClient.asos_url/3 accepts a list of station codes.
- IemClient.fetch_asos_batch/3 fetches N stations in one call and
returns rows grouped by station_code (with absent codes filled as
empty lists so callers can stub them).
- parse_asos_csv/1 now exposes the station_code column it was
previously discarding.
- WeatherFetchWorker gains an "asos_batch" fetch_type clause that
unpacks rows per (station_id, station_code), upserting or stubbing
each. The single-station "asos" clause stays for already-queued
retryable jobs.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one batch
job per (lat, lon, 4h window) covering every uncovered nearby
station (sorted for deterministic unique-args).
Expected effect on backfill: ~10-20x fewer IEM requests per contact
enrichment cycle, matching drop in 429 retry traffic.
Removes the six OTel packages (opentelemetry, _api, _exporter,
_phoenix, _oban, _bandit) and their 9 transitive deps. Tracing was
only wired for Phoenix/Bandit/Oban auto-instrumentation — PromEx
already covers the metrics we actually use on the status page +
Grafana. Also drops the OTEL_EXPORTER_OTLP_ENDPOINT env var and
its comment block from both deployment manifests.
Adds ~600 new test cases (2013 → 2613 tests; 170 properties; 0
failures) across 12 new test files plus expansions of eight existing
files. Big lifts per module:
ContactLive.Mechanism 33 → 100%
MetricsPlug 54 → ~100%
Microwaveprop.Release 15.8 → 70%+
Telemetry 38 → 76%
HrrrNativeGridWorker 27.7 → 76%
ContactLive.Show 34 → 48% (handler + render branches)
Admin.ContactEditLive 49.7 → 70%+
GefsFetchWorker 35.8 → ~55%
IonosphereFetchWorker 56.3 → 75%
PathLive 58.9 → 67%
WeatherMapLive 66.9 → 80.2%
RoverLive 0 → 70.3%
ContactMapLive 59.2 → 89.8%
ContactMapController 0 → 100%
Mix tasks (Rust.Golden, Notebook, Backtest, HrrrBackfill,
HrrrClimatology, HrrrNativeBackfill, NexradBackfill,
RadarBackfill, ImportContestLogs, PropagationGrid,
ResetEnrichment, Hrrr.PurgeGridPoints) 0 → ~60-100%
Two incidental fixes made while adding tests:
- ContactLive.Show.handle_event("toggle_flag", ...) was passing
socket.assigns.current_scope to admin?/1 instead of socket.assigns,
so admins never matched the pattern and every toggle ran the
"Admins only." flash branch. Flag now toggles for admins again.
- Commercial.PollWorker.fetch_weather/1 promoted from private to
@doc'd public so the IEM-fetch + ASOS-upsert path can be tested
directly without driving perform/1 through live SNMP (which was
timing out for ~50 s per test in the earlier attempt).
Stable property-test additions cover the dewpoint-from-RH monotonicity,
nearest_run/1 idempotence, and the ionosphere station envelope.
New /eme page — antenna aiming + link budget for moonbounce.
Given a station (callsign / grid / lat,lon), TX power, antenna gain,
detection bandwidth, and system Tsys, the page shows in real time:
- Moon azimuth / elevation from the observer (ticks every 10 s)
- Slant range to the Moon
- EIRP breakdown: TX(dBm) + gain(dBi) = EIRP(dBm)
- Path-loss decomposition (Earth → Moon → Earth):
free-space spreading (×2)
− moon reflection gain 10·log(4π σ / λ²) where σ = ρ·π·R²
= round-trip path loss
- Band-dependent lunar albedo ρ (VHF ≈ 0.065, microwave ≈ 0.08,
mm-wave ≈ 0.02) surfaced as a named line item with the current
ρ shown to three decimals
- Received power, thermal noise floor, and SNR with a badge that
classifies the margin (strong / marginal / below noise)
- Self-echo Doppler shift with its radial-velocity driver
New modules:
- `Microwaveprop.Moon` — low-precision lunar ephemeris
(Astronomical Almanac Sec. D.4 truncated series). Provides
`julian_day/1`, `geocentric/1`, `observer_position/3`, and
`radial_velocity_km_s/3`. Accurate to ~0.3° position / ~200 km
distance — well inside any amateur EME-dish beamwidth.
- `Microwaveprop.Propagation.Eme` — path-loss helpers:
`moon_albedo/1` (band lookup), `moon_rcs_m2/1`,
`fspl_round_trip_db/2`, `moon_reflection_gain_db/2`,
`path_loss_db/3`, `received_power_dbm/1`, `noise_floor_dbm/2`,
`snr_db/1`, `doppler_shift_hz/2`.
Wired into the main nav ("EME" button) and the router between
/path and /rover.
Tests: 3 properties + 41 unit tests cover JD conversion, moon
position envelope, band-dependent albedo lookup, decomposition
identity, linearity/monotonicity of path loss and Doppler, and a
LiveView integration test against the rendered DOM.
Full suite: 2,460 tests + 170 properties; credo strict clean.
Root cause of 14 stuck hrrr-queued contacts: hrrr_point_rs's CONUS-only
grid silently writes zero profiles for out-of-bounds points (UK,
Winnipeg, Alberta, >50° lat), but the fetch task still completes
successfully. ContactWeatherEnqueueWorker's hrrr_placeholder_jobs
always emitted a queued placeholder for any contact with a pos1,
which meant mark_hrrr_status! re-flagged :queued every backfill tick
forever.
Introduce Propagation.Grid.contains?/1 as the canonical in-CONUS
check (inclusive on all four edges, nil-safe). Use it in two places
of ContactWeatherEnqueueWorker:
- hrrr_placeholder_jobs/1 now returns [] for OCONUS contacts, same
as pre-2014 / already-complete ones.
- mark_hrrr_status!/3 empty-list clause now branches on Grid.contains?
first (→ :unavailable) before falling through to the existing
NARR-coverage / :complete logic.
Next backfill tick will flip the 14 stuck OCONUS contacts in prod to
:unavailable automatically. The remaining 13 "CONUS" stuck ones are
right at the 50° edge (lat=50.1875 rounds outside the box) — same
reconciliation path.
Coverage additions:
- Grid.contains?/1 — corner-inclusive, OCONUS positive/negative
fixtures based on the actual stuck-contact pos1 values, nil/missing-
key/nil-value handling.
- enqueue_for_contact regression tests: pos1 in the UK and pos1 just
north of the lat cap both land at :unavailable, not :queued.
Suite: 2,416 tests + 159 properties (was 2,409 + 159); credo strict
clean.
Changes under lib/ (source):
- iemre_fetch_worker: replace `if transient_failure?/1` boolean predicate
+ `if/else` with a classifier that pattern-matches the error reason
directly, returning `:transient | :permanent`, and dispatch via
`case`. Three-clause head is clearer than the boolean predicate.
- scores_file.extract_points and nexrad_client.extract_box: force `//1`
step on the row/col comprehensions so an image box whose centre
lands outside the raster collapses to an empty iteration instead
of firing Elixir 1.19's ambiguous-range warning. Adds a regression
test for the right-edge box.
Test coverage added:
- Feature wrappers in Backtest.Features (theta_e_jump, shear_at_top,
duct_thickness, best_duct_freq, duct_usable_for_band,
distance_to_front / parallel_to_front stubs, all_features/0).
- NotifyListener handle_info tolerance for malformed payloads +
unknown messages, plus the PubSub broadcast side effect.
- Viewshed.effective_reach_km across every verdict / diffraction
tier / ducting-score tier combination.
- SnmpClient parse_af60_output unit conversions + unknown-column
handling; parse_snmpget_output OID normalisation and junk-line
tolerance.
- HrrrNativeClient native_level_count, native_variables,
native_messages shape, duct_messages / duct_byte_ranges.
- Changeset coverage for GeomagneticObservation, SolarFluxObservation,
SolarXrayObservation, Ionosphere.Observation, HrrrClimatology, and
Metar5minObservation — lifted each from ~50% / 0% to 100%.
- Property tests: GefsClient.dewpoint_from_rh invariants (Td ≤ T,
monotonic in RH, finite across the operating envelope) + idempotent
nearest_run. NexradClient.pixel_to_dbz monotonicity + bounded
output; latlon_to_pixel round-trip for every grid cell.
Also cleared several unrelated compiler/linter warnings:
- Three private test helpers (create_contact, insert_contact,
current_hour) had `\\ %{}` / `\\ 0` defaults that no caller used.
- profiles_file_test bound `dir` in a pattern match without using it.
Suite: 2,359 tests + 146 properties pass (was 2,300 / 139); coverage
70.38% → 70.89%; credo strict clean.
Rust workers (prop-grid-rs) that die mid-claim (SIGKILL, OOM, node
drain) leave grid_tasks rows stuck in status='running' forever,
because claim_next uses FOR UPDATE SKIP LOCKED and nothing resets the
orphan. The /status page then shows a permanent spinner with stale
f00/f10/f18 badges — currently 21 rows claimed as far back as
2026-04-20.
GridTaskEnqueuer.reclaim_stale_running/1 flips rows whose claimed_at
is older than 15 minutes back to 'queued'. Rows that have already
burned through 5 claim/reclaim cycles become 'failed' with a
reclaim-orphan error so the next hourly seed can replace them.
Wired into PropagationGridWorker.seed_chain/0 so it runs every :05
cron tick before new rows are seeded.
Also rename the status panel "Retrying" column to "Failed" — it was
always showing terminal `failed` rows, never retrying ones.
IemreFetchWorker now writes an empty stub observation when IEM returns
a permanent-failure status (404, 422), mirroring the existing {:ok, []}
path. The backfill cron's next tick sees the stub via
has_iemre_observation?/3, generates no job, and mark_status!/3 flips
the contact's iemre_status from :queued to :complete — draining the
51 contacts that have been stuck behind cancelled out-of-grid IEMRE
jobs.
Raise hot pod CPU limit 2 → 3 so BEAM gets 3 schedulers online.
Observed run queues of [2, 10, 0, 0, 0, 0] on the 2-scheduler
configuration during the :05 propagation chain, starving /live and
/health probe handlers and tripping intermittent liveness timeouts.
ADIF records frequently include operator commentary in the
<NOTES> field (multi-line, detailed) or <COMMENT> (short
one-liner). Now that contacts have a notes column, pass that
value through the import pipeline instead of dropping it.
Preference is NOTES over COMMENT to match the ADIF 3.1.4 spec's
intended split — NOTES is the richer field. Whitespace-only
values collapse to nil so a program that always emits a blank
NOTES tag doesn't fill every row with an empty string.
Tests cover NOTES capture, COMMENT fallback, NOTES-over-COMMENT
preference, and the absent-field nil case. ADIF upload blurb
lists the new carry-over so users see it before confirming.
