Three additions that bring the diagram closer to the SPC sounding
viewer's full-information chart:
* Wet-bulb temperature trace, plotted as a thinner blue line under
the dewpoint trace. Comes from a Stipanuk-style bracket between
the level's dewpoint and dry-bulb temperatures (already used by
`SkewtParams` for the wet-bulb-zero level — now public so the
renderer can call it per level).
* Environment virtual-temperature trace and parcel virtual-T
trajectory, both as thin pinkish dashed lines. The parcel
virtual-T is the actual buoyancy curve CAPE integrates against,
so showing it next to the dry-bulb parcel ascent makes the
moisture correction visible. New `SkewtParams.virtual_temp_c/2`
and `/3` helpers, plus `SkewtParams.mixing_ratio/2`.
* Critical-level rail on the right edge. Each of LCL / LFC / EL /
0 °C / WBZ that resolves on the current profile gets a short
horizontal tick at the right of the plot and a `<label> <p> mb`
tag just outside it — matches the SPC chart's `143 mb (mix)` /
`158 mb (mix)` annotation column. SkewtLive computes the
pressures (and converts the height-keyed 0 °C and WBZ from
SkewtParams back to pressure via `pressure_at_height/2`) before
handing them to `SkewtSvg.render/2`.
Wind-derived overlays (hodograph, wind barbs) remain off — HRRR
profile data is wind-free above 10 m AGL.
mix test: 27 / 27 across the four /skewt-related test files,
2,902 / 2,902 + 221 properties on the full suite. mix credo --strict
clean (one cyclomatic-complexity refactor extracted along the way).
When the on-disk ProfilesFile is empty SkewtLive falls back to the
hrrr_profiles Postgres table, but per-QSO HrrrFetchWorker rows
sometimes land with a partial profile (4–12 levels) when individual
byte-range fetches fail. The fallback was happily picking those rows,
which rendered a stubby Skew-T missing the upper troposphere.
HrrrProfileLookup.{list_valid_times_near, read_point_near}/3 now
filter on `coalesce(cardinality(profile), 0) >= min_profile_levels`,
defaulting to 13 (HRRR's standard pressure-level set: 1000/925/850/
700/500/400/300/250/200/150/100 mb plus the two boundary surfaces).
Pass `min_profile_levels: 0` to keep the partial rows for diagnostic
queries.
Confirmed against the dev mirror: the recent table has ~3.1 M rows
at 13 levels and ~250 k at 4–12 levels; the new default skips the
partial slice cleanly.
Tests: 8/8 HrrrProfileLookupTest cases (added two new ones for the
threshold). mix credo --strict clean.
Two changes that bring /skewt visually and analytically into parity
with the NOAA SPC sounding viewer
(https://www.spc.noaa.gov/exper/soundings/):
NOAA-style line set
-------------------
Updated the SkewtSvg style block to match the public SPC chart:
* Temperature trace ........ pure red (#ff0000) at 2.5 px stroke
* Dewpoint trace ........... pure green (#00aa00) at 2.5 px stroke
* Isotherms ................ dashed teal (#14b8a6)
* Dry adiabats ............. solid orange (#f59e0b)
* Moist (saturated) adiabats dashed sky-blue (#38bdf8)
* Mixing-ratio lines ....... dotted darker green (#16a34a), with
labels at 700 mb showing the g/kg
value, drawn only below 600 mb (matches
the SPC chart's lower-band number row)
* Parcel ascent ............ new dashed grey (#475569) line, drawn
whenever SkewtParams.derive returns a
non-empty parcel_trace
The mixing-ratio set is the SPC standard (0.4 / 0.7 / 1 / 2 / 3 / 5 /
8 / 12 / 16 / 20 g/kg) instead of the prior arbitrary set.
SkewtParams: SPC parcel-theory + indices module
-----------------------------------------------
New `Microwaveprop.Weather.SkewtParams` derives the parameter set that
ships in the SPC `*.txt` companion file:
* LCL pressure / temperature / height (Bolton 1980 eq 22)
* LFC pressure / height
* EL pressure / height
* SBCAPE, SBCIN (J/kg, virtual-temperature integration with
surface parcel)
* 3 km CAPE
* Lifted Index (parcel ↔ environment T at 500 mb)
* Freezing level (T = 0 °C height) and Wet-Bulb Zero
* DCAPE (downdraft CAPE — picks the min-θₑ source level in
400–700 mb, descends moist-adiabatically to surface)
* Layer lapse rates: sfc–3 km, 700–500 mb, 850–500 mb
* `parcel_trace` — pressure/temperature trajectory used by
SkewtSvg to draw the dashed parcel-ascent overlay
Wind-derived indices (SRH, BWD, Bunkers motion, SCP, STP, SHIP) are
deliberately *not* produced and the LiveView surfaces a one-line note
explaining why: HRRR persists wind only at 10 m AGL, so per-pressure-
level shear can't be computed from this data source.
LiveView panel restructured into grouped sections (Surface, Convective
parcel, Levels, Lapse rates, Moisture/stability, Refractivity) so the
layout matches the SPC viewer's order. The Levels rows show pressure
*and* height ("872 mb · 1,140 m") for quick cross-reference between
the diagram axis and the readout.
Tests: 8/8 new SkewtParamsTest cases (LCL Bolton check, saturated-air
edge case, full field-set presence, summer profile produces SBCAPE > 0
and LI < 0, capped profile produces SBCAPE = 0 and LI > 0, freezing
level interpolated correctly, sfc–3 km lapse rate, empty profile no
crash). 25/25 across all skewt-related test files. mix test green
(2,902 / 2,902 + 221 properties), mix credo --strict clean, mix
assets.build clean.
Two follow-ups to the earlier /skewt work:
1. Interactive crosshair (matches the SPC-viewer feel from the
reference screenshot). Mousing over the diagram now drops a
horizontal pressure cursor with a top-left readout box showing the
pressure at the cursor's altitude (mb), the height in m and ft,
and the linearly interpolated temperature and dewpoint at that
level. Two coloured dots track the T and Td traces along the
cursor so the user can read the spread visually.
Mechanics: SkewtSvg.geometry/0 exposes the plot geometry (left,
right, top, bottom, p_bottom, p_top, t_min, t_max, skew) as JSON;
SkewtSvg emits an initially-hidden <g class="skewt-cursor"> layer.
A new Skewt hook in assets/js/app.ts inverts pressure_y to recover
pressure from the cursor's viewBox-Y, walks the (descending-pres-
sorted) profile to interp T/Td/height, and updates the elements.
The hook also handles mouseleave to hide the cursor and uses the
SVG's screen-CTM so the math stays correct under any container
aspect-ratio resize.
2. DB fallback for `available_valid_times`/`load_profile`. When the
on-disk `ProfilesFile` has nothing for the location-and-window
(true on dev, transient on prod between chain runs), SkewtLive
now falls back to `HrrrProfileLookup.{list_valid_times_near,
read_point_near}/3` against the `hrrr_profiles` Postgres table.
The lookup uses the same ±0.07° spatial tolerance and the
`(lat, lon, valid_time)` unique index that `Weather.find_nearest_
hrrr/3` already relies on, so the queries are index-only.
Returned shape matches `ProfilesFile.read_point/3` so SkewtLive
reads from either source transparently. NaiveDateTime → UTC
normalisation is centralised in `ensure_utc/1` so the LiveView
sees `%DateTime{time_zone: "Etc/UTC"}` regardless of which side
produced the value.
Tests: 6/6 new HrrrProfileLookupTest cases (list/limit/tolerance,
read nearest cell, nil for far cells, picks closest among matches),
3/3 SkewtLiveTest still green, 2,894 total Elixir tests + 221
properties green via mix test (one pre-existing flaky test in
admin/contact_edit_live re-runs clean). mix credo --strict clean.
mix assets.build clean.
Both retired in light of the 2026-04-25 revision report
(`docs/algo-reports/2026-04-25-algo-revisions.md`):
* HPBL boundary-layer multiplier in `Scorer.score_refractivity/{3,4,5}`
is gone. On the n=47,418 10 GHz HRRR-matched corpus rho_hpbl is
+0.004; binned distance is flat to within ±5 % across 200–2,000 m.
The previously reported 2.3× ratio between shallow and deep HPBL
bins was a small-corpus artefact that disappeared once the matched
corpus passed ~5,000 contacts.
* Native-profile 1.15× duct boost is gone. At 10 GHz on n=52,341
matched contacts, cells where `best_duct_band_ghz` ≥ band ran
*shorter* than no-duct cells (198 km vs 211 km, n=173 vs 44,658).
The Bulk Richardson gate that existed only to suppress that boost
in turbulent conditions is also retired; arity preserved on every
`score_refractivity` overload so callers compile unchanged.
Same retirements applied in the Rust port (`rust/prop_grid_rs/src/scorer.rs`)
to keep the Elixir/Rust parity tests green.
algo.md updated end-to-end:
* Finding 4 (HPBL) rewritten to "no usable signal" with the
n=47,418 bin table.
* Finding 5 (gradient) tightened: load-bearing only at 24 GHz,
set band-specific gradient weight to 0 outside [10, 47] GHz on
the next derive_band_weights run.
* Finding 8 (time-of-day) augmented with the robust hour-bucketed
amplitude index (40 / 82 / 101 % at 10 / 24 / 47 GHz) and a
selection-bias caveat for VHF/UHF (where 129–148 % amplitude
reflects evening contest scheduling, not propagation).
* Finding 9 (sounding) refreshed against the n=27,058 corpus.
* Part 2c "Native-profile duct boost" rewritten as
"retired 2026-04-25" with the falsifying join.
* Part 2c "Commercial-link inverse sensor" promoted from deferred
with the 22-day, 38k-sample SNMP corpus, the +0.61 / −0.61 PWAT
and pressure correlations against the 37-hour HRRR overlap, and
the 04–05 / 10–12 local two-peak morning fade window.
* Part 2b "Pressure" annotated with the U-shape at 10 GHz and the
next-iteration target.
* Part 2d "What stayed, what left" + "Open items" updated to
reflect the four code/doc moves.
119 Elixir scorer tests green, 134 Rust scorer tests green, 2,888
total Elixir tests + 221 properties green via `mix precommit`.
Each fix is covered by a regression test that fails on `main` and
passes on this commit.
Round 1 (initial review):
* propagation: thread `latitude` into the conditions map so
`score_season/4` actually picks up regional multipliers
* hrrr_client / fetcher.rs: `nearest_hrrr_hour` rounds DOWN, never at
a future cycle that NOAA hasn't published yet
* radio: spherical-vector great-circle midpoint replaces the
arithmetic mean — anti-meridian paths no longer fold to Greenwich
* weather: `reconcile_weather_statuses` scales the longitude band by
`1 / cos(lat)` so the bbox stays ~150 km wide at every latitude
* radio/maidenhead: clamp 90°/180° below the field-bucket overflow so
`from_latlon` never emits invalid characters like 'S'
* prop_grid_rs/pipeline: merge HRRR + NEXRAD-derived rain rates and
read `best_duct_freq_ghz` into `best_duct_band_ghz` so the Native
Duct Boost actually fires
* propagation/region (Elixir + Rust): inclusive upper bounds so points
exactly at lat_max get the regional multiplier
* weather/sounding_params (Elixir + Rust): drop the 10 m gradient
floor so HRRR's thin near-surface layers stop hiding sharp ducts
* weather/sounding_params: when the profile ends inside a duct,
finalize it with the highest sample as the top instead of throwing
it away (Rust port already correct)
Round 2 (post-fix sweep):
* radio + commercial: single canonical haversine in Radio (atan2
form); Commercial delegates instead of carrying a second copy that
could disagree at threshold distances
* prop_grid_rs/profiles_file: `snap_coords` matches Elixir's
step-aware snap (`round(coord/0.125) * 0.125`, then 3-dp round) so
Rust-keyed and Elixir-keyed profile maps land on the same cell
* weather/grib2/wgrib2: `parse_lon_val_segment` uses `Float.parse`
uniformly — wgrib2 dropping the trailing `.0` from a longitude no
longer crashes the whole chain step
`apply_edit_to_contact/2` is the path for both owner direct-edits and
admin reviewed-edits. The public contact-map + total-count + gzipped
controller payload caches are built from `private == false` rows, but
only the insert path was clearing them — every other update left
/api/contacts/map and /contacts/map serving the previous snapshot for
up to 10 minutes. A `private: false -> true` flip on a contact already
present in the cached payload was a temporary privacy leak; grid
corrections, callsign changes, and flagged_invalid flips silently
diverged from the live row.
Extracts the three `Cache.invalidate` calls into
`invalidate_contact_map_caches/0` and runs it from both
`apply_edit_to_contact/2` and the existing insert path so the public
view stays in sync.
The /contacts/:id edit form always submits the `private` checkbox, but
`current_value/2` had no `"private"` clause and fell through to the
catch-all `nil`. The Contact schema defaults `private: false`, so an
unchanged box compared `false != nil` and was always tagged as a
change — for non-owners that meant a spurious pending review queued
on every save, and for owners/admins it triggered a no-op direct
update.
Adds the missing clause so the diff sees boolean-vs-boolean and
correctly returns `:no_changes` when the box is untouched.
The /contacts/:id edit form pre-fills qso_timestamp via
`Calendar.strftime(ts, "%Y-%m-%d %H:%M")` and submits whatever the
user typed back unchanged. Two failure modes resulted:
1. `diff_against_contact/2` compared that string against the stored
`%DateTime{}`, never matched, and treated every untouched submit
as a modification.
2. `build_contact_changes/2` then ran `DateTime.from_iso8601/1` on
the space-separated, no-seconds, no-Z form and crashed with
`MatchError` on `{:error, :invalid_format}` — direct owner/admin
submits raised instead of saving.
Adds `normalize_timestamp_field/2` to `normalize_proposed/1` that
parses both the form's `"YYYY-MM-DD HH:MM[:SS]"` and the strict
`"YYYY-MM-DDTHH:MM:SSZ"` shapes into a UTC `%DateTime{}` (with
`NaiveDateTime.from_iso8601` doing the heavy lifting after a small
seconds-padding pass). Empty strings drop the field; unparseable
input drops it too so a typo becomes "no changes" rather than a
500.
PropagationGridWorker hard-coded `now-2h` for run_time selection on
the conservative theory that NOAA always finishes publishing within
two hours. In practice f18 is on the mirror by HH:55–HH:00 of the
next hour, so the cron at HH:05 was always picking a cycle that was
already an hour stale.
`HrrrClient.cycle_available?/1` HEADs the wrfprsf18 idx (the slowest
file in the cycle) — a 200 means every earlier forecast hour is also
on disk. `pick_run_time/1` probes `now-1h` first; if the probe
returns true we use that cycle, otherwise we fall back to the
existing `now-2h` slot. Test hook via :hrrr_cycle_available_fn env
keeps the behaviour fully exercisable without hitting NOAA.
End-to-end: weather/score data lag drops from ~2 h to ~1 h on every
hour where NOAA delivers on time. The fallback path keeps the chain
running on slow-publish hours.
850 mb T alone shows that warm air is present aloft but doesn't reveal
whether it's the textbook cap structure — a warm 700 mb above an EML
plume. The three new layers expose the canonical capping diagnostics
visually:
* T @ 700 mb — ≥10 °C is the southern Plains "moderate cap" threshold
* Td @ 700 mb — wide T-Td depression at 700 mb signals the EML plume
* Lapse 850→700 — steep mid-layer lapse rate (≥7 °C/km) over a moist
boundary layer is the cap mechanism itself
All three derive purely from the existing pressure-level profile (Rust
already fetches up through 700 mb), so no Rust pipeline changes needed.
ScoreCache held 437 entries × ~1.94 MiB each (~850 MiB per pod) of
{band_mhz, valid_time} grids — data already on NFS as compact .prop
files. NotifyListener and ScoreCacheReconciler both eagerly materialised
every band into ETS on each Rust completion / 60s sweep, so 5 hot
replicas wasted ~4.2 GiB of redundant cache and OOMed at 6 GiB.
Bound the cache to 32 entries with eviction by oldest valid_time, drop
the eager warm loops, and let LiveView callers lazy-fill via the
existing ScoresFile read path. ScoreCacheReconciler had no remaining
purpose and is deleted; runbook + prom_ex counters updated to match.
Steady-state cache footprint ~60 MiB per pod instead of ~850 MiB.
pg_stat_user_tables tracks two timestamps — `last_autoanalyze`
is only bumped by the autovacuum daemon, while `last_analyze` is
bumped by manual ANALYZE statements (including the ones our Task
issues). The previous query only filtered on `last_autoanalyze`,
so on a re-run within the skip_recent window every manually-
analyzed table came back for another ANALYZE pass.
Uses `GREATEST(last_analyze, last_autoanalyze)` so fresh manual
analyses count as recent.
Running ANALYZE serially against the 13M-row HRRR monthly partitions
took longer than the kubectl exec idle timeout (~300s per partition
on the Turing Pi 2), so the one-shot RPC kept getting killed before
the big partitions ran. Moves the loop into a supervised Task on
the caller's node so the RPC returns immediately; the BEAM keeps
running ANALYZE regardless of kubectl connection state.
Also sorts tables ascending by n_live_tup so small tables finish
first and their fresh stats are visible to the planner right away.
Progress is logged per table via `Logger.info`.
Two issues spotted in pg_stat_user_tables:
1. Most `hrrr_profiles` partitions have `last_autoanalyze IS NULL`.
The partitions are append-only from the hourly grid worker, so
autovacuum never crosses the analyze threshold — but without
stats the planner falls back to estimates and picks bad joins
on per-QSO lookups.
2. `contacts`, `grid_tasks`, `hrrr_fetch_tasks`, and
`contact_common_volume_radar` see high UPDATE volumes
(enrichment status flips, queue claim/complete cycles, mechanism
classifier writes) but only accrue ~5% dead before the default
20% / 10% autovacuum triggers fire. BackfillEnqueueWorker's
status-priority ORDER BY + the Rust workers' FOR UPDATE SKIP
LOCKED claim scan both want fresher stats than that.
Changes:
- Migration `20260424204656_tune_autovacuum_for_high_churn` sets
per-table `autovacuum_*_scale_factor` + `_threshold = 50` on the
four tables above. Aggressive enough to keep stats current;
tame enough that vacuum doesn't thrash the Turing Pi 2 Postgres
node. Reversible.
- `Weather.analyze_all/1` walks `pg_stat_user_tables`, runs
`ANALYZE` on every table not auto-analyzed in the last 6 h,
reports counts. Meant to be invoked once via
`bin/microwaveprop rpc 'Microwaveprop.Weather.analyze_all()'`
after deploy to seed statistics on the cold partitions;
idempotent on re-run.
Root cause of the "81,980 / 81,994" display that wouldn't advance:
14 contacts had pos1 inside the HRRR CONUS grid but pos2 or the
great-circle midpoint landed outside it (50°N+ into Canada, mid-
Atlantic, Pacific, Caribbean, Alaska). The Rust hrrr-point-worker
silently returns no profile for OCONUS points, so
`hrrr_data_fully_present?/1` never evaluated true and the contact
sat `:queued` forever.
The existing ContactWeatherEnqueueWorker.mark_hrrr_status!/3 already
flips to `:unavailable` on empty job lists via NarrClient.in_coverage?
+ Grid.contains?(pos1), but those checks only considered pos1 — not
the full contact_path_points list. When pos1 was in-grid but a
downstream point was not, the contact entered `:queued` and the
reconciler couldn't rescue it.
`reconcile_hrrr_statuses/0` now sweeps `:queued` contacts into two
buckets:
- ANY path point OCONUS → `:unavailable` (matches what the enqueuer
would emit had it known about the downstream points);
- else, all points present → `:complete`.
Applied once in prod to 14 stuck contacts; cron path picks up any
new occurrences going forward.
The Rust hrrr_points worker now persists surface_refractivity /
min_refractivity_gradient / ducting_detected at upsert time (previous
commit), but the ~80k rows written before that fix have the scalar
columns NULL even though the pressure-level `profile` JSONB is
populated. Re-fetching them would re-pull tens of GB of HRRR grids
we already parsed.
Weather.backfill_hrrr_scalars/1 streams rows with
`surface_refractivity IS NULL`, runs the existing
SoundingParams.derive/1 against the stored profile, and writes the
scalars back — no network, no GRIB decode. Batched (500 rows per
tick, 1000 tick cap) to keep transactions short on the Turing Pi 2
PG node.
Meant to be run once via `bin/microwaveprop rpc` after deploy.
Idempotent; the WHERE clause excludes already-backfilled rows.
Found by auditing the ProfilesFile, hrrr_profiles, and weather-layer
contracts against what the Rust writers actually emit.
Bug 1 (HIGH) — Rust `hrrr_points` worker upserted hrrr_profiles rows
with NULL `surface_refractivity`, `min_refractivity_gradient`, and
`ducting_detected`. The Elixir HrrrClient path always wrote them
via SoundingParams.derive. Per-QSO pages that rely on these
scalars ("N:", "dN/dh:", duct badge) showed "—" for any contact
enriched through Stream C. Fixed by porting
`surface_refractivity` + `ducting_detected` into Rust's
`sounding_params.rs` and deriving at upsert time.
Bug 2 (HIGH) — Rust f01..f18 `cell_to_profile_entry` never emitted
wind_u / wind_v / cloud_cover_pct / precip_mm into ProfilesFile
cells. `Propagation.factors_for` recomputes factor scores from the
profile cell on /map click, and these four fields are read
(wind_speed_kts, sky_cover_pct, precip_to_rate_mmhr). Result: every
f01..f18 point-detail popup showed 0 for wind/sky/rain factors.
Fixed by extracting the grib values at cell_to_profile_entry time
and whitelisting the keys for atomization in
profiles_file.ex `@mp_atom_keys`.
Bug 4 (MEDIUM) — `WeatherLayers.duct_field/2` only knew the
`SoundingParams.detect_ducts` shape (`d["base"]` / `d["strength"]`).
The Rust/HrrrNativeClient shape uses `:base_m` / `:thickness_m`
atom keys (post-ProfilesFile.read atomization). On /weather, the
Duct Base / Duct Strength layers returned nil for every Rust-origin
cell. Tolerant lookup now reads either shape; uses `thickness_m`
as the "strength" proxy for the native shape (thicker duct traps
a wider band).
Skipped (verified but lower impact):
- Rust ProfilesFile aggregates ducts as scalars, not per-layer — the
`ducts` array in the popup is always empty. Requires larger
DuctMetrics refactor in Rust; deferred.
- `complete_hrrr_task` emits no NOTIFY (vs `complete` which does).
Elixir has no matching listener yet, so adding NOTIFY alone is a
no-op; the existing cron reconciler catches it.
- `:best_duct_band_ghz` fallback in propagation.ex is dead code;
cleanup, not a bug.
133 Rust tests + 2842 Elixir tests + credo green.
The Rust f01..f18 pipeline writes ProfilesFile cells, and the Rust
hrrr_points worker writes hrrr_profiles rows, with profile entries
keyed "pres_mb" / "hght_m" / "tmpc" / "dwpc" (string keys pre-
atomization; atom keys post-ProfilesFile.read). Elixir's
SoundingParams.derive and WeatherLayers.sort_profile filtered on
"pres" / "hght" — so every derived field returned nil for any
Rust-provided profile.
Visible symptoms on /weather: N-gradient, Refractivity, T @ 850mb,
Td @ 850mb, Lapse Rate, Inversion, Inv. Base all rendered no overlay.
For per-contact analysis, min_refractivity_gradient from the Rust
HRRR point worker's rows silently dropped.
- SoundingParams.normalize_profile_entry/1: single-source normalizer
accepting legacy, Rust-string, and Rust-atom shapes; returns the
canonical "pres"/"hght"/"tmpc"/"dwpc"/"drct"/"sknt" shape.
- SoundingParams.derive/1 and WeatherLayers.sort_profile/1 both run
every entry through it before the rest of the existing logic.
- Weather.build_grid_cache_row/4 now falls back to native_min_gradient
when SoundingParams can't derive one (surface-only profiles).
Also:
- Default /weather overlay changed to Temperature (LiveView assign +
JS fallback in the hook's dataset default) per user request.
Extends the weather-status reconciler with symmetric sweeps for HRRR
and IEMRE. Same failure mode: the worker writes data to the shared
table but has no back-pointer to the contact that triggered the
fetch, so contact-level status never flips unless a user views the
page.
- Weather.reconcile_iemre_statuses/0: iterates :queued contacts;
flips :complete when every path point has an iemre_observations
row at the rounded 0.125° grid cell for the qso_timestamp's date.
- Weather.reconcile_hrrr_statuses/0: same shape; flips when
hrrr_data_fully_present?/1 holds.
- Both run alongside the existing weather sweep at the tail of
ContactWeatherEnqueueWorker.perform/1.
Kept in Elixir (not SQL) because contact_path_points/1 emits 1-3
points per contact and grid-rounding each in pure SQL is awkward.
The stuck-at-steady-state count is always tiny (<20), so the
iteration is cheap.
2,019 contacts sat permanently in weather_status=:queued even after
their ASOS data had been ingested. The only code path that flipped
:queued → :complete was MicrowavepropWeb.ContactLive.Show on page
view — contacts that nobody visited stayed :queued forever.
Adds Weather.reconcile_weather_statuses/0, a single SQL UPDATE that
flips every :queued contact whose ±2h / 150km window now contains
at least one surface observation. Called at the tail of the
existing ContactWeatherEnqueueWorker cron so the correction runs
regardless of whether anyone opens the UI.
Visible effect: the status page's "Weather done" count (previously
capped at 79,975 while data kept landing) will converge on the real
total after the next cron tick.
2838 tests + credo green.
Replaces per-QSO-endpoint asos_batch granularity with per-station-day
granularity. Each contact in the same UTC day near the same station
now produces identical unique-args and collapses to ONE Oban job, so
the backlog shrinks as cross-contact duplicates dedup.
Each fetch also covers a full 24h window (vs the previous 4h) so one
IEM request returns ~24 hourly observations instead of ~1-2. The
effective bytes-per-request is 10x higher, amortizing the
IemRateLimiter gap + 429 retry tail across far more data.
Changes:
- Weather.station_day_covered?/2 + station_day_pairs_covered/1 — cheap
UTC-day coverage checks for the enqueuer + worker skip paths.
- WeatherFetchWorker gains an "asos_day" clause that fetches the full
UTC day for one station and upserts. Skips if the day is already
covered in the DB.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one asos_day
job per (station_id, UTC date) pair. ±2h windows crossing midnight
enumerate both dates.
- Microwaveprop.Weather.RebatchAsos.to_day_jobs/1 migrates any
already-queued asos/asos_batch jobs into the new shape. Idempotent;
supports dry_run.
- asos_batch worker clause retained for jobs already in flight at
deploy time.
2835 tests + credo green.
Mix.Task.run isn't available in a compiled release, so the in-prod
`bin/microwaveprop eval 'Mix.Tasks.Weather.RebatchAsos.run(...)'`
dispatch crashes. Moves the real logic into a plain module
(`Microwaveprop.Weather.RebatchAsos`) that both the Mix task and a
release eval can call. Uses IO.puts over Mix.shell for the same
reason.
- IemRateLimiter gains AIMD-style adaptive spacing. signal_429/0
widens the current gap (*= 1.5, capped at max_interval_ms default
10s); signal_success/0 narrows it back toward the configured base
(*= 0.92, floored at interval_ms). Self-tunes to IEM's moving
ceiling without needing the manual "safe for 4 pods" constant.
- IemClient now routes every response through a central handle_response
helper that fires the widen/narrow feedback signals, eliminating the
four near-identical case blocks.
- Mix.Tasks.Weather.RebatchAsos collapses any already-queued single-
station "asos" jobs into the batched "asos_batch" shape the
enqueuer now emits, so the pending backfill queue converts to the
new per-request-efficient path instead of draining at the old rate.
Idempotent; supports --dry-run.
2833 tests + credo green.
IEM's ASOS CSV endpoint accepts multiple station= params in a single
request. Prior code enqueued one WeatherFetchWorker job per nearby
station per QSO endpoint — N jobs × 1 station each — which paid the
IemRateLimiter's 1500ms gap and the IEM 429-retry tail N times per
contact.
Changes:
- IemClient.asos_url/3 accepts a list of station codes.
- IemClient.fetch_asos_batch/3 fetches N stations in one call and
returns rows grouped by station_code (with absent codes filled as
empty lists so callers can stub them).
- parse_asos_csv/1 now exposes the station_code column it was
previously discarding.
- WeatherFetchWorker gains an "asos_batch" fetch_type clause that
unpacks rows per (station_id, station_code), upserting or stubbing
each. The single-station "asos" clause stays for already-queued
retryable jobs.
- ContactWeatherEnqueueWorker.build_asos_jobs/3 now emits one batch
job per (lat, lon, 4h window) covering every uncovered nearby
station (sorted for deterministic unique-args).
Expected effect on backfill: ~10-20x fewer IEM requests per contact
enrichment cycle, matching drop in 429 retry traffic.
Removes the six OTel packages (opentelemetry, _api, _exporter,
_phoenix, _oban, _bandit) and their 9 transitive deps. Tracing was
only wired for Phoenix/Bandit/Oban auto-instrumentation — PromEx
already covers the metrics we actually use on the status page +
Grafana. Also drops the OTEL_EXPORTER_OTLP_ENDPOINT env var and
its comment block from both deployment manifests.
Adds ~600 new test cases (2013 → 2613 tests; 170 properties; 0
failures) across 12 new test files plus expansions of eight existing
files. Big lifts per module:
ContactLive.Mechanism 33 → 100%
MetricsPlug 54 → ~100%
Microwaveprop.Release 15.8 → 70%+
Telemetry 38 → 76%
HrrrNativeGridWorker 27.7 → 76%
ContactLive.Show 34 → 48% (handler + render branches)
Admin.ContactEditLive 49.7 → 70%+
GefsFetchWorker 35.8 → ~55%
IonosphereFetchWorker 56.3 → 75%
PathLive 58.9 → 67%
WeatherMapLive 66.9 → 80.2%
RoverLive 0 → 70.3%
ContactMapLive 59.2 → 89.8%
ContactMapController 0 → 100%
Mix tasks (Rust.Golden, Notebook, Backtest, HrrrBackfill,
HrrrClimatology, HrrrNativeBackfill, NexradBackfill,
RadarBackfill, ImportContestLogs, PropagationGrid,
ResetEnrichment, Hrrr.PurgeGridPoints) 0 → ~60-100%
Two incidental fixes made while adding tests:
- ContactLive.Show.handle_event("toggle_flag", ...) was passing
socket.assigns.current_scope to admin?/1 instead of socket.assigns,
so admins never matched the pattern and every toggle ran the
"Admins only." flash branch. Flag now toggles for admins again.
- Commercial.PollWorker.fetch_weather/1 promoted from private to
@doc'd public so the IEM-fetch + ASOS-upsert path can be tested
directly without driving perform/1 through live SNMP (which was
timing out for ~50 s per test in the earlier attempt).
Stable property-test additions cover the dewpoint-from-RH monotonicity,
nearest_run/1 idempotence, and the ionosphere station envelope.
New /eme page — antenna aiming + link budget for moonbounce.
Given a station (callsign / grid / lat,lon), TX power, antenna gain,
detection bandwidth, and system Tsys, the page shows in real time:
- Moon azimuth / elevation from the observer (ticks every 10 s)
- Slant range to the Moon
- EIRP breakdown: TX(dBm) + gain(dBi) = EIRP(dBm)
- Path-loss decomposition (Earth → Moon → Earth):
free-space spreading (×2)
− moon reflection gain 10·log(4π σ / λ²) where σ = ρ·π·R²
= round-trip path loss
- Band-dependent lunar albedo ρ (VHF ≈ 0.065, microwave ≈ 0.08,
mm-wave ≈ 0.02) surfaced as a named line item with the current
ρ shown to three decimals
- Received power, thermal noise floor, and SNR with a badge that
classifies the margin (strong / marginal / below noise)
- Self-echo Doppler shift with its radial-velocity driver
New modules:
- `Microwaveprop.Moon` — low-precision lunar ephemeris
(Astronomical Almanac Sec. D.4 truncated series). Provides
`julian_day/1`, `geocentric/1`, `observer_position/3`, and
`radial_velocity_km_s/3`. Accurate to ~0.3° position / ~200 km
distance — well inside any amateur EME-dish beamwidth.
- `Microwaveprop.Propagation.Eme` — path-loss helpers:
`moon_albedo/1` (band lookup), `moon_rcs_m2/1`,
`fspl_round_trip_db/2`, `moon_reflection_gain_db/2`,
`path_loss_db/3`, `received_power_dbm/1`, `noise_floor_dbm/2`,
`snr_db/1`, `doppler_shift_hz/2`.
Wired into the main nav ("EME" button) and the router between
/path and /rover.
Tests: 3 properties + 41 unit tests cover JD conversion, moon
position envelope, band-dependent albedo lookup, decomposition
identity, linearity/monotonicity of path loss and Doppler, and a
LiveView integration test against the rendered DOM.
Full suite: 2,460 tests + 170 properties; credo strict clean.
Root cause of 14 stuck hrrr-queued contacts: hrrr_point_rs's CONUS-only
grid silently writes zero profiles for out-of-bounds points (UK,
Winnipeg, Alberta, >50° lat), but the fetch task still completes
successfully. ContactWeatherEnqueueWorker's hrrr_placeholder_jobs
always emitted a queued placeholder for any contact with a pos1,
which meant mark_hrrr_status! re-flagged :queued every backfill tick
forever.
Introduce Propagation.Grid.contains?/1 as the canonical in-CONUS
check (inclusive on all four edges, nil-safe). Use it in two places
of ContactWeatherEnqueueWorker:
- hrrr_placeholder_jobs/1 now returns [] for OCONUS contacts, same
as pre-2014 / already-complete ones.
- mark_hrrr_status!/3 empty-list clause now branches on Grid.contains?
first (→ :unavailable) before falling through to the existing
NARR-coverage / :complete logic.
Next backfill tick will flip the 14 stuck OCONUS contacts in prod to
:unavailable automatically. The remaining 13 "CONUS" stuck ones are
right at the 50° edge (lat=50.1875 rounds outside the box) — same
reconciliation path.
Coverage additions:
- Grid.contains?/1 — corner-inclusive, OCONUS positive/negative
fixtures based on the actual stuck-contact pos1 values, nil/missing-
key/nil-value handling.
- enqueue_for_contact regression tests: pos1 in the UK and pos1 just
north of the lat cap both land at :unavailable, not :queued.
Suite: 2,416 tests + 159 properties (was 2,409 + 159); credo strict
clean.
Changes under lib/ (source):
- iemre_fetch_worker: replace `if transient_failure?/1` boolean predicate
+ `if/else` with a classifier that pattern-matches the error reason
directly, returning `:transient | :permanent`, and dispatch via
`case`. Three-clause head is clearer than the boolean predicate.
- scores_file.extract_points and nexrad_client.extract_box: force `//1`
step on the row/col comprehensions so an image box whose centre
lands outside the raster collapses to an empty iteration instead
of firing Elixir 1.19's ambiguous-range warning. Adds a regression
test for the right-edge box.
Test coverage added:
- Feature wrappers in Backtest.Features (theta_e_jump, shear_at_top,
duct_thickness, best_duct_freq, duct_usable_for_band,
distance_to_front / parallel_to_front stubs, all_features/0).
- NotifyListener handle_info tolerance for malformed payloads +
unknown messages, plus the PubSub broadcast side effect.
- Viewshed.effective_reach_km across every verdict / diffraction
tier / ducting-score tier combination.
- SnmpClient parse_af60_output unit conversions + unknown-column
handling; parse_snmpget_output OID normalisation and junk-line
tolerance.
- HrrrNativeClient native_level_count, native_variables,
native_messages shape, duct_messages / duct_byte_ranges.
- Changeset coverage for GeomagneticObservation, SolarFluxObservation,
SolarXrayObservation, Ionosphere.Observation, HrrrClimatology, and
Metar5minObservation — lifted each from ~50% / 0% to 100%.
- Property tests: GefsClient.dewpoint_from_rh invariants (Td ≤ T,
monotonic in RH, finite across the operating envelope) + idempotent
nearest_run. NexradClient.pixel_to_dbz monotonicity + bounded
output; latlon_to_pixel round-trip for every grid cell.
Also cleared several unrelated compiler/linter warnings:
- Three private test helpers (create_contact, insert_contact,
current_hour) had `\\ %{}` / `\\ 0` defaults that no caller used.
- profiles_file_test bound `dir` in a pattern match without using it.
Suite: 2,359 tests + 146 properties pass (was 2,300 / 139); coverage
70.38% → 70.89%; credo strict clean.
Rust workers (prop-grid-rs) that die mid-claim (SIGKILL, OOM, node
drain) leave grid_tasks rows stuck in status='running' forever,
because claim_next uses FOR UPDATE SKIP LOCKED and nothing resets the
orphan. The /status page then shows a permanent spinner with stale
f00/f10/f18 badges — currently 21 rows claimed as far back as
2026-04-20.
GridTaskEnqueuer.reclaim_stale_running/1 flips rows whose claimed_at
is older than 15 minutes back to 'queued'. Rows that have already
burned through 5 claim/reclaim cycles become 'failed' with a
reclaim-orphan error so the next hourly seed can replace them.
Wired into PropagationGridWorker.seed_chain/0 so it runs every :05
cron tick before new rows are seeded.
Also rename the status panel "Retrying" column to "Failed" — it was
always showing terminal `failed` rows, never retrying ones.
IemreFetchWorker now writes an empty stub observation when IEM returns
a permanent-failure status (404, 422), mirroring the existing {:ok, []}
path. The backfill cron's next tick sees the stub via
has_iemre_observation?/3, generates no job, and mark_status!/3 flips
the contact's iemre_status from :queued to :complete — draining the
51 contacts that have been stuck behind cancelled out-of-grid IEMRE
jobs.
Raise hot pod CPU limit 2 → 3 so BEAM gets 3 schedulers online.
Observed run queues of [2, 10, 0, 0, 0, 0] on the 2-scheduler
configuration during the :05 propagation chain, starving /live and
/health probe handlers and tripping intermittent liveness timeouts.
ADIF records frequently include operator commentary in the
<NOTES> field (multi-line, detailed) or <COMMENT> (short
one-liner). Now that contacts have a notes column, pass that
value through the import pipeline instead of dropping it.
Preference is NOTES over COMMENT to match the ADIF 3.1.4 spec's
intended split — NOTES is the richer field. Whitespace-only
values collapse to nil so a program that always emits a blank
NOTES tag doesn't fill every row with an empty string.
Tests cover NOTES capture, COMMENT fallback, NOTES-over-COMMENT
preference, and the absent-field nil case. ADIF upload blurb
lists the new carry-over so users see it before confirming.
Operators want a place to jot observations that aren't captured by
the structured fields — weather anecdotes, propagation mode
commentary, equipment details, band conditions. Add a text column
on contacts plus the two ingest paths users submit through:
* /submit single-contact form gets a 3-row textarea under the
other fields. Optional, max 2000 chars, with a live length cap
via maxlength. Whitespace-only input collapses to NULL so the
column reflects "no notes" rather than an empty string.
* CSV importer recognises a `notes` (or `note`) column via the
existing header_aliases table and flows values straight through
submission_changeset. Sample template gets a matching example
row with embedded commas so the quoted-field round-trip is
exercised on download.
ADIF import and the refinement/refinement-notify paths are out of
scope — users specifically asked for CSV + single-contact. Tests
cover the changeset (accept/blank/over-length), CSV header parsing
(plain + RFC-4180 quoted), and LiveView form submit end-to-end.
When the Rust hrrr-point-worker marks a task as `failed` (upstream
`idx HTTP 404` or wgrib2 decode error on old archive formats), the
corresponding contacts sit at `hrrr_status = :queued` until the
generic 3-day stale-queued reconcile flips them. Speed that up: on
every BackfillEnqueue tick, join contacts to `hrrr_fetch_tasks` by
rounded HRRR hour and flip any `:queued` contact whose cycle is
permanently `failed` to `:unavailable` immediately.
Motivation: a handful of 2018-08-04/05 cycles (t20z/t21z/t15z/t17z)
are genuinely missing `wrfsfcf`/`wrfprsf` files on the NOAA S3
archive — other hours on the same dates are fine, so this is an
upstream data gap rather than a structural pre-2019 issue. Users
see an accurate "Partial" marker on those contacts instead of a
three-day `:queued` stall.
Tested: 4 new cases covering rounding, failed-vs-done, and the
`hrrr not in types` skip branch; full suite 2283/0.
Per-pod IEM rate limiter was 700ms (~1.4 req/sec); four pods put the
fleet at ~5.7 req/sec, above the observed 429 threshold. Retry storm
(Req exponential backoff up to 32s) piled concurrent sleeps on top of
live workers and tripped Bandit's acceptor + Oban's leader heartbeat.
- Raise default interval to 1500ms (~2.7 req/sec fleet).
- Hot pod /live timeout 3s → 5s (acceptor stalls under retry sleeps).
- prop-backfill exec probe period 30s → 120s, timeout 15s → 30s; the
release CLI round-trips through distributed Erlang and 15s wasn't
enough under load, restart-looping every ~10 min.
Three bugs were letting mechanism/terrain/radar contacts ping-pong
between :complete and :queued on every backfill cron:
1. enqueue_for_contact/2 now filters out types that are already
:complete on the contact before building jobs. Previously
mark_status!/3 demoted every non-empty jobs_by_type entry back to
:queued, overwriting the :complete that workers had just set.
2. reconcile_stale_queued/1 now also flips mechanism_status and
radar_status back to :complete when the companion data
(propagation_mechanism / contact_common_volume_radar) already
exists. Drains the existing ~14k row backlog on the first post-
deploy tick.
3. IemClient.parse_iemre_json/1 accepts "" and nil so a 200-with-
empty-body IEMRE response doesn't FunctionClauseError the worker
through four retry attempts.
lgdc.uml.edu's TLS handshake doesn't include the InCommon RSA Server
CA 2 intermediate, and Erlang/OTP's chain decode trips on an ASN.1
table-constraint mismatch even when the intermediate is merged into
the trust store. The data is public, read-only, unauthenticated
ionosonde measurements, so verify_none is scoped to this one client
rather than let the pipeline stay dark.
Hot pods were restart-looping every ~25 minutes on liveness probe
timeouts. Root cause: ScoreCacheReconciler mirrored every .prop file
from NFS into ETS, including 7 days of GEFS Day 2-7 forecasts. With
23 bands × 43 valid_times × ~2 MB per grid = 1.95 GB in the
propagation_score_cache table alone; GC sweeps starved the
scheduler enough that /live dropped its 3 s budget.
The /map UI only ever requests the [now-1h, now+18h] window. Share
that bound as Propagation.hot_cache_window/0 and apply it in the
reconciler's disk-scan path plus NotifyListener's post-warm prune.
Long-horizon GEFS files stay on disk and are still served via the
lazy read_from_disk_and_cache path when requested directly.
Adds ScoreCache.prune_outside_window/2 (inclusive bounds) and
updates the reconciler tests to use relative-to-now timestamps
since hardcoded fixture dates now drift out of window.
**Wire format — /map scores payload.** Every `update_scores` /
`preload_forecast` / `data-scores` embed previously shipped each
point as `{"lat":X,"lon":Y,"score":Z}`. At 95k cells in full CONUS
that's ~36 bytes/point × 95k = 3.4 MB of JSON, ~50% of it repeating
the three key names. Packing as `[lat, lon, score]` tuples drops
per-point overhead to ~19 bytes → ~1.8 MB on the wire, ~45% smaller.
New `Propagation.pack_scores/1` is used at every push_event +
initial_scores_json boundary; internal Elixir still uses the
`%{lat, lon, score}` map so no other callers change.
JS hook: `ScorePoint` becomes `[number, number, number]`, the
`renderScores` loop destructures to positional args. Regression
test in map_live_test updated to match the new shape.
**ProfilesFile compression.** Rust writer was at gzip level 6
(default). Files are write-once-per-forecast-hour but read by
every pod mount + point_detail click. Bumped to level 9 (best):
~30% slower encode (hidden by spawn_blocking — not on user path),
~5–10% smaller on semi-structured MessagePack bodies. No reader
changes needed — same gzip stream.
**TypeScript strictness.**
- `app.ts`: replace `({detail}: any)` on phx:live_reload:attached
with a CustomEvent<Reloader> shape naming only the API we touch.
- `weather_map_hook.ts`: drop the `(this as any)._map` pair by
declaring a `GridLayerWithMap = L.GridLayer & { _map: L.Map }`
intersection on the overlay's createTile `this` parameter.
- `global.d.ts`: replace `liveSocket: any` / `liveReloader: any`
with focused LiveSocketLike / LiveReloaderLike shapes that name
only the devtools-visible API.
No more `any` in the codebase (verified via rg).
Two wins that directly shrink /map and /weather latency under
interactive use:
**Debounce the moveend → repaint path (150ms).** Leaflet fires
`moveend` once per pan but 5–10× per second during a wheel-zoom.
Previously each one ran a full `scores_at` (or `weather_grid_at`)
read + a ~20–100KB websocket push. Now they coalesce into a single
trailing-edge flush via `schedule_bounds_update` (and the
/weather twin `schedule_weather_flush`) — the same pattern we
already use for forecast preload. 150ms is short enough that the
user doesn't perceive the pause; >90% of the burst disappears.
**Parallelize point_forecast across 4 tasks.** The sparkline that
pops up on point click was reading up to 18 `.prop` files
sequentially off NFS. Each read is cheap (pread one byte at
row*cols+col) but RTT-dominated. `Task.async_stream` with
max_concurrency=4, ordered=true brings 18-file wall time from
~50ms down to ~15ms on NFS without changing the public API.
No test changes: existing map_bounds test only asserts the bounds
assign lands, which the debounced path still does synchronously.
The main /map timeline was rendering each forecast hour twice
('Now 22:00' + '0h 22:00', '+1h 23:00' + '+1h 23:00', …).
Root cause: during the rolling `.ntms` → `.prop` rename window,
both extensions live on NFS for the same valid_time. `list_score_files`
matches either, so `list_valid_times_raw` mapped 2 files → 2 DateTimes
with no dedupe. Propagation.available_valid_times inherited the dup,
push_timeline pushed it to the JS hook, renderTimeline drew one chip
per entry — hence the visible duplicate.
Fix: `Enum.uniq` before the sort. Regression test writes a `.prop`
file, copies it as the same valid_time's `.ntms` legacy twin, and
asserts list_valid_times returns a single DateTime.
Also land a small refactor that was queued behind this: move the three
pure mechanism display helpers (label/1, badge_class/1, explainer/1)
out of the 2500-line contact_live/show.ex into a new
MicrowavepropWeb.ContactLive.Mechanism module. No behavior change — just
shrinks the LiveView and gives the display mapping a testable home.
StatusLive had a hand-rolled thousands-separator format_number/1 and a
binary-prefix format_bytes/1 — duplication that was already part of the
review punch-list. Move both to Microwaveprop.Format (which previously
held only distance_km) and delete the local defps. Fourteen call sites
switched to Format.number / Format.bytes.
Also extend the IEM error classifier to snooze on Req.TransportError
{:closed, :timeout, :econnrefused, :nxdomain}. Same rationale as the
429 snooze in batch 1: upstream TCP resets mid-fetch aren't a job
failure, they're transient — returning {:error, _} blows them up into
Oban.PerformError logspam with full stacktraces.
Batched from a system-wide review pass.
**Rate-limit + transient-error log hygiene**
- WeatherFetchWorker: classify IEM HTTP 429 as {:snooze, 300} instead
of {:error}. Stops Oban.PerformError stack-trace spam on every
routine rate-limit during backfill, keeps the retry semantics.
- IonosphereFetchWorker: compress GIRO TLS :unknown_ca error from a
~400-char inspect blob to 'TLS unknown_ca (CA bundle missing)'.
- FreshnessMonitor: log only when an enqueue actually lands (the
Oban unique conflict case was silently dropping jobs but still
producing 'enqueuing grid worker' info lines every 5 minutes).
**Perf**
- Rust grid_level_keys(): pre-format 'TMP:{p} mb' / DPT / HGT keys
once via OnceLock instead of format!()'ing per-cell. Removes ~3.6M
String allocations per f01..f18 chain step across pipeline.rs,
hrrr_points.rs. Same for the wgrib2 :(TMP|DPT|HGT): pattern in
hrrr_points.process_batch (sfc_pattern / prs_pattern OnceLock).
- MapLive preload_forecast: Task.async_stream with max_concurrency=4
replaces the 18-wide Enum.map serial walk. Forecast cache warms
~4× faster after a band change, with ordering preserved.
- grid_tasks: partial composite index on (run_time DESC,
forecast_hour ASC) WHERE status='queued' AND kind='forecast',
matching the Rust claim query's ORDER BY. Drops the old
status-only partial that forced a sort per claim.
**Correctness**
- ContactImportWorker: add Oban unique:[keys: [:import_run_id,
:offset]]. Was missing on a worker whose perform() does a
non-idempotent atomic counter increment — a retry would double-
count imported rows.
- CommonVolumeRadarWorker: x_min..x_max default step is -1 when
x_min > x_max, triggering a runtime deprecation warning. Force
Range.new(_, _, 1) explicitly.
**Cleanup**
- Drop unused tmp_dir parameter from hrrr_point_worker + its
process_batch signature.
Two unrelated production warnings from the same log window:
WeatherFetchWorker raised Postgrex 21000 cardinality_violation on
ASOS upserts when IEM returned two METAR records for the same
timestamp (routine + special at the same minute). Dedupe by
(station_id, observed_at) in upsert_surface_observations, keeping
the last occurrence since IEM's ordering is chronological.
SwpcClient.fetch_xrays warned with the full truncated response
body (~650 KB of GOES X-ray JSON) dumped into the log aggregator
whenever SWPC's CDN cut the response mid-stream. Intercept
Jason.DecodeError in fetch_and_parse and format it short: keep
the byte position and an 80-char preview, drop the rest.
Two fixes cover the blank "analysis breakdown" panel the user reported:
1. Bound the point_detail fallback lookback to 24h. Previously
factors_from_fallback_profile would happily use a week-old analysis
profile when no current one existed — now anything older than 24h
returns an empty factor map so the UI can surface "breakdown
unavailable" instead of silently misattributing.
2. Move the Plausible analytics snippet from Layouts.app into
root.html.heex. Full-bleed LiveViews (/map, /contacts/map,
/weather, home) bypass Layouts.app, so the snippet only loaded on
the navbar-wrapped pages. root.html.heex is loaded once per HTTP
document so coverage is now universal.
Added ~30 tests locking both down:
• point_detail fallback: exact-time wins, 24h boundary accepted,
30h-stale rejected, multi-band coverage, missing-cell degrades to
empty factors, default-valid_time path
• analytics_test.exs: Plausible script present on 12 representative
pages, exactly once, loaded async, surviving a login round-trip
Also fixed pre-existing credo issues per standing rule: shortened the
map_live_test ETS match_delete call, extracted a function from the
deeply-nested hrrr_point_enqueuer.enqueue_for_contacts/1 cond, and
swapped Mix.Tasks.Rust.Golden's IO.inspect for Mix.shell().info.
Score-grid files now land at `<band>/<iso>.prop` with a 4-byte "PROP"
magic header. Readers still accept the legacy `.ntms` extension and
"NTMS" magic so a rolling deploy doesn't invalidate any file already
on the shared NFS mount — legacy files age out through normal
retention.
Applied both sides of the seam: Elixir ScoresFile + regex parsers,
Rust scores_file writer + decoder. Updated the comments in all
modules that mention the extension (NotifyListener, Reconciler,
gefs_fetch_worker, map_live) and the runbook diagram. talos5 is a
DB host now, not a Rust worker — corrected the runbook accordingly.
Convert every context/worker upsert that used the catch-all
`:replace_all_except` form to the explicit `{:replace, [:col1, ...]}`
form. Reviewers can now see exactly which columns an upsert overwrites,
and adding a new column to a schema no longer silently opts it into the
update path.
Behaviour is preserved bit-for-bit: each new explicit list contains
every column the old `:replace_all_except` would have overwritten.
Touched:
- Microwaveprop.SpaceWeather (Kp / F10.7 / X-ray shared helper)
- Microwaveprop.Ionosphere (observation upsert)
- NexradWorker, HrrrNativeGridWorker (insert_all grids)
- CommonVolumeRadarWorker, RadarFrameWorker (per-contact stats)
Also pins the conflict behaviour for the ContactCommonVolumeRadar
upsert path in RadarFrameWorkerTest so a future column addition that
isn't reflected in the explicit list fails loudly.
Wrap a Task.Supervisor in a PartitionSupervisor (one partition per
scheduler) and route the four sustained async_stream callers through
`{:via, PartitionSupervisor, {Microwaveprop.TaskSupervisor, self()}}`.
Concurrent work — the hourly PropagationGridWorker pulling f00-f18
HRRR range downloads, the GEFS worker scoring its grid, and the
Recalibrator's 20-way positive/negative fan-out — no longer funnels
through a single supervisor PID.
Light/one-shot Task.async + Task.start sites (hrrr_client surface/
pressure fan-out, application boot warmup, weather fire-and-forget)
are left alone; partitioning only helps under sustained concurrency.
PromEx.Plugins.Oban counts job exceptions as a Prometheus counter but
does not tell us which worker/queue failed, whether the job is about to
be retried, or the exception class. This adds a telemetry handler on
[:oban, :job, :exception] that:
- emits a single structured Logger.error with worker, queue, job_id,
attempt, max_attempts, retry_exhausted?, kind, reason class, and
args-keys summary (never values — args can contain PII/tokens);
- dispatches opt-in per-worker recovery callbacks
(on_permanent_failure/1 when retries exhausted,
on_transient_failure/1 otherwise);
- catches callback failures so a bug in one worker's recovery path
cannot detach the handler or break logging for other jobs;
- does not emit a second telemetry event, so PromEx counts are not
double-counted.
Handler is attached from Application.start/2 after the Oban supervisor
starts.
