`mix prop.compare` runs both scorers against a recent contact sample
and measures both against achieved contact distance. Each run appends
to priv/calibration/history.jsonl, so trends in alg-ML divergence and
algorithm-distance correlation surface over weeks of running.
When drift exceeds threshold (alg-ML RMSE > 8 score points, or current
correlation > 0.10 below the history median), the task writes a
recommendation pointing at the right remediation:
mix recalibrate_scorer # algorithm drift → refit weights
mix propagation_train # ML drift → retrain on the new algorithm
That is the feedback loop: measure → recommend → recalibrate/retrain →
loop. Operational rather than automatic, so a bad data window can't
silently corrupt the model.
Pure analysis lives in Microwaveprop.Propagation.Calibration with its
own unit tests; the Mix task only handles data loading, file I/O, and
console formatting.
First step of the disk-backed scores migration from the DuckDB plan
doc. Ended up shipping the raw-binary variant instead of Parquet
because the data is disposable after ~2h — the ecosystem benefits
of Parquet only pay off for long-lived datasets, and the binary
path has zero new dependencies.
Microwaveprop.Propagation.ScoresFile writes one file per
(band_mhz, valid_time) tuple under the configured scores dir,
default /data/scores in prod and priv/dev_scores in dev. Layout is
a 33-byte header plus a dense n_rows × n_cols uint8 array (255 is
the no-data sentinel). The whole CONUS grid serializes to ~93 KB
per band, and writes use the temp-then-rename pattern so NFSv4
concurrent readers never see a partial file.
Propagation.replace_scores/2 now materializes the score stream
once and dual-writes: Postgres on the primary path, then one
ScoresFile per band as a best-effort follow-up (any file error is
logged but doesn't fail the DB write, so we can verify the file
path in prod before cutting readers over).
Propagation.prune_old_scores/0 also clears expired score files so
the existing 15-minute prune cron covers both storage layers.
Dev configuration points at priv/dev_scores/, added to .gitignore.
Test configuration points at a per-run tmp directory.
The /weather LiveView was timing out on pod restart because
latest_weather_grid/1 ran load_weather_grid_from_db/1 synchronously on
cache miss — that query reads 176k hrrr_profiles rows with two huge
JSONB columns (profile array + duct_characteristics), and JSONB
decoding blocked the LiveView process past the 15-second pool
ownership timeout.
On cache miss latest_weather_grid/1 now returns empty immediately and
kicks off a deduped background fill through GridCache.claim_fill/1 (a
per-valid_time ETS lock so N concurrent mounts after restart don't each
fire the slow query). When the fill completes it broadcasts
weather:updated, and the handle_info in WeatherMapLive hits the warm
cache and pushes rows over the socket.
Added load_weather_grid/1 as a synchronous sibling for tests and any
caller that genuinely needs inline data. Tests updated to use it and
to clear GridCache in setup so ETS state doesn't leak between cases.
Also untrack k8s/secret.yaml and add both it and k8s/*-secret.yaml to
.gitignore — those manifests hold plaintext SMTP and database
credentials that should not live in the repo. Secrets already in git
history should be rotated separately.
Enqueues ERA5 fetch jobs for contacts where HRRR is unavailable.
Deduplicates by rounded grid point and hour, skips existing profiles.
Usage: mix era5_backfill [limit]
ML Integration:
- Load trained model at app startup, cache compiled predict fn in persistent_term
- Grid worker uses batched ML prediction (10K chunks) when model loaded,
falls back to algorithm scorer when not
- ML score replaces composite, algorithm factor scores preserved for detail view
- Fix process explosion: single EXLA call per chunk instead of per-grid-point
QSO Features:
- Callsign search (ILIKE on station1/station2) with trigram indexes
- Reciprocal QSO grouping (same pair, same band, same hour)
- Wider layout (max-w-7xl) for data table pages
- QSO Training Data link on map page
Infrastructure:
- Re-enable hourly propagation grid worker in dev
- Track ML model weights in git for Docker builds
- Add btree indexes on qsos (timestamp, band, distance_km)
- Remove nav icons from layout header
Eliminates the external wgrib2 C tool dependency that blocked HRRR
processing. Implements Lambert Conformal projection, simple packing
(Template 5.0), complex packing with spatial differencing (Template 5.3),
and GRIB2 section parsing — enough to extract point values from HRRR
grid data using only Elixir.
