Python pskr_mqtt_listen.py:
- Guard against malformed CONNACK/SUBACK/PUBLISH packets
- Fix _s() truthiness: is not None instead of if v (zero SNR was lost)
- EINTR-safe select loop, OOB data handling, VBInt overflow check
- Proper socket cleanup with try/finally + DISCONNECT on shutdown
Elixir:
- Log dropped spot errors in aggregator (was silently discarded)
- Wire retain_scores_window into NotifyListener chain completion
- Recurse sweep_tmp_dir into band/weather_scalars subdirectories
- Rescue recalibrator.run/0 to write failed status row on crash
- Handle nil in fmt_snr/fmt_callsigns (no more nil dB crashes)
- Replace Stream.run with Enum.reduce logging exits in poll_worker
- Handle File.stat race in ms_footprints prune, grid_center nil log
- Fix unused variables, stale comments, missing get_path!/1
Rust:
- Graceful JoinError handling in fetcher/hrdps_fetcher (no more panics)
- round_to_5min returns Option (leap-second safe)
- Acquire/Release ordering on shutdown flags (was Relaxed)
- Parameterized NOTIFY in db.rs, defensive scalar sweep, NaN guard
- OsString::push for tmp naming, clippy fixes
Replace the two-script Python pipeline (analysis report + Elixir-source
emitter) with a single `scripts/recalibrate.py` that fits per-band
weights from PSKR spot density (VHF/UHF) and contacts↔HRRR correlations
(microwave), writing `priv/algo/band_weights.json` as a machine-readable
artifact. A new Elixir BandWeights module loads this JSON once via
`:persistent_term` cache; BandConfig.weights/1 consults it before falling
back to in-source overrides or global defaults. The script never touches
Elixir source — recalibration is now `python3 scripts/recalibrate.py`
followed by an app restart.
Both RTMA and METAR5 schemas + clients + workers were defined but
never had a consumer. `find_nearest_rtma/3` and `recent_surface_obs/3`
existed in `Microwaveprop.Weather` with zero callers cluster-wide;
RtmaFetchWorker had test coverage but was never enqueued anywhere;
no IEM 5-min ASOS fetcher was ever written. The tables sat empty in
prod and the recalibrate audit was permanently flagging them BROKEN
even though the empty state was the steady state.
Drop the lot — schemas, clients, workers, queue config, Req.Test
stubs, audit checks, and the per-table unique tests in the
observation-changesets suite. New migration drops the now-unreferenced
`rtma_observations` and `metar_5min_observations` tables (both 0
rows in prod). Trim the `:rtma` slot out of `backfill_only_queues`
in runtime.exs so Oban Pro's Smart engine stops trying to manage a
queue with no producer.
Audit thresholds reset on the surviving NARR check: drop the
"WARN < 5000 absolute rows" rule that didn't account for
NarrFetchWorker's 0.13° spatial dedup, and point remediation at
the existing BackfillEnqueueWorker cron instead of the dev-only
`mix narr.backfill` task.
scripts/recalibrate_algo.py: ROW_COUNT_SOURCES + DATA_GAP_SQL +
DATA_GAP_RULES all stop referencing the dropped tables.
The hrrr_climatology table previously rebuilt only when an operator
remembered to run `mix hrrr_climatology` from a workstation, which
the prod environment can't do. AdminTaskWorker already exposes the
same logic as a "climatology" task, so a one-line cron entry
(02:30 UTC daily) lets prod self-heal: empty/stale rows refold from
the hrrr_profiles archive within 24h, no manual intervention.
Recalibration audit remediation now points at the cron path instead
of the dev-only mix task.
The "Data gaps" section was a flat key/value dump that buried what was
actually broken (hrrr_climatology=0, rtma=0, metar5=0) under metrics
that are merely informational (pre2014_contacts=207). Operators reading
the report had to know each metric's healthy range by heart.
This commit:
* Classifies every gap metric BROKEN / WARN / OK via DATA_GAP_RULES,
with a one-line remediation hint per non-OK row (e.g. "run
`mix hrrr_climatology`").
* Prints BROKEN items to stderr at runtime so a CI/cron run surfaces
them without anyone opening the markdown file.
* Adds NARR-coverage-by-year and HRRR-status-by-year tables. With
only 358 NARR rows, the year breakdown shows whether the backfill
ever started; with 36k pending HRRR contacts, the year breakdown
shows whether the backlog is a stalled historical backfill or a
stuck live queue — different remediations.
* Reports HRRR enrichment completeness as a percentage instead of
forcing the reader to do the division.
scripts/sync_prod_to_local.sh pulls only rows whose updated_at is
newer than the corresponding local row, versus restore_prod_to_local
which does a full pg_dump + drop + restore. For day-to-day refreshes
the incremental path is dramatically cheaper — the full dump is ~12
GB, while an hourly incremental is typically a few thousand rows.
Approach:
- Discover every public table that carries `updated_at` (skips the
oban_* runtime tables and schema_migrations by default).
- For each, intersect the column list across local and prod so schema
drift (a new column on prod not yet migrated locally, or vice-versa)
doesn't break the sync — we transfer the columns that exist on
both sides and leave the rest alone.
- Resolve the ON CONFLICT target: primary key first, else the
shortest non-partial unique index (important for partitioned tables
like hrrr_profiles which have no top-level PK).
- Stream the changed rows via `\COPY ... TO STDOUT` into a per-table
temp file, then load into a LIKE-cloned temp table and UPSERT
across `SET session_replication_role = replica` so FK ordering
doesn't block the dev-only run.
- Partitioned parents (relkind='p', relispartition=false) are kept in
the discovery list; partition children (relkind='r',
relispartition=true) are excluded so INSERTs route through the
parent.
Flags: `--dry-run` counts without writing; a positional argument
limits the run to a single table. `SKIP_TABLES` env var overrides
the exclusion list.
Also added a pointer from restore_prod_to_local.sh's header comment
to the new script.
Ran recalibrate_algo.py against the full local prop_dev (81,994 contacts,
18.6M HRRR rows) and derived per-band composite weights for the nine
bands with >=200 matched contacts. Moisture (dewpoint/PWAT/surface N)
is consistently beneficial through 5.76 GHz, reverses at 24 GHz; rain
scales sqrt(rain_k) not linearly so 24 GHz gets 0.215 rain weight
instead of the linear-ratio 0.95. 10 GHz stays as the reference band
(defaults); 47+ GHz inherits defaults (n<200).
- BandConfig.weights/1 returns per-band override or default fallback
- @band_configs carries :weights on 222/432/902/1296/2304/3400/5760/24G
- Recalibrator.compute_factors/3 + fit(band_mhz:) band-aware fitting
- Scorer + ContactLive.Show pass band_config to weights/1
- algo.md Part 2d documents the 2026-04-18 analysis + derivation rule
- scripts/derive_band_weights.py turns correlations into weight maps
- report preserved at docs/algo-reports/2026-04-18-recalibration.md
Swap the SSH+scp-from-backup approach for a direct pg_dump against the
prod Postgres using PROP_PROD_DB_URL already in .envrc. Pin the
pg_dump/pg_restore/psql binaries to the @18 series so they match the
prod server version, and stream per-object progress via --verbose on
both tools.
Rename the local dev database from microwaveprop_dev to prop_dev so
the name matches the production database. Add scripts/restore_prod_to_local.sh,
which pulls the latest pg_dump off the backup server, drops the local
prop_dev, recreates it, and pg_restores into it.
Three new analysis passes in scripts/recalibrate_algo.py that match the
scoring factors landed in commit 86364f4:
* NEXRAD composite reflectivity vs distance per band — checks that the
Marshall-Palmer rain rate in Scorer.dbz_to_rain_rate_mmhr/1 moves the
needle in the expected direction at 24+ GHz.
* hrrr_native_profiles.best_duct_band_ghz vs distance per band —
validates the 1.15× boost in Scorer.score_refractivity/4. Current
corpus shows essentially zero contacts with a native duct supporting
≥5 GHz at the target band, consistent with the 0.12% rate in Part 2c;
the boost is physics-correct but almost never fires today.
* Commercial-link rx_power degradation vs contemporaneous DFW contacts —
pearson + binned distance. Empty today because commercial_samples only
covers Mar 30–Apr 13 and contests don't overlap. Scaffolding is in
place so the moment contest-season overlap exists, the signal shows.
Report rebuilt at docs/algo-reports/2026-04-13-recalibration.md.
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k
surface obs, new hrrr_native_profiles table) drive these changes:
* Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity
score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus
removed" conclusion was an artifact of the prior matching strategy;
with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km
avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins.
* Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in
those bands stop being silently dropped. Coefficients extrapolate
ITU-R P.676/P.838 trends from the existing 134/241 GHz entries.
* Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker
max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump
the full failure body when CDS omits the message field — the prior
"ERA5 job failed: nil" was masking real error reasons in oban_jobs.
* Add scripts/recalibrate_algo.py so this analysis can be re-run any
time new data lands without manual SQL. Reads PROP_PROD_DB_URL from
.envrc, drops a Markdown report into docs/algo-reports/.
* Append a dated Part 2c to algo.md documenting the corpus expansion,
the honest accounting of historical HRRR coverage (only ~1,020 of 58k
contacts are precision-matchable), and the empty era5/rtma/climatology
tables. Update the gaseous-absorption and rain-attenuation tables to
include the new bands.
Test suite: 1,335 tests, 0 failures.
