scripts/sync_prod_to_local.sh pulls only rows whose updated_at is
newer than the corresponding local row, versus restore_prod_to_local
which does a full pg_dump + drop + restore. For day-to-day refreshes
the incremental path is dramatically cheaper — the full dump is ~12
GB, while an hourly incremental is typically a few thousand rows.
Approach:
- Discover every public table that carries `updated_at` (skips the
oban_* runtime tables and schema_migrations by default).
- For each, intersect the column list across local and prod so schema
drift (a new column on prod not yet migrated locally, or vice-versa)
doesn't break the sync — we transfer the columns that exist on
both sides and leave the rest alone.
- Resolve the ON CONFLICT target: primary key first, else the
shortest non-partial unique index (important for partitioned tables
like hrrr_profiles which have no top-level PK).
- Stream the changed rows via `\COPY ... TO STDOUT` into a per-table
temp file, then load into a LIKE-cloned temp table and UPSERT
across `SET session_replication_role = replica` so FK ordering
doesn't block the dev-only run.
- Partitioned parents (relkind='p', relispartition=false) are kept in
the discovery list; partition children (relkind='r',
relispartition=true) are excluded so INSERTs route through the
parent.
Flags: `--dry-run` counts without writing; a positional argument
limits the run to a single table. `SKIP_TABLES` env var overrides
the exclusion list.
Also added a pointer from restore_prod_to_local.sh's header comment
to the new script.
Ran recalibrate_algo.py against the full local prop_dev (81,994 contacts,
18.6M HRRR rows) and derived per-band composite weights for the nine
bands with >=200 matched contacts. Moisture (dewpoint/PWAT/surface N)
is consistently beneficial through 5.76 GHz, reverses at 24 GHz; rain
scales sqrt(rain_k) not linearly so 24 GHz gets 0.215 rain weight
instead of the linear-ratio 0.95. 10 GHz stays as the reference band
(defaults); 47+ GHz inherits defaults (n<200).
- BandConfig.weights/1 returns per-band override or default fallback
- @band_configs carries :weights on 222/432/902/1296/2304/3400/5760/24G
- Recalibrator.compute_factors/3 + fit(band_mhz:) band-aware fitting
- Scorer + ContactLive.Show pass band_config to weights/1
- algo.md Part 2d documents the 2026-04-18 analysis + derivation rule
- scripts/derive_band_weights.py turns correlations into weight maps
- report preserved at docs/algo-reports/2026-04-18-recalibration.md
Swap the SSH+scp-from-backup approach for a direct pg_dump against the
prod Postgres using PROP_PROD_DB_URL already in .envrc. Pin the
pg_dump/pg_restore/psql binaries to the @18 series so they match the
prod server version, and stream per-object progress via --verbose on
both tools.
Rename the local dev database from microwaveprop_dev to prop_dev so
the name matches the production database. Add scripts/restore_prod_to_local.sh,
which pulls the latest pg_dump off the backup server, drops the local
prop_dev, recreates it, and pg_restores into it.
Three new analysis passes in scripts/recalibrate_algo.py that match the
scoring factors landed in commit 86364f4:
* NEXRAD composite reflectivity vs distance per band — checks that the
Marshall-Palmer rain rate in Scorer.dbz_to_rain_rate_mmhr/1 moves the
needle in the expected direction at 24+ GHz.
* hrrr_native_profiles.best_duct_band_ghz vs distance per band —
validates the 1.15× boost in Scorer.score_refractivity/4. Current
corpus shows essentially zero contacts with a native duct supporting
≥5 GHz at the target band, consistent with the 0.12% rate in Part 2c;
the boost is physics-correct but almost never fires today.
* Commercial-link rx_power degradation vs contemporaneous DFW contacts —
pearson + binned distance. Empty today because commercial_samples only
covers Mar 30–Apr 13 and contests don't overlap. Scaffolding is in
place so the moment contest-season overlap exists, the signal shows.
Report rebuilt at docs/algo-reports/2026-04-13-recalibration.md.
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k
surface obs, new hrrr_native_profiles table) drive these changes:
* Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity
score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus
removed" conclusion was an artifact of the prior matching strategy;
with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km
avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins.
* Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in
those bands stop being silently dropped. Coefficients extrapolate
ITU-R P.676/P.838 trends from the existing 134/241 GHz entries.
* Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker
max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump
the full failure body when CDS omits the message field — the prior
"ERA5 job failed: nil" was masking real error reasons in oban_jobs.
* Add scripts/recalibrate_algo.py so this analysis can be re-run any
time new data lands without manual SQL. Reads PROP_PROD_DB_URL from
.envrc, drops a Markdown report into docs/algo-reports/.
* Append a dated Part 2c to algo.md documenting the corpus expansion,
the honest accounting of historical HRRR coverage (only ~1,020 of 58k
contacts are precision-matchable), and the empty era5/rtma/climatology
tables. Update the gaseous-absorption and rain-attenuation tables to
include the new bands.
Test suite: 1,335 tests, 0 failures.
