scripts/sync_prod_to_local.sh pulls only rows whose updated_at is
newer than the corresponding local row, versus restore_prod_to_local
which does a full pg_dump + drop + restore. For day-to-day refreshes
the incremental path is dramatically cheaper — the full dump is ~12
GB, while an hourly incremental is typically a few thousand rows.
Approach:
- Discover every public table that carries `updated_at` (skips the
oban_* runtime tables and schema_migrations by default).
- For each, intersect the column list across local and prod so schema
drift (a new column on prod not yet migrated locally, or vice-versa)
doesn't break the sync — we transfer the columns that exist on
both sides and leave the rest alone.
- Resolve the ON CONFLICT target: primary key first, else the
shortest non-partial unique index (important for partitioned tables
like hrrr_profiles which have no top-level PK).
- Stream the changed rows via `\COPY ... TO STDOUT` into a per-table
temp file, then load into a LIKE-cloned temp table and UPSERT
across `SET session_replication_role = replica` so FK ordering
doesn't block the dev-only run.
- Partitioned parents (relkind='p', relispartition=false) are kept in
the discovery list; partition children (relkind='r',
relispartition=true) are excluded so INSERTs route through the
parent.
Flags: `--dry-run` counts without writing; a positional argument
limits the run to a single table. `SKIP_TABLES` env var overrides
the exclusion list.
Also added a pointer from restore_prod_to_local.sh's header comment
to the new script.
Swap the SSH+scp-from-backup approach for a direct pg_dump against the
prod Postgres using PROP_PROD_DB_URL already in .envrc. Pin the
pg_dump/pg_restore/psql binaries to the @18 series so they match the
prod server version, and stream per-object progress via --verbose on
both tools.
Rename the local dev database from microwaveprop_dev to prop_dev so
the name matches the production database. Add scripts/restore_prod_to_local.sh,
which pulls the latest pg_dump off the backup server, drops the local
prop_dev, recreates it, and pg_restores into it.
