graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
1400 commits 1 branch 0 tags 8.1 MiB
Find a file
Graham McIntire b3c035793a
Add incremental prod-to-local DB sync script 
scripts/sync_prod_to_local.sh pulls only rows whose updated_at is
newer than the corresponding local row, versus restore_prod_to_local
which does a full pg_dump + drop + restore. For day-to-day refreshes
the incremental path is dramatically cheaper — the full dump is ~12
GB, while an hourly incremental is typically a few thousand rows.

Approach:
- Discover every public table that carries `updated_at` (skips the
  oban_* runtime tables and schema_migrations by default).
- For each, intersect the column list across local and prod so schema
  drift (a new column on prod not yet migrated locally, or vice-versa)
  doesn't break the sync — we transfer the columns that exist on
  both sides and leave the rest alone.
- Resolve the ON CONFLICT target: primary key first, else the
  shortest non-partial unique index (important for partitioned tables
  like hrrr_profiles which have no top-level PK).
- Stream the changed rows via `\COPY ... TO STDOUT` into a per-table
  temp file, then load into a LIKE-cloned temp table and UPSERT
  across `SET session_replication_role = replica` so FK ordering
  doesn't block the dev-only run.
- Partitioned parents (relkind='p', relispartition=false) are kept in
  the discovery list; partition children (relkind='r',
  relispartition=true) are excluded so INSERTs route through the
  parent.

Flags: `--dry-run` counts without writing; a positional argument
limits the run to a single table. `SKIP_TABLES` env var overrides
the exclusion list.

Also added a pointer from restore_prod_to_local.sh's header comment
to the new script.
2026-04-23 13:57:22 -05:00
.forgejo/workflows ci(grid-rs): surface docker daemon diagnostics on wait-step failure 2026-04-20 08:41:53 -05:00
.github Add contact edit approval system with admin review queue 2026-04-11 16:15:49 -05:00
assets perf(wire)+ts: pack scores as flat tuples, bump ProfilesFile gzip, tighten TS types 2026-04-21 17:45:06 -05:00
bin Fix livebook path resolution in bin/notebook 2026-04-07 11:15:49 -05:00
config perf(oban): move IEM-heavy queues off hot pods 2026-04-23 10:53:26 -05:00
docs refactor(scores): rename file extension .ntms -> .prop with legacy reads 2026-04-21 15:54:12 -05:00
k8s chore: update prop image to git.mcintire.me/graham/prop:main-1776968936-764643b [skip ci] 2026-04-23 18:31:46 +00:00
lib chore(deps): upgrade live_stash 0.1.2 → 0.2.0 2026-04-23 13:56:28 -05:00
lib_ml fix(dialyzer): clear 125+ warnings under strict flags 2026-04-21 10:30:06 -05:00
notebooks refactor: normalize pos1/pos2 JSONB key to 'lon' everywhere 2026-04-17 09:10:32 -05:00
priv feat(contacts): add free-form notes field 2026-04-23 11:02:27 -05:00
rel Add Erlang clustering via libcluster and expose LiveDashboard 2026-04-06 09:46:48 -05:00
rust/prop_grid_rs fix(rust): snap HRRR point lookups to 0.125° grid 2026-04-22 16:50:58 -05:00
scripts Add incremental prod-to-local DB sync script 2026-04-23 13:57:22 -05:00
test test: broaden coverage across parsers, schemas, and pure helpers 2026-04-23 13:56:29 -05:00
vendor Vendor oban_pro 1.6.13 2026-04-09 14:14:49 -05:00
.credo.exs credo 2026-04-12 10:26:53 -05:00
.dockerignore Optimize Dockerfile layer caching 2026-04-01 10:32:56 -05:00
.formatter.exs initial 2026-03-28 11:28:47 -05:00
.gitignore Dual-write propagation scores to binary files on /data 2026-04-14 14:36:55 -05:00
.tool-versions update deps 2026-04-11 12:03:43 -05:00
AGENTS.md initial 2026-03-28 11:28:47 -05:00
algo.md docs(algo): align algo.md with Richardson-gated scoring implementation 2026-04-18 12:42:20 -05:00
app.json Migrate healthchecks from CHECKS to app.json 2026-04-04 11:24:56 -05:00
CLAUDE.md fix(maps): same reconnect/visibility fix for rover and weather hooks 2026-04-18 13:25:38 -05:00
Dockerfile fix(docker): use BuildKit cache mounts for apt so .debs skip the layer 2026-04-18 11:07:45 -05:00
elevation.md add elevation findings 2026-03-29 16:23:03 -05:00
LICENSE Add GPLv3 license 2026-04-23 13:57:07 -05:00
mix.exs chore(deps): upgrade live_stash 0.1.2 → 0.2.0 2026-04-23 13:56:28 -05:00
mix.lock chore(deps): upgrade live_stash 0.1.2 → 0.2.0 2026-04-23 13:56:28 -05:00
prediction.md HRRR forecast hours (f00-f18) with timeline map UI 2026-03-31 16:44:47 -05:00
README.md initial 2026-03-28 11:28:47 -05:00
tail_logs.sh Add type filter to backfill enqueue 2026-04-06 11:42:26 -05:00
updates.md Update seasonal scores from sounding data, document commercial link patterns 2026-04-07 11:45:44 -05:00

README.md

Microwaveprop

To start your Phoenix server:

  • Run mix setup to install and setup dependencies
  • Start Phoenix endpoint with mix phx.server or inside IEx with iex -S mix phx.server

Now you can visit localhost:4000 from your browser.

Ready to run in production? Please check our deployment guides.

Learn more