graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
833 commits 1 branch 0 tags 8.1 MiB
Find a file
Graham McIntire 156b170a5c
Speed up ERA5 batch pipeline: parallel CDS + stream downloads + hot index 
Four independent wins from a perf audit of the era5_batch worker path:

1. Parallel CDS submits per job. fetch_single_level_month and
   fetch_pressure_level_month have no ordering dependency — each blocks
   30+ min on CDS queue/download — so running them in two linked Tasks
   halves wall time for every job in the backlog.

2. Stream GRIB2 downloads directly to disk via Req `into: File.stream!`.
   A month-tile GRIB is 50–200 MB and was previously slurped into the
   BEAM heap and then written back out to a temp file before wgrib2 ran.
   New Era5Client.submit_and_download_to_file/3 skips both copies. Saves
   100–200 MB heap per worker × 12 concurrent ≈ 1.2–2.4 GB peak under
   full parallelism. Partial files are removed on HTTP error.

3. Wgrib2.extract_grid_messages_from_file/3 lets batch decoding read the
   streamed temp file directly. The binary variant still exists for the
   single-point path and now delegates through the same helper.

4. Composite index (valid_time, lat, lon) on era5_profiles, matching the
   three-range scan used by Weather.find_nearest_era5 (contact detail +
   path profiles) and StatusLive.count_era5_done (status page EXISTS
   subquery). The unique index on (lat, lon, valid_time) couldn't serve
   it efficiently.

Also bumps the bulk insert chunk size 2k → 4k to halve insert round-trips
per month-tile (still well under Postgres' 65k-parameter cap).
2026-04-13 16:00:19 -05:00
.forgejo/workflows Install Docker CLI in CI runner container 2026-04-11 09:04:22 -05:00
.github Add contact edit approval system with admin review queue 2026-04-11 16:15:49 -05:00
assets Refresh viewshed on band change when a point is selected 2026-04-13 09:41:58 -05:00
bin Fix livebook path resolution in bin/notebook 2026-04-07 11:15:49 -05:00
config Bump era5_batch parallelism to 12 concurrent (CDS-Beta ceiling) 2026-04-13 15:48:24 -05:00
docs Refresh 2026-04-13 recalibration report 2026-04-13 12:29:51 -05:00
k8s chore: update prop image to git.mcintire.me/graham/prop:main-1776113583-6c99798 [skip ci] 2026-04-13 20:54:35 +00:00
lib Speed up ERA5 batch pipeline: parallel CDS + stream downloads + hot index 2026-04-13 16:00:19 -05:00
lib_ml Move ML deps to dev/test only, exclude from production build 2026-04-01 12:10:31 -05:00
notebooks Auto-deploy analysis notebook as Livebook app 2026-04-07 11:15:49 -05:00
priv Speed up ERA5 batch pipeline: parallel CDS + stream downloads + hot index 2026-04-13 16:00:19 -05:00
rel Add Erlang clustering via libcluster and expose LiveDashboard 2026-04-06 09:46:48 -05:00
scripts Extend recalibrate_algo.py with NEXRAD, native-duct, commercial sections 2026-04-13 12:14:05 -05:00
test Show ERA5 fallback on contact detail, fix status page ERA5 progress 2026-04-13 13:44:31 -05:00
vendor Vendor oban_pro 1.6.13 2026-04-09 14:14:49 -05:00
.credo.exs credo 2026-04-12 10:26:53 -05:00
.dockerignore Optimize Dockerfile layer caching 2026-04-01 10:32:56 -05:00
.formatter.exs initial 2026-03-28 11:28:47 -05:00
.gitignore Fix /weather 500 on cold cache, untrack k8s secret 2026-04-12 13:56:59 -05:00
.tool-versions update deps 2026-04-11 12:03:43 -05:00
AGENTS.md initial 2026-03-28 11:28:47 -05:00
algo.md Wire NEXRAD, native ducts, and commercial links into scoring 2026-04-13 12:08:15 -05:00
app.json Migrate healthchecks from CHECKS to app.json 2026-04-04 11:24:56 -05:00
CLAUDE.md docs: update deployment notes for Kubernetes 2026-04-13 15:30:59 -05:00
Dockerfile Mount Oban Web dashboard at /admin/oban 2026-04-09 14:07:33 -05:00
elevation.md add elevation findings 2026-03-29 16:23:03 -05:00
mix.exs Unblock CI: pull igniter into :prod for live_table compile 2026-04-12 17:58:13 -05:00
mix.lock Add live_table dep and convert /users to use it 2026-04-12 17:07:07 -05:00
prediction.md HRRR forecast hours (f00-f18) with timeline map UI 2026-03-31 16:44:47 -05:00
README.md initial 2026-03-28 11:28:47 -05:00
tail_logs.sh Add type filter to backfill enqueue 2026-04-06 11:42:26 -05:00
updates.md Update seasonal scores from sounding data, document commercial link patterns 2026-04-07 11:45:44 -05:00

README.md

Microwaveprop

To start your Phoenix server:

  • Run mix setup to install and setup dependencies
  • Start Phoenix endpoint with mix phx.server or inside IEx with iex -S mix phx.server

Now you can visit localhost:4000 from your browser.

Ready to run in production? Please check our deployment guides.

Learn more