graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
551 commits 1 branch 0 tags 8.1 MiB
Find a file
Graham McIntire 8637253fda Batch ERA5 fetches by month and 2° tile 
Per-point ERA5 fetches were tragically slow because every point-hour
triggered its own asynchronous CDS job (submit → poll → assemble →
download). For backfill this meant thousands of independent jobs
queued against Copernicus. The new path groups requests by calendar
month and a 2° × 2° lat/lon tile so one CDS cycle populates ~60k
profiles at once, and Oban uniqueness on (year, month, tile_lat,
tile_lon) collapses every duplicate enqueue.

- Era5BatchClient builds the monthly CDS requests, extracts every
  (lat, lon, hour) from the GRIB2 blob with wgrib2, derives
  refractivity params, and bulk-inserts in 2k-row chunks with
  on_conflict: :nothing. fetch_month_into_db/1 short-circuits when
  the month-tile already has any cached profile.
- Era5MonthBatchWorker runs the batch on the :era5 queue with a
  generous backoff (10m → 1d) and the uniqueness key above.
- Era5FetchWorker is now a thin router: cache hit → :ok, cache miss
  → enqueue the month-batch for the point's tile-month and return.
  No more per-point CDS calls.
- Wgrib2 grows extract_grid_messages/3 which preserves per-message
  datetimes by parsing `d=YYYYMMDDHH[MMSS]` from the inventory, so a
  single GRIB2 file carrying a whole month decodes correctly.
- The era5_backfill mix task enqueues month-tile batches directly.
2026-04-09 13:01:49 -05:00
.forgejo/workflows Add k8s manifests and Forgejo CI for cluster deployment 2026-04-06 09:25:24 -05:00
assets Compact beacon detail page, toggleable coverage, About page formatting 2026-04-09 09:12:32 -05:00
bin Fix livebook path resolution in bin/notebook 2026-04-07 11:15:49 -05:00
config Decouple propagation_scores pruning from the compute worker 2026-04-09 12:30:10 -05:00
docs Path-integrated HRRR scoring across pos1/mid/pos2 2026-04-07 12:04:16 -05:00
k8s chore: update prop image to git.mcintire.me/graham/prop:main-1775756620-2bd432a [skip ci] 2026-04-09 17:44:43 +00:00
lib Batch ERA5 fetches by month and 2° tile 2026-04-09 13:01:49 -05:00
lib_ml Move ML deps to dev/test only, exclude from production build 2026-04-01 12:10:31 -05:00
notebooks Auto-deploy analysis notebook as Livebook app 2026-04-07 11:15:49 -05:00
priv Add beacon bearing, beamwidth, and weak-signal keyings 2026-04-09 08:59:16 -05:00
rel Add Erlang clustering via libcluster and expose LiveDashboard 2026-04-06 09:46:48 -05:00
test Batch ERA5 fetches by month and 2° tile 2026-04-09 13:01:49 -05:00
.dockerignore Optimize Dockerfile layer caching 2026-04-01 10:32:56 -05:00
.formatter.exs initial 2026-03-28 11:28:47 -05:00
.gitignore Add mix era5_backfill task and gitignore .envrc 2026-04-07 12:24:18 -05:00
.tool-versions Normalize ML features to prevent NaN gradient explosion 2026-04-01 09:08:33 -05:00
AGENTS.md initial 2026-03-28 11:28:47 -05:00
algo.md Document ERA5, RTMA, and path-integrated HRRR in algo.md 2026-04-07 12:37:51 -05:00
app.json Migrate healthchecks from CHECKS to app.json 2026-04-04 11:24:56 -05:00
CLAUDE.md Comprehensive CLAUDE.md update with full project architecture and setup 2026-03-31 17:52:57 -05:00
Dockerfile Fix Dockerfile: copy algo.md before mix compile (read at compile time) 2026-04-01 10:38:35 -05:00
elevation.md add elevation findings 2026-03-29 16:23:03 -05:00
mix.exs Add beacon monitor registration and fix prod SMTP 2026-04-08 11:30:28 -05:00
mix.lock Add beacon monitor registration and fix prod SMTP 2026-04-08 11:30:28 -05:00
prediction.md HRRR forecast hours (f00-f18) with timeline map UI 2026-03-31 16:44:47 -05:00
README.md initial 2026-03-28 11:28:47 -05:00
tail_logs.sh Add type filter to backfill enqueue 2026-04-06 11:42:26 -05:00
updates.md Update seasonal scores from sounding data, document commercial link patterns 2026-04-07 11:45:44 -05:00

README.md

Microwaveprop

To start your Phoenix server:

  • Run mix setup to install and setup dependencies
  • Start Phoenix endpoint with mix phx.server or inside IEx with iex -S mix phx.server

Now you can visit localhost:4000 from your browser.

Ready to run in production? Please check our deployment guides.

Learn more