graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
1138 commits 1 branch 0 tags 8.1 MiB
Find a file
Graham McIntire f122eedfa8
perf(propagation): parallel chain fan-out + hoist shared factors 
Two wins on the hourly propagation pipeline:

1. Parallelize the chain. seed_chain was enqueuing only f00 and
   each step self-enqueued f00+1, serializing ~2.5 min × 19
   forecast hours into a ~48 min chain. Fan out all 19 jobs at
   once — they're genuinely independent (different HRRR URLs,
   different output files) — and let queue concurrency (2 slots
   × 3 pods = 6 parallel workers) drop wall time to ~10 min.

   Side effect: one permanently-failing step no longer takes out
   the rest of the chain, so the rescue-chain logic I added last
   commit becomes unnecessary. Removed.

   The :final cleanup (retain_window + purge) used to run on the
   fh=18 transition; with parallel execution we don't know which
   step finishes last. Cleanup now relies on the existing
   PropagationPruneWorker 15-min cron for time-based pruning.
   Stale chain leftovers are already a minor concern with the
   15-min prune; if file bloat becomes real, PruneWorker can
   gain retain_window later.

2. Hoist band-invariant factors. score_time_of_day, score_sky,
   score_wind, score_pressure depend on conditions only, not the
   band — but composite_score was recomputing them 17 times per
   point. Added Scorer.precompute_band_invariants/1, called once
   per point before the bands loop; composite_score uses the
   cached values when present and falls back to computing for
   any caller that doesn't pre-warm (test harness, path
   integrator). Saves ~30% of the scoring inner loop.
2026-04-19 09:39:06 -05:00
.forgejo/workflows ci: install buildx and use docker buildx build --push 2026-04-18 15:21:30 -05:00
.github Add contact edit approval system with admin review queue 2026-04-11 16:15:49 -05:00
assets fix(propagation): chain survives permanent step failures 2026-04-19 09:26:20 -05:00
bin Fix livebook path resolution in bin/notebook 2026-04-07 11:15:49 -05:00
config feat(telemetry): PromEx Prometheus exporter at /metrics 2026-04-18 16:39:39 -05:00
docs docs: plan for GFS-driven extended-horizon forecasts (18 h → 240 h) 2026-04-18 12:35:08 -05:00
k8s chore: update prop image to git.mcintire.me/graham/prop:main-1776608794-5cfb9e6 [skip ci] 2026-04-19 14:28:47 +00:00
lib perf(propagation): parallel chain fan-out + hoist shared factors 2026-04-19 09:39:06 -05:00
lib_ml refactor: normalize pos1/pos2 JSONB key to 'lon' everywhere 2026-04-17 09:10:32 -05:00
notebooks refactor: normalize pos1/pos2 JSONB key to 'lon' everywhere 2026-04-17 09:10:32 -05:00
priv feat(contacts): private flag with scope-aware visibility 2026-04-18 17:49:01 -05:00
rel Add Erlang clustering via libcluster and expose LiveDashboard 2026-04-06 09:46:48 -05:00
scripts feat(propagation): per-band weight calibration from full-corpus correlation 2026-04-18 10:19:26 -05:00
test perf(propagation): parallel chain fan-out + hoist shared factors 2026-04-19 09:39:06 -05:00
vendor Vendor oban_pro 1.6.13 2026-04-09 14:14:49 -05:00
.credo.exs credo 2026-04-12 10:26:53 -05:00
.dockerignore Optimize Dockerfile layer caching 2026-04-01 10:32:56 -05:00
.formatter.exs initial 2026-03-28 11:28:47 -05:00
.gitignore Dual-write propagation scores to binary files on /data 2026-04-14 14:36:55 -05:00
.tool-versions update deps 2026-04-11 12:03:43 -05:00
AGENTS.md initial 2026-03-28 11:28:47 -05:00
algo.md docs(algo): align algo.md with Richardson-gated scoring implementation 2026-04-18 12:42:20 -05:00
app.json Migrate healthchecks from CHECKS to app.json 2026-04-04 11:24:56 -05:00
CLAUDE.md fix(maps): same reconnect/visibility fix for rover and weather hooks 2026-04-18 13:25:38 -05:00
Dockerfile fix(docker): use BuildKit cache mounts for apt so .debs skip the layer 2026-04-18 11:07:45 -05:00
elevation.md add elevation findings 2026-03-29 16:23:03 -05:00
mix.exs feat(telemetry): PromEx Prometheus exporter at /metrics 2026-04-18 16:39:39 -05:00
mix.lock feat(telemetry): PromEx Prometheus exporter at /metrics 2026-04-18 16:39:39 -05:00
prediction.md HRRR forecast hours (f00-f18) with timeline map UI 2026-03-31 16:44:47 -05:00
README.md initial 2026-03-28 11:28:47 -05:00
tail_logs.sh Add type filter to backfill enqueue 2026-04-06 11:42:26 -05:00
updates.md Update seasonal scores from sounding data, document commercial link patterns 2026-04-07 11:45:44 -05:00

README.md

Microwaveprop

To start your Phoenix server:

  • Run mix setup to install and setup dependencies
  • Start Phoenix endpoint with mix phx.server or inside IEx with iex -S mix phx.server

Now you can visit localhost:4000 from your browser.

Ready to run in production? Please check our deployment guides.

Learn more