graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
1165 commits 1 branch 0 tags 8.1 MiB
Find a file
Graham McIntire b80878056d
feat(grid-rs): Rust worker for HRRR f01..f18 propagation chain 
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.

Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.

Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
  - grid, region, band_config, scores_file, sounding_params
  - scorer: all 10 factors + composite. Matches the Elixir scorer
    byte-for-byte across 115 golden-fixture samples (5 scenarios
    × 23 bands).
  - decoder: wgrib2 subprocess + Fortran-record lola-binary parser
  - fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
    parallel byte-range downloads with 429/5xx retry
  - pipeline: end-to-end chain step, f00 rejected at the boundary
  - db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
  - bin/worker: tokio main loop, JSON logs, SIGTERM-safe

91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.

Elixir additions:
  - `GridTaskEnqueuer` seeds fh=1..18 rows from
    `PropagationGridWorker.seed_chain/0`
  - `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
  - `ShadowComparator` diffs prod vs shadow `.ntms` bodies
  - `mix rust.golden` task writes the Rust-side golden fixture

k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.

The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
2026-04-19 15:42:49 -05:00
.forgejo/workflows Revert "ci: build against a dind sidecar instead of the host socket" 2026-04-19 12:29:24 -05:00
.github Add contact edit approval system with admin review queue 2026-04-11 16:15:49 -05:00
assets fix(map): anchor timeline offsets to the Now slot, not wall-clock 2026-04-19 12:32:41 -05:00
bin Fix livebook path resolution in bin/notebook 2026-04-07 11:15:49 -05:00
config fix(oban): use paused queues instead of local_limit:0 on backfill pods 2026-04-19 15:42:49 -05:00
docs docs: plan for GFS-driven extended-horizon forecasts (18 h → 240 h) 2026-04-18 12:35:08 -05:00
k8s feat(grid-rs): Rust worker for HRRR f01..f18 propagation chain 2026-04-19 15:42:49 -05:00
lib feat(grid-rs): Rust worker for HRRR f01..f18 propagation chain 2026-04-19 15:42:49 -05:00
lib_ml refactor: normalize pos1/pos2 JSONB key to 'lon' everywhere 2026-04-17 09:10:32 -05:00
notebooks refactor: normalize pos1/pos2 JSONB key to 'lon' everywhere 2026-04-17 09:10:32 -05:00
priv feat(grid-rs): Rust worker for HRRR f01..f18 propagation chain 2026-04-19 15:42:49 -05:00
rel Add Erlang clustering via libcluster and expose LiveDashboard 2026-04-06 09:46:48 -05:00
rust/prop_grid_rs feat(grid-rs): Rust worker for HRRR f01..f18 propagation chain 2026-04-19 15:42:49 -05:00
scripts feat(propagation): per-band weight calibration from full-corpus correlation 2026-04-18 10:19:26 -05:00
test feat(grid-rs): Rust worker for HRRR f01..f18 propagation chain 2026-04-19 15:42:49 -05:00
vendor Vendor oban_pro 1.6.13 2026-04-09 14:14:49 -05:00
.credo.exs credo 2026-04-12 10:26:53 -05:00
.dockerignore Optimize Dockerfile layer caching 2026-04-01 10:32:56 -05:00
.formatter.exs initial 2026-03-28 11:28:47 -05:00
.gitignore Dual-write propagation scores to binary files on /data 2026-04-14 14:36:55 -05:00
.tool-versions update deps 2026-04-11 12:03:43 -05:00
AGENTS.md initial 2026-03-28 11:28:47 -05:00
algo.md docs(algo): align algo.md with Richardson-gated scoring implementation 2026-04-18 12:42:20 -05:00
app.json Migrate healthchecks from CHECKS to app.json 2026-04-04 11:24:56 -05:00
CLAUDE.md fix(maps): same reconnect/visibility fix for rover and weather hooks 2026-04-18 13:25:38 -05:00
Dockerfile fix(docker): use BuildKit cache mounts for apt so .debs skip the layer 2026-04-18 11:07:45 -05:00
elevation.md add elevation findings 2026-03-29 16:23:03 -05:00
mix.exs feat(telemetry): PromEx Prometheus exporter at /metrics 2026-04-18 16:39:39 -05:00
mix.lock feat(telemetry): PromEx Prometheus exporter at /metrics 2026-04-18 16:39:39 -05:00
prediction.md HRRR forecast hours (f00-f18) with timeline map UI 2026-03-31 16:44:47 -05:00
README.md initial 2026-03-28 11:28:47 -05:00
tail_logs.sh Add type filter to backfill enqueue 2026-04-06 11:42:26 -05:00
updates.md Update seasonal scores from sounding data, document commercial link patterns 2026-04-07 11:45:44 -05:00

README.md

Microwaveprop

To start your Phoenix server:

  • Run mix setup to install and setup dependencies
  • Start Phoenix endpoint with mix phx.server or inside IEx with iex -S mix phx.server

Now you can visit localhost:4000 from your browser.

Ready to run in production? Please check our deployment guides.

Learn more