graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
1751 commits 1 branch 0 tags 8.1 MiB
Find a file
Graham McIntire 9edaee9344
feat(weather): GridCache backend pluggable, Valkey for cross-pod cache 
Each cached HRRR valid_time is ~32 MiB compressed × cap of 8 = up to
256 MiB per pod. With 4 hot pods + 1 backfill that's ~1.25 GiB of
cluster RAM holding identical content. Replacing the per-pod ETS
replicas with a single shared Valkey copy collapses that to ~256 MiB
total (off-pod) and makes the BEAM heap on each replica meaningfully
smaller — directly addresses the headroom the 2026-05-03 OOM cascade
ate into.

* New `Microwaveprop.Valkey`: single Redix connection (named conn,
  sync_connect: false, exit_on_disconnection: false). `start_link/0`
  returns `:ignore` when VALKEY_URL is unset so dev/test boot without
  Valkey. Helpers wrap GET/MGET/SET-with-TTL/ZADD/ZREVRANGE/SCAN with
  `:erlang.term_to_binary` round-tripping under the safe atom flag.

* `Microwaveprop.Weather.GridCache` now branches on
  `Valkey.configured?/0`. Public API is identical so callers (Weather,
  GridCachePruneWorker, MapLive) don't change. ETS path is preserved
  unchanged for dev/test.

* Storage layout when on Valkey:
    prop:wg:vts                   ZSET of valid_time iso8601 strings
    prop:wg:<vt>:<lat>:<lon>      one chunk per 5°×5° spatial bucket
  fetch_bounds MGETs only the chunks intersecting the viewport, so a
  regional view is 1-4 round-trips instead of pulling the full grid.

* Per-key TTL = 8 h (matches the ETS valid_time cap). Eviction is
  automatic; prune_keep_latest is a no-op on Valkey, prune_older_than
  uses ZRANGEBYSCORE + DEL.

* All Valkey errors fall back to disk (ScalarFile.read_bounds), logged
  as warnings — page works through Valkey outages, no user-visible
  break beyond a one-shot latency bump.

ScoreCache, Microwaveprop.Cache, NexradCache, telemetry/Postgres
protocol/libcluster registry tables remain ETS — they're sub-ms
hot-path lookups where a network hop would be felt.

VALKEY_URL must be present in prop-secrets for production rollout
(applied out-of-band, secret.yaml is git-ignored).
2026-05-03 17:00:51 -05:00
.forgejo/workflows ci: rebuild prop-base weekly to pull Debian security patches 2026-05-03 11:51:08 -05:00
.github Add contact edit approval system with admin review queue 2026-04-11 16:15:49 -05:00
assets style: switch site font to Cascadia Code 2026-05-03 14:29:14 -05:00
bin Fix livebook path resolution in bin/notebook 2026-04-07 11:15:49 -05:00
config feat(weather): GridCache backend pluggable, Valkey for cross-pod cache 2026-05-03 17:00:51 -05:00
docs docs(hrdps): record stage progress + dormant scaffolding state 2026-04-29 17:13:08 -05:00
k8s ops: prop deployment floor at 2 replicas, surge to 4 2026-05-03 15:06:37 -05:00
lib feat(weather): GridCache backend pluggable, Valkey for cross-pod cache 2026-05-03 17:00:51 -05:00
lib_ml feat(ml): add prop.compare task for algorithm-vs-ML-vs-reality calibration 2026-04-28 14:38:02 -05:00
notebooks refactor: normalize pos1/pos2 JSONB key to 'lon' everywhere 2026-04-17 09:10:32 -05:00
priv feat(rover-planning): multi-band missions + smart reconcile 2026-05-03 14:08:49 -05:00
rel Add Erlang clustering via libcluster and expose LiveDashboard 2026-04-06 09:46:48 -05:00
rust/prop_grid_rs fix(prop-grid): keep HRDPS lat=50 boundary row to close /weather seam 2026-04-30 16:44:53 -05:00
scripts chore: ignore python __pycache__ across the tree 2026-04-25 16:03:01 -05:00
test feat(rover-planning): worker pre-computes full /path output, PathLive renders cached 2026-05-03 14:58:16 -05:00
vendor deps: bump oban 2.22.0 → 2.22.1, oban_web (vendored) 2.12.1 → 2.12.3 2026-04-30 09:34:45 -05:00
.credo.exs credo 2026-04-12 10:26:53 -05:00
.dockerignore Optimize Dockerfile layer caching 2026-04-01 10:32:56 -05:00
.formatter.exs initial 2026-03-28 11:28:47 -05:00
.gitignore feat(ml): add prop.compare task for algorithm-vs-ML-vs-reality calibration 2026-04-28 14:38:02 -05:00
.tool-versions update deps 2026-04-11 12:03:43 -05:00
AGENTS.md initial 2026-03-28 11:28:47 -05:00
algo.md feat(weather): /weather-ca endpoint shows HRDPS-only Canadian data 2026-04-30 12:08:48 -05:00
app.json Migrate healthchecks from CHECKS to app.json 2026-04-04 11:24:56 -05:00
bugs.md fix: April 2026 codebase review — address 13 bugs across propagation chain 2026-04-25 10:52:51 -05:00
CLAUDE.md fix(about): per-table count fallback so missing tables don't blank everything 2026-05-03 14:21:53 -05:00
Dockerfile ci: switch app Dockerfile to FROM prop-base 2026-05-03 11:46:28 -05:00
Dockerfile.base ci: rebuild prop-base weekly to pull Debian security patches 2026-05-03 11:51:08 -05:00
elevation.md add elevation findings 2026-03-29 16:23:03 -05:00
LICENSE Add GPLv3 license 2026-04-23 13:57:07 -05:00
mix.exs feat(weather): GridCache backend pluggable, Valkey for cross-pod cache 2026-05-03 17:00:51 -05:00
mix.lock feat(weather): GridCache backend pluggable, Valkey for cross-pod cache 2026-05-03 17:00:51 -05:00
prediction.md HRRR forecast hours (f00-f18) with timeline map UI 2026-03-31 16:44:47 -05:00
README.md initial 2026-03-28 11:28:47 -05:00
tail_logs.sh Add type filter to backfill enqueue 2026-04-06 11:42:26 -05:00
updates.md Update seasonal scores from sounding data, document commercial link patterns 2026-04-07 11:45:44 -05:00

README.md

Microwaveprop

To start your Phoenix server:

  • Run mix setup to install and setup dependencies
  • Start Phoenix endpoint with mix phx.server or inside IEx with iex -S mix phx.server

Now you can visit localhost:4000 from your browser.

Ready to run in production? Please check our deployment guides.

Learn more