graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
1333 commits 1 branch 0 tags 8.1 MiB
Find a file
Graham McIntire a2701780ab
fix(probes): stop DB saturation from SIGKILLing every pod 
Investigated why all three prop pods were restart-looping every few
minutes (RESTARTS counts 7–11 in 99m). The trail:

  20:53:33.058 [error] Postgrex.Protocol ("db_conn_10") disconnected:
    client (:"Elixir.Microwaveprop.PromEx.Poller.5000") timed out
    because it queued and checked out the connection for longer than
    15000ms

PromEx's Oban queue-depth poller was scanning the oban_jobs table
every 5s and holding an Ecto connection past its checkout timeout.
That starved the rest of the pool, so /health's `SELECT 1` couldn't
acquire a connection inside the 3s liveness probe timeout, the
kubelet's failureThreshold=3 tripped, and SIGKILL landed on the
BEAM. Every replica cycled on the same cadence.

Two fixes:

1. Split /live (no DB) from /health (DB ping). Kubernetes livenessProbe
   now points at /live so a saturated Ecto pool can never cascade into
   SIGKILL. Readiness still hits /health so a genuine DB outage drains
   the pod from Service endpoints gracefully.

2. Turn off the PromEx Oban plugin in prod via the same flag already
   used in test. The queue-depth query isn't worth pod instability;
   the oban_web dashboard surfaces the same information without
   scanning the job table on a timer.

Locked both down with HealthController tests that verify /live never
touches the Repo (no sandbox owner, controller.live/2 called directly,
200 ok) and /health does (sanity query round-trips).
2026-04-21 16:00:04 -05:00
.forgejo/workflows ci(grid-rs): surface docker daemon diagnostics on wait-step failure 2026-04-20 08:41:53 -05:00
.github Add contact edit approval system with admin review queue 2026-04-11 16:15:49 -05:00
assets feat(map): restore point-detail panel across navigation 2026-04-20 14:54:44 -05:00
bin Fix livebook path resolution in bin/notebook 2026-04-07 11:15:49 -05:00
config fix(probes): stop DB saturation from SIGKILLing every pod 2026-04-21 16:00:04 -05:00
docs refactor(scores): rename file extension .ntms -> .prop with legacy reads 2026-04-21 15:54:12 -05:00
k8s fix(probes): stop DB saturation from SIGKILLing every pod 2026-04-21 16:00:04 -05:00
lib fix(probes): stop DB saturation from SIGKILLing every pod 2026-04-21 16:00:04 -05:00
lib_ml fix(dialyzer): clear 125+ warnings under strict flags 2026-04-21 10:30:06 -05:00
notebooks refactor: normalize pos1/pos2 JSONB key to 'lon' everywhere 2026-04-17 09:10:32 -05:00
priv perf(db): add (valid_time, lat, lon) composite index on hrrr[_native]_profiles 2026-04-21 13:20:22 -05:00
rel Add Erlang clustering via libcluster and expose LiveDashboard 2026-04-06 09:46:48 -05:00
rust/prop_grid_rs refactor(scores): rename file extension .ntms -> .prop with legacy reads 2026-04-21 15:54:12 -05:00
scripts feat(propagation): per-band weight calibration from full-corpus correlation 2026-04-18 10:19:26 -05:00
test fix(probes): stop DB saturation from SIGKILLing every pod 2026-04-21 16:00:04 -05:00
vendor Vendor oban_pro 1.6.13 2026-04-09 14:14:49 -05:00
.credo.exs credo 2026-04-12 10:26:53 -05:00
.dockerignore Optimize Dockerfile layer caching 2026-04-01 10:32:56 -05:00
.formatter.exs initial 2026-03-28 11:28:47 -05:00
.gitignore Dual-write propagation scores to binary files on /data 2026-04-14 14:36:55 -05:00
.tool-versions update deps 2026-04-11 12:03:43 -05:00
AGENTS.md initial 2026-03-28 11:28:47 -05:00
algo.md docs(algo): align algo.md with Richardson-gated scoring implementation 2026-04-18 12:42:20 -05:00
app.json Migrate healthchecks from CHECKS to app.json 2026-04-04 11:24:56 -05:00
CLAUDE.md fix(maps): same reconnect/visibility fix for rover and weather hooks 2026-04-18 13:25:38 -05:00
Dockerfile fix(docker): use BuildKit cache mounts for apt so .debs skip the layer 2026-04-18 11:07:45 -05:00
elevation.md add elevation findings 2026-03-29 16:23:03 -05:00
mix.exs feat(logging): structured JSON logs in prod via LOG_FORMAT env var 2026-04-21 14:17:54 -05:00
mix.lock feat(logging): structured JSON logs in prod via LOG_FORMAT env var 2026-04-21 14:17:54 -05:00
prediction.md HRRR forecast hours (f00-f18) with timeline map UI 2026-03-31 16:44:47 -05:00
README.md initial 2026-03-28 11:28:47 -05:00
tail_logs.sh Add type filter to backfill enqueue 2026-04-06 11:42:26 -05:00
updates.md Update seasonal scores from sounding data, document commercial link patterns 2026-04-07 11:45:44 -05:00

README.md

Microwaveprop

To start your Phoenix server:

  • Run mix setup to install and setup dependencies
  • Start Phoenix endpoint with mix phx.server or inside IEx with iex -S mix phx.server

Now you can visit localhost:4000 from your browser.

Ready to run in production? Please check our deployment guides.

Learn more