fix: isolate expensive health checks behind limiter #23
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/w6-health-limiter"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Fixes W6 from bugs.md.
HealthCheckplug now answers only/liveand/startupwith allocation-free liveness — no DB pool checkout, no PubSub broadcast, no GenServer call/healthand/health/dbmoved behind the router ontoPageController.healthwith a dedicated per-IP limiter (30/min), well above K8s probe cadenceShutdownHandler.shutting_down?/0now reads a:persistent_termflag written at boot and at shutdown initiation, removing the 5 s serialisingGenServer.callfrom every probeNote:
/healthnow returns JSON instead of the previous text/plain since it is served by the controller.Verification: focused tests passed (26), full suite passed (2225),
mix credo --strict, and commit hooks including Dialyzer passed.🤖 Skippy PR review
2 findings — 1 blocking before merge.
lib/aprsme/shutdown_handler.ex:153lib/aprsme_web/router.ex:108Reviewed
336179cd88a0. Commentskippy reviewto re-run.@ -148,6 +150,7 @@ defmodule Aprsme.ShutdownHandler dodefp initiate_shutdown(state) do# Mark as shutting down:persistent_term.put(@shutting_down_key, true)🟡 Warning — New global flag is never cleared, so it leaks across tests
shutting_down?/0is now backed by a process-global:persistent_termthat onlyinit/1ever writes back tofalse. Two cleanups exist in the suite precisely because a test drives a real shutdown —test/aprsme/signal_handler_test.exs:31-34(SIGTERM -> the realShutdownHandler.shutdown()) and the setup attest/aprsme_web/controllers/page_controller_test.exs:7-12— but both only resetstate.shutting_downand:health_status, so they no longer clear the state health checks read. Once that module runs,shutting_down?()staystruefor the rest of the VM and later modules fail depending on order:GET /ready200 andPageController.health/2200 (page_controller_test.exs:22,46),GET /health200 (health_check_test.exs:55), andshutting_down?() == false(shutdown_handler_test.exs:30,108). Reset the term too —:persistent_term.erase(@shutting_down_key)(or aShutdownHandler.reset/0) in both cleanups — rather than only the GenServer state.@ -98,3 +105,4 @@get "/status.json", PageController, :status_jsonend# Full readiness (database + PubSub). More expensive than /ready, so it🔵 Suggestion — Comment claims a PubSub readiness check that no longer exists
This says "database + PubSub", and the pipeline comment above (line 59) says "hit the DB pool and PubSub", but the old plug's
Phoenix.PubSub.broadcast/3ping was deleted in this PR andPageController.health/2only runsRepo.query!("SELECT 1"). Nothing in the app now fails readiness when PubSub is dead. Either restore the probe or drop PubSub from both comments so the next person reading the gate isn't misled.🤖 Skippy PR review
1 finding — 1 blocking before merge.
test/aprsme_web/controllers/page_controller_test.exs:13Resolved 1 earlier finding (the stale PubSub readiness comments in router.ex). 1 still open, plus 1 new: the persistent_term reset went to a module that only reads the flag.
Reviewed
1ee986179b92. Commentskippy reviewto re-run.@ -9,2 +10,4 @@# directly, and it outlives the GenServer state and the Application env.if pid = Process.whereis(Aprsme.ShutdownHandler) do:sys.replace_state(pid, fn state -> %{state | shutting_down: false} end):persistent_term.put({Aprsme.ShutdownHandler, :shutting_down}, false)🟡 Warning — The flag reset landed on a reader; the module that sets it is still unpatched
This clears the flag in
PageControllerTest's setup, but this module only reads it. The site that sets it istest/aprsme/signal_handler_test.exs:31-34:SignalHandler.handle_info({:signal, :sigterm}, ...)spawn_links the realShutdownHandler.shutdown()(lib/aprsme/signal_handler.ex:28-29), whoseinitiate_shutdown/1puts the termtrue, and that cleanup only restoresstate.shutting_downand:health_status.Aprsme.ShutdownHandleris started in every env (lib/aprsme/application.ex:67-70), so once that module runs the VM keepsshutting_down?() == trueand the order-dependent failures my earlier comment listed survive in the modules this commit did not touch:test/aprsme/shutdown_handler_test.exs:30and:108(shutting_down?() == false) andtest/aprsme_web/plugs/health_check_test.exs:54-58(GET /healthexpects 200, gets 503shutting_down).Fix it where the flag is written (add
ShutdownHandler.reset/0and call it from that test's cleanup, or:persistent_term.erase(@shutting_down_key)), and make the reset here unconditional rather than nested underif Process.whereis(...).1ee986179bcfad9e201a🤖 Skippy PR review
2 findings — 1 blocking before merge.
lib/aprsme_web/router.ex:113test/aprsme_web/controllers/page_controller_test.exs:24Findings outside the commentable diff
test/aprsme_web/controllers/page_controller_test.exs:24— Stale comment: the plug no longer serves /healthThis says the
/healthendpoint is otherwise served by theHealthCheckplug as text/plain. That stopped being true in this PR: the plug passes/healththrough (your ownhealth_check_test.exsasserts exactly that) andPageController.health/2is what /health reaches. One-line fix so the next reader is not sent looking for liveness logic in the plug.Resolved 2 earlier findings: the shutdown
:persistent_termflag is now cleared in each cleanup that sets it. 0 still open. Bookkeeping:newdifffell back to the full PR diff (the previous head is no longer in PR history).Reviewed
cfad9e201ade. Commentskippy reviewto re-run.@ -101,0 +110,4 @@# only cheap liveness for /live and /startup.scope "/", AprsmeWeb dopipe_through [:accepts_json, :health_limiter]get "/health", PageController, :health🟠 High — Branch is 12 commits behind main; the merge conflicts here and can restore the public /metrics route
This branch is based on
0d034ae6, 12 commits behindmain:#11(SIGTERM routed through a:gen_eventhandler) and#12(PromEx moved to a pod-only listener on 4001, public/metricsscope deleted) both landed after it. Merging conflicts in this file and intest/aprsme/signal_handler_test.exs(git merge-tree origin/main cfad9e20reports content conflicts in both; the PR itself reportsmergeable: false). Your new health scope lands exactly where#12deleted the/metricsblock, so a resolution that keeps this branch's side of that hunk re-addsforward "/", PromEx.Plugto the public router and undoes#12. Rebase ontomainbefore resolving, and when you re-apply it, put the:persistent_termreset into#11's rewrittensignal_handler_test.exscleanup or the test leak from the earlier thread returns.cfad9e201aa974259277🤖 Skippy PR review
2 findings — 1 blocking before merge.
lib/aprsme_web/router.ex:123test/aprsme/shutdown_handler_test.exs:26Findings outside the commentable diff
test/aprsme/shutdown_handler_test.exs:26— Two comments still explain the rescue clause this PR deletedshutting_down?/0no longer calls the GenServer or rescues anything — it reads:persistent_term— but this describe block still justifies the assertion with "If theGenServer.callraises, therescue _ -> falsekicks in", and theshutdown/0 as an external APItest (~line 120) still saysshutting_down?/0"is safe to call — it rescues when the GenServer isn't running". The "without the GenServer running" framing is moot too: the read now works with no process at all. Two comment edits so the next reader isn't sent looking for a rescue clause that no longer exists.Follow-up on
cfad9e20..a9742592. Resolved 1 earlier finding: the branch is rebased ontomain(04c1bc00), so the merge conflict is gone and the public/metricsroute stays deleted. 1 earlier suggestion (stale PubSub comments) was resolved in an earlier round. Bookkeeping:newdifffell back to the full PR diff — the previous head is no longer in history — so this review covers the whole 7-file PR against the current base (75 files/+2100 in the webhook payload is the stale pre-rebase figure; Forgejo now reports 7 files, +106/-374).Reviewed
a9742592770d. Commentskippy reviewto re-run.@ -111,0 +120,4 @@# only cheap liveness for /live and /startup.scope "/", AprsmeWeb dopipe_through [:accepts_json, :health_limiter]get "/health", PageController, :health🟡 Warning — The new /health route drops the 1 s bound the plug put on the DB probe
This line makes
PageController.health/2the production/healthhandler, but that action's DB check isAprsme.Repo.query!("SELECT 1")(lib/aprsme_web/controllers/page_controller.ex:60) with no timeout, so it inherits the repo's:timeout—DB_TIMEOUT, default 30_000 ms inconfig/runtime.exs. The plug this PR deletes capped the same probe attimeout: 1000.A slow-but-alive Postgres/PgBouncer is exactly when
/healthgets polled hardest, and each poll now holds a pool connection for up to 30 s instead of 1 s.POOL_SIZEis 25 ink8s/deployment.yaml, so 25 concurrent/healthrequests from a single IP — well inside the new 30/min limiter, and the endpoint is public and anonymous — can pin the whole pool and stall packet writes. That is the W6 symptom this PR is fixing.Keep the bound at the call site:
Aprsme.Repo.query!("SELECT 1", [], timeout: 1_000); the existingrescuealready turns the checkout timeout into a 503.🤖 Skippy PR review
Clean on
d8c7bb0c40bf— nothing new in921cfa5f..d8c7bb0c.Resolved 1 of 1 earlier findings: the
/healthDB probe is now bounded at the call site (Aprsme.Repo.query!("SELECT 1", [], timeout: 1_000)inlib/aprsme_web/controllers/page_controller.ex:63), matching thetimeout: 1000the deleted endpoint-level plug applied, so a slow Postgres/PgBouncer no longer holds a pool connection for the full 30 sDB_TIMEOUTper poll. Therescuestill turns that into a 503.Also addressed, both suggestions from earlier rounds: the stale comments on
shutting_down?/0intest/aprsme/shutdown_handler_test.exsare gone, and the:persistent_termreset intest/aprsme_web/controllers/page_controller_test.exsis now unconditional instead of nested underif Process.whereis(...).0 findings open. Bookkeeping:
newdiff --since 921cfa5freturned the incremental diff (no[FULL-DIFF-FALLBACK]), so this round covers onlyd8c7bb0c.Reviewed
d8c7bb0c40bf. Commentskippy reviewto re-run.