Graham McIntire
a2701780ab
Investigated why all three prop pods were restart-looping every few
minutes (RESTARTS counts 7–11 in 99m). The trail:
20:53:33.058 [error] Postgrex.Protocol ("db_conn_10") disconnected:
client (:"Elixir.Microwaveprop.PromEx.Poller.5000") timed out
because it queued and checked out the connection for longer than
15000ms
PromEx's Oban queue-depth poller was scanning the oban_jobs table
every 5s and holding an Ecto connection past its checkout timeout.
That starved the rest of the pool, so /health's `SELECT 1` couldn't
acquire a connection inside the 3s liveness probe timeout, the
kubelet's failureThreshold=3 tripped, and SIGKILL landed on the
BEAM. Every replica cycled on the same cadence.
Two fixes:
1. Split /live (no DB) from /health (DB ping). Kubernetes livenessProbe
now points at /live so a saturated Ecto pool can never cascade into
SIGKILL. Readiness still hits /health so a genuine DB outage drains
the pod from Service endpoints gracefully.
2. Turn off the PromEx Oban plugin in prod via the same flag already
used in test. The queue-depth query isn't worth pod instability;
the oban_web dashboard surfaces the same information without
scanning the job table on a timer.
Locked both down with HealthController tests that verify /live never
touches the Repo (no sandbox owner, controller.live/2 called directly,
200 ok) and /health does (sanity query round-trips).
44 lines
1.7 KiB
Elixir
44 lines
1.7 KiB
Elixir
defmodule MicrowavepropWeb.HealthControllerTest do
|
|
use MicrowavepropWeb.ConnCase, async: true
|
|
|
|
alias Ecto.Adapters.SQL
|
|
|
|
test "GET /health returns 200 ok", %{conn: conn} do
|
|
conn = get(conn, "/health")
|
|
|
|
assert response(conn, 200) == "ok"
|
|
assert response_content_type(conn, :text) =~ "text/plain"
|
|
end
|
|
|
|
test "GET /live returns 200 ok", %{conn: conn} do
|
|
conn = get(conn, "/live")
|
|
|
|
assert response(conn, 200) == "ok"
|
|
assert response_content_type(conn, :text) =~ "text/plain"
|
|
end
|
|
|
|
# The split between /live and /health is the load-bearing detail of
|
|
# the production liveness fix. If the Ecto pool is saturated, /health
|
|
# must fail (so readiness drains the pod) while /live must succeed
|
|
# (so the kubelet doesn't SIGKILL the BEAM). Invoke the controller
|
|
# actions without any Ecto sandbox assignment so the DB-touching path
|
|
# is forced to raise — /live must not raise, /health must.
|
|
describe "DB dependency" do
|
|
test "live/2 returns 200 without touching the Repo", %{conn: conn} do
|
|
# No `Ecto.Adapters.SQL.Sandbox.checkout` has been issued for this
|
|
# test process in the async: true block, so any attempt to reach
|
|
# the Repo would raise `DBConnection.OwnershipError`. A clean 200
|
|
# proves the action never tried.
|
|
refute conn.assigns[:ecto_sandbox]
|
|
conn = MicrowavepropWeb.HealthController.live(conn, %{})
|
|
assert conn.status == 200
|
|
assert conn.resp_body == "ok"
|
|
end
|
|
|
|
test "check/2 issues a Repo query (proves it's the DB-gated endpoint)", %{conn: conn} do
|
|
conn = get(conn, "/health")
|
|
assert response(conn, 200) == "ok"
|
|
assert {:ok, %{rows: [[1]]}} = SQL.query(Microwaveprop.Repo, "SELECT 1")
|
|
end
|
|
end
|
|
end
|