Graham McIntire
a2701780ab
Investigated why all three prop pods were restart-looping every few
minutes (RESTARTS counts 7–11 in 99m). The trail:
20:53:33.058 [error] Postgrex.Protocol ("db_conn_10") disconnected:
client (:"Elixir.Microwaveprop.PromEx.Poller.5000") timed out
because it queued and checked out the connection for longer than
15000ms
PromEx's Oban queue-depth poller was scanning the oban_jobs table
every 5s and holding an Ecto connection past its checkout timeout.
That starved the rest of the pool, so /health's `SELECT 1` couldn't
acquire a connection inside the 3s liveness probe timeout, the
kubelet's failureThreshold=3 tripped, and SIGKILL landed on the
BEAM. Every replica cycled on the same cadence.
Two fixes:
1. Split /live (no DB) from /health (DB ping). Kubernetes livenessProbe
now points at /live so a saturated Ecto pool can never cascade into
SIGKILL. Readiness still hits /health so a genuine DB outage drains
the pod from Service endpoints gracefully.
2. Turn off the PromEx Oban plugin in prod via the same flag already
used in test. The queue-depth query isn't worth pod instability;
the oban_web dashboard surfaces the same information without
scanning the job table on a timer.
Locked both down with HealthController tests that verify /live never
touches the Repo (no sandbox owner, controller.live/2 called directly,
200 ok) and /health does (sanity query round-trips).
38 lines
1.3 KiB
Elixir
38 lines
1.3 KiB
Elixir
defmodule MicrowavepropWeb.HealthController do
|
|
@moduledoc """
|
|
Two separate probe endpoints for Kubernetes:
|
|
|
|
* `live/2` — liveness only. Returns 200 as long as the BEAM can
|
|
serve an HTTP request. Must NOT touch the database: a slow or
|
|
saturated Ecto pool would otherwise cascade into SIGKILL across
|
|
every replica simultaneously. Used as the `livenessProbe`
|
|
target.
|
|
|
|
* `check/2` — readiness. Also pings the Repo via `SELECT 1` so
|
|
the pod is taken out of the Service endpoint set when the DB
|
|
is unreachable. Used as the `readinessProbe` target.
|
|
|
|
Splitting these two concerns is specifically a fix for an incident
|
|
where PromEx's Oban queue-depth poller held a connection for > 15s,
|
|
queued up every other checkout, and the liveness probe's `SELECT 1`
|
|
timed out before it could ever acquire the pool — so the kubelet
|
|
killed every pod every few minutes even though the BEAM was healthy.
|
|
"""
|
|
use MicrowavepropWeb, :controller
|
|
|
|
alias Ecto.Adapters.SQL
|
|
|
|
def live(conn, _params) do
|
|
conn
|
|
|> put_resp_content_type("text/plain")
|
|
|> send_resp(200, "ok")
|
|
end
|
|
|
|
def check(conn, _params) do
|
|
_ = SQL.query!(Microwaveprop.Repo, "SELECT 1")
|
|
|
|
conn
|
|
|> put_resp_content_type("text/plain")
|
|
|> send_resp(200, "ok")
|
|
end
|
|
end
|