Kubernetes keeps sending traffic to a pod whose database connection is dead #50

New issue

Open

opened 2026-11-03 18:52:27 +00:00 by skippy-bot · 0 comments

skippy-bot commented

2026-11-03 18:52:27 +00:00

Problem

The readiness and startup probes both point at /ready, which only looks at the drain flag. With Postgres unreachable or the Ecto pool exhausted, every pod still reports Ready, Endpoints stays fully populated, users get 500s on every page, and a rolling update completes over the top of pods that cannot serve. The database-backed check already exists at /health and /health/db, and no probe references it.

Evidence

At 2f150f66:

k8s/deployment.yaml:254 is path: /ready for the readiness probe and :239 is path: /startup for the only gate before a new pod takes traffic.
lib/aprsme_web/router.ex:131 routes /ready and :139 routes /health.
lib/aprsme_web/controllers/page_controller.ex:121 comments Cheap readiness check without database dependency, and :133 is the only non-drain condition before it answers ready.
lib/aprsme_web/controllers/page_controller.ex:63 is _ = Aprsme.Repo.query!("SELECT 1", [], timeout: 1_000), reached only from /health.

Expected

Readiness answers 503 when the application cannot reach the database, so the pod leaves the Endpoints list and a rollout stalls instead of completing. Either point the readiness and startup probes at /health, or make /ready run the same bounded query while keeping the drain check.

How to verify

Stop Postgres or block egress from a pod: kubectl -n aprs get endpoints stays populated while curl pod:4000/ready returns 200 and /health returns 503. Cheaper: a controller test stubs Repo.query! to raise and asserts /ready is 503.

Acceptance criteria

A pod that cannot reach the database fails its readiness probe.
The startup probe does not admit a pod that has never reached the database.
The probe path is cheap enough to run every 5 seconds and is bounded by a timeout.
A test pins the 503 on a failed database check.

Effort: S
Lens: ops

## Problem The readiness and startup probes both point at `/ready`, which only looks at the drain flag. With Postgres unreachable or the Ecto pool exhausted, every pod still reports Ready, Endpoints stays fully populated, users get 500s on every page, and a rolling update completes over the top of pods that cannot serve. The database-backed check already exists at `/health` and `/health/db`, and no probe references it. ## Evidence At `2f150f66`: - `k8s/deployment.yaml:254` is `path: /ready` for the readiness probe and `:239` is `path: /startup` for the only gate before a new pod takes traffic. - `lib/aprsme_web/router.ex:131` routes `/ready` and `:139` routes `/health`. - `lib/aprsme_web/controllers/page_controller.ex:121` comments `Cheap readiness check without database dependency`, and `:133` is the only non-drain condition before it answers `ready`. - `lib/aprsme_web/controllers/page_controller.ex:63` is `_ = Aprsme.Repo.query!("SELECT 1", [], timeout: 1_000)`, reached only from `/health`. ## Expected Readiness answers 503 when the application cannot reach the database, so the pod leaves the Endpoints list and a rollout stalls instead of completing. Either point the readiness and startup probes at `/health`, or make `/ready` run the same bounded query while keeping the drain check. ## How to verify Stop Postgres or block egress from a pod: `kubectl -n aprs get endpoints` stays populated while `curl pod:4000/ready` returns 200 and `/health` returns 503. Cheaper: a controller test stubs `Repo.query!` to raise and asserts `/ready` is 503. ## Acceptance criteria - [ ] A pod that cannot reach the database fails its readiness probe. - [ ] The startup probe does not admit a pod that has never reached the database. - [ ] The probe path is cheap enough to run every 5 seconds and is bounded by a timeout. - [ ] A test pins the 503 on a failed database check. **Effort:** S **Lens:** ops

skippy-bot added the

Medium

bug

effort: S

labels

2026-11-03 18:52:27 +00:00