fix(k8s): tier label isolates hot pods from backfill endpoints

Both the hot (`prop`) Deployment and the `prop-backfill` Deployment
labelled their pods with `app: prop` so both joined the same libcluster
Erlang cluster. The `prop` Service selector was `app: prop` too — which
meant the backfill pod (PHX_SERVER=false, no listener on :5000) was in
the Service endpoint pool.

With sessionAffinity: ClientIP, Cloudflared's source IP got hashed to
the backfill endpoint after a backfill pod roll and every request
from that client returned 502 (connection refused on :5000).

Fix: add `tier: hot` to hot-pod template labels; narrow the Service
selector to `{app: prop, tier: hot}`. Backfill keeps `tier: backfill`
and is cleanly excluded. libcluster selector (still `app: prop`) is
unchanged so backfill stays in the Erlang cluster for Oban queue
leader election and Oban.Pro's Smart engine.
This commit is contained in:
Graham McIntire 2026-04-19 16:58:12 -05:00
parent 8ab2d836fb
commit 6c73012d09
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
2 changed files with 10 additions and 0 deletions

View file

@ -17,7 +17,12 @@ spec:
template:
metadata:
labels:
# `tier: hot` distinguishes these pods from prop-backfill (same
# app: prop label) so the Service selector can exclude backfill
# from its endpoint pool — backfill runs PHX_SERVER=false and
# doesn't listen on :5000.
app: prop
tier: hot
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane

View file

@ -11,6 +11,11 @@ spec:
timeoutSeconds: 3600
selector:
app: prop
# Exclude prop-backfill (tier: backfill) — it runs PHX_SERVER=false
# and doesn't listen on :5000. A pre-tier-label deploy caused a
# Cloudflare 502 when ClientIP affinity pinned a source to the
# backfill endpoint.
tier: hot
ports:
- name: http
port: 5000