External Prometheus at 10.0.15.31 already runs the kubernetes-pods job
in prometheus.yml.j2; pods need prometheus.io/scrape + port + path +
job annotations to be picked up. The prop-metrics NodePort selector
also tightened to tier=hot so backfill (PHX_SERVER=false, no :5000
listener) is excluded from the scrape pool.
The image tags here are placeholders — the live tag is set by the
ArgoCD Application's spec.source.kustomize.images field, same as
before. Only the registry hostname and path changed.
Pull secret 'forgejo-registry' in the prop namespace now carries
both git.mcintire.me and codeberg.org auths during the transition.
A single-pod floor meant any pod restart (rolling deploy mid-run,
OOM kill, node drain) took the whole site down for the boot window.
Bump HPA to minReplicas: 2 / maxReplicas: 4 so the floor is HA and
load surges (LiveView traffic coinciding with the hourly propagation
chain) still get headroom. Combined with the existing rolling-update
strategy (maxSurge: 1, maxUnavailable: 0) this means deploys go
2 -> 3 -> 2 with zero downtime.
The ingress is cloudflared (Cloudflare Tunnel), which keeps an HTTP/2
connection pool to upstream Service IPs and reuses connections
aggressively. After a pod is removed from Service endpoints,
cloudflared can still hold in-flight HTTP/2 streams to the dying pod
for a noticeable window — 10 s wasn't enough, deploys still bounced
502 on LiveView reconnect.
Bump preStop sleep 10s → 25s and terminationGracePeriodSeconds 40s →
60s so cloudflared has time to turn over its upstream connections
before Phoenix begins draining.
Deploys were causing 502 Bad Gateway on LiveView reconnect. The kubelet
sent SIGTERM to the old pod and Phoenix immediately stopped accepting
connections, but the Service endpoint controller and ingress hadn't
yet propagated the pod's removal from endpoints — the browser's
reconnect raced the endpoint update and bounced 502 against the dead
pod, then gave up.
Add a 10s preStop sleep on the container and bump
terminationGracePeriodSeconds to 40 so the load balancer has time to
drain this pod before Phoenix begins shutting down.
maxUnavailable: 1 was killing the only pod before the new one was
ready, taking the site down for the entire image pull + boot window
on every deploy. Switch to maxSurge: 1 / maxUnavailable: 0 so K8s
spins up the replacement and waits for its readinessProbe to pass
before tearing down the old pod.
