ops(prop): bump preStop sleep to 25s for cloudflared connection drain

The ingress is cloudflared (Cloudflare Tunnel), which keeps an HTTP/2
connection pool to upstream Service IPs and reuses connections
aggressively. After a pod is removed from Service endpoints,
cloudflared can still hold in-flight HTTP/2 streams to the dying pod
for a noticeable window — 10 s wasn't enough, deploys still bounced
502 on LiveView reconnect.

Bump preStop sleep 10s → 25s and terminationGracePeriodSeconds 40s →
60s so cloudflared has time to turn over its upstream connections
before Phoenix begins draining.
This commit is contained in:
Graham McIntire 2026-04-29 14:17:40 -05:00
parent 45d8466808
commit ed8b080b13
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59

View file

@ -38,17 +38,20 @@ spec:
serviceAccountName: prop serviceAccountName: prop
imagePullSecrets: imagePullSecrets:
- name: forgejo-registry - name: forgejo-registry
# The deploy was producing 502s on LiveView reconnect: kubelet sent # The ingress here is cloudflared (Cloudflare Tunnel) which keeps
# SIGTERM to the old pod and Phoenix immediately stopped accepting # an HTTP/2 connection pool to upstream Service IPs and reuses
# WebSocket connections, but the Service endpoint controller and # connections aggressively. After a pod is removed from Service
# ingress hadn't yet propagated the pod's removal from endpoints. # endpoints, cloudflared can still hold an in-flight HTTP/2 stream
# The browser's reconnect attempt would route to the dead pod and # to the dying pod for a noticeable window — long enough that
# bounce 502. # client LiveView reconnect attempts during a deploy land on the
# dead pod and bounce 502.
# #
# 30s grace + a 10s preStop sleep on the container gives the # The 25 s preStop sleep keeps the pod listening (and accepting
# endpoint controller a comfortable window to drain this pod from # final requests) while Service endpoint propagation + cloudflared
# the load balancer before Phoenix starts shutting down. # connection-pool turnover both complete. SIGTERM only fires after
terminationGracePeriodSeconds: 40 # that window, and Phoenix gets up to 60 s total before SIGKILL to
# finish draining open LiveView WebSockets.
terminationGracePeriodSeconds: 60
securityContext: securityContext:
runAsUser: 65534 runAsUser: 65534
runAsNonRoot: true runAsNonRoot: true
@ -111,14 +114,13 @@ spec:
periodSeconds: 10 periodSeconds: 10
failureThreshold: 3 failureThreshold: 3
timeoutSeconds: 5 timeoutSeconds: 5
# Sleep before SIGTERM hits Phoenix so the Service endpoint # See terminationGracePeriodSeconds comment above — this sleep
# controller has time to remove this pod from the LB pool — # bridges the cloudflared connection-pool turnover so deploys
# without it, in-flight LiveView reconnects raced the # don't bounce 502 on LiveView reconnect.
# endpoint update and bounced 502 against the dying pod.
lifecycle: lifecycle:
preStop: preStop:
exec: exec:
command: ["sh", "-c", "sleep 10"] command: ["sh", "-c", "sleep 25"]
resources: resources:
requests: requests:
cpu: 100m cpu: 100m