ops(prop): bump preStop sleep to 25s for cloudflared connection drain
The ingress is cloudflared (Cloudflare Tunnel), which keeps an HTTP/2 connection pool to upstream Service IPs and reuses connections aggressively. After a pod is removed from Service endpoints, cloudflared can still hold in-flight HTTP/2 streams to the dying pod for a noticeable window — 10 s wasn't enough, deploys still bounced 502 on LiveView reconnect. Bump preStop sleep 10s → 25s and terminationGracePeriodSeconds 40s → 60s so cloudflared has time to turn over its upstream connections before Phoenix begins draining.
This commit is contained in:
parent
45d8466808
commit
ed8b080b13
1 changed files with 17 additions and 15 deletions
|
|
@ -38,17 +38,20 @@ spec:
|
||||||
serviceAccountName: prop
|
serviceAccountName: prop
|
||||||
imagePullSecrets:
|
imagePullSecrets:
|
||||||
- name: forgejo-registry
|
- name: forgejo-registry
|
||||||
# The deploy was producing 502s on LiveView reconnect: kubelet sent
|
# The ingress here is cloudflared (Cloudflare Tunnel) which keeps
|
||||||
# SIGTERM to the old pod and Phoenix immediately stopped accepting
|
# an HTTP/2 connection pool to upstream Service IPs and reuses
|
||||||
# WebSocket connections, but the Service endpoint controller and
|
# connections aggressively. After a pod is removed from Service
|
||||||
# ingress hadn't yet propagated the pod's removal from endpoints.
|
# endpoints, cloudflared can still hold an in-flight HTTP/2 stream
|
||||||
# The browser's reconnect attempt would route to the dead pod and
|
# to the dying pod for a noticeable window — long enough that
|
||||||
# bounce 502.
|
# client LiveView reconnect attempts during a deploy land on the
|
||||||
|
# dead pod and bounce 502.
|
||||||
#
|
#
|
||||||
# 30s grace + a 10s preStop sleep on the container gives the
|
# The 25 s preStop sleep keeps the pod listening (and accepting
|
||||||
# endpoint controller a comfortable window to drain this pod from
|
# final requests) while Service endpoint propagation + cloudflared
|
||||||
# the load balancer before Phoenix starts shutting down.
|
# connection-pool turnover both complete. SIGTERM only fires after
|
||||||
terminationGracePeriodSeconds: 40
|
# that window, and Phoenix gets up to 60 s total before SIGKILL to
|
||||||
|
# finish draining open LiveView WebSockets.
|
||||||
|
terminationGracePeriodSeconds: 60
|
||||||
securityContext:
|
securityContext:
|
||||||
runAsUser: 65534
|
runAsUser: 65534
|
||||||
runAsNonRoot: true
|
runAsNonRoot: true
|
||||||
|
|
@ -111,14 +114,13 @@ spec:
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
failureThreshold: 3
|
failureThreshold: 3
|
||||||
timeoutSeconds: 5
|
timeoutSeconds: 5
|
||||||
# Sleep before SIGTERM hits Phoenix so the Service endpoint
|
# See terminationGracePeriodSeconds comment above — this sleep
|
||||||
# controller has time to remove this pod from the LB pool —
|
# bridges the cloudflared connection-pool turnover so deploys
|
||||||
# without it, in-flight LiveView reconnects raced the
|
# don't bounce 502 on LiveView reconnect.
|
||||||
# endpoint update and bounced 502 against the dying pod.
|
|
||||||
lifecycle:
|
lifecycle:
|
||||||
preStop:
|
preStop:
|
||||||
exec:
|
exec:
|
||||||
command: ["sh", "-c", "sleep 10"]
|
command: ["sh", "-c", "sleep 25"]
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
cpu: 100m
|
cpu: 100m
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue