A single-pod floor meant any pod restart (rolling deploy mid-run,
OOM kill, node drain) took the whole site down for the boot window.
Bump HPA to minReplicas: 2 / maxReplicas: 4 so the floor is HA and
load surges (LiveView traffic coinciding with the hourly propagation
chain) still get headroom. Combined with the existing rolling-update
strategy (maxSurge: 1, maxUnavailable: 0) this means deploys go
2 -> 3 -> 2 with zero downtime.
HPA: prop-grid-rs and hrrr-point-rs were sitting at 4 pods for
~16 min after the hourly chain spike — 10-min scaleDown stabilization
window plus a 1-pod-per-2-min drain policy. Both deployments stayed
visibly over-provisioned even when CPU dropped to 1m/pod.
Tighten scale-down to: 2-min stabilization (long enough that a real
multi-cycle surge doesn't immediately collapse) + Percent: 100 per
60s drain (back to min=1 within 1 minute of stabilization clearing).
Net: spike → scale up to handle queue (unchanged) → back to 1 pod
within ~3 min of spike ending. scaleUp left untouched.
CLAUDE.md updates so this doesn't bite again:
- Document `cargo clippy --all-targets -- -D warnings` as part of
the local Rust precommit ritual. The HRDPS commits in this branch
passed `cargo build` locally but failed CI on lints (manual_range_
contains, iter_kv_map). Adding the recipe under Commands plus a
Rust subsection in Key Conventions so future Rust touches don't
ship clippy regressions.
- Added a Testing note about not anchoring relative-time fixtures on
utc_now-truncated-to-hour — the weather_map_live one-hour-cutoff
test (fix in 2768fc68) was a textbook example: 50% flake rate
driven entirely by which minute of the hour the test ran in.
The ingress is cloudflared (Cloudflare Tunnel), which keeps an HTTP/2
connection pool to upstream Service IPs and reuses connections
aggressively. After a pod is removed from Service endpoints,
cloudflared can still hold in-flight HTTP/2 streams to the dying pod
for a noticeable window — 10 s wasn't enough, deploys still bounced
502 on LiveView reconnect.
Bump preStop sleep 10s → 25s and terminationGracePeriodSeconds 40s →
60s so cloudflared has time to turn over its upstream connections
before Phoenix begins draining.
Deploys were causing 502 Bad Gateway on LiveView reconnect. The kubelet
sent SIGTERM to the old pod and Phoenix immediately stopped accepting
connections, but the Service endpoint controller and ingress hadn't
yet propagated the pod's removal from endpoints — the browser's
reconnect raced the endpoint update and bounced 502 against the dead
pod, then gave up.
Add a 10s preStop sleep on the container and bump
terminationGracePeriodSeconds to 40 so the load balancer has time to
drain this pod before Phoenix begins shutting down.
maxUnavailable: 1 was killing the only pod before the new one was
ready, taking the site down for the entire image pull + boot window
on every deploy. Switch to maxSurge: 1 / maxUnavailable: 0 so K8s
spins up the replacement and waits for its readinessProbe to pass
before tearing down the old pod.
Set minReplicas == maxReplicas == 1 on the `prop` HPA so the cluster
runs exactly one app pod. HPA still owns the replica count (Flux
ignores spec.replicas on the Deployment), so this is the right knob —
no need to touch deployment.yaml. Restore autoscaling by raising both
bounds together.
