Root cause of the deadlock prod hit today:
- prop-backfill's podAntiAffinity matched `app: prop`, which applies to
backfill pods themselves as well as hot pods.
- The hot Deployment runs maxSurge=1/maxUnavailable=1 with HPA min 2 /
max 5; when HPA was pegged at max and a rollout began, the old
ReplicaSet's requiredDuringScheduling/IgnoredDuringExecution rule
stayed pinned on every already-running hot pod.
- backfill's rule then excluded new hot pods from every physical host
that had an app=prop pod on it, which was all of them, and the
rollout couldn't progress because no old hot pod could come down
until a new one went Ready first.
Narrowing the selector to `{app: prop, tier: hot}` means:
- backfill never constrains itself (it carries tier=backfill).
- Hot pods carry no anti-affinity of their own, so HPA can continue
past the 4-physical-host count by co-locating when it has to.
- Backfill still lands on whichever physical host hot pods haven't
filled up, which is the original intent.
Graham McIntire
0bc8af22dd