Graham McIntire
30a0b9a3a0
Fixes race condition where Valkey pod tries to start before Flannel CNI
is fully initialized during node restarts.
Problem:
- Valkey pod had 24 restarts due to FailedCreatePodSandBox errors
- During node restarts, pods would start before Flannel wrote /run/flannel/subnet.env
- CNI plugin would fail: 'failed to load flannel subnet.env file'
- Pod would retry until Flannel finished initialization
Solution:
- Added priorityClassName: system-cluster-critical to Valkey StatefulSet
- This gives Valkey same priority level as etcd, coredns, and other cluster services
- Scheduler will ensure CNI (Flannel) is ready before starting Valkey
- Reduces race condition window during node restarts
Priority Classes:
- system-node-critical (2000001000): Flannel, kube-proxy
- system-cluster-critical (2000000000): Valkey, etcd, coredns
- default (0): Regular application pods
Combined with previous commit's Redis health checks and Exq supervisor
improvements, the application is now resilient to both infrastructure
issues and Redis connection failures.
🤖 Generated with Claude Code
|
||
|---|---|---|
| .. | ||
| certificate.yaml | ||
| deployment.yaml | ||
| Dockerfile | ||
| ingressroute.yaml | ||
| kustomization.yaml | ||
| namespace.yaml | ||
| poddisruptionbudget.yaml | ||
| README.md | ||
| service-headless.yaml | ||
| service.yaml | ||
| valkey-service.yaml | ||
| valkey-statefulset.yaml | ||
Kubernetes Deployment
Secrets Management
Secrets are managed directly in the cluster and must be created before deploying the application.
Required secrets in the towerops namespace:
gitlab-registry- Docker registry credentials for pulling imagestowerops-secrets- Application secrets (RELEASE_COOKIE, SECRET_KEY_BASE)towerops-db- Database connection credentialstowerops-aws- AWS credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION)
For local development, the project root .envrc is used by direnv.
Deployment Timestamp
The application footer displays the deployment timestamp to track when the current version was deployed. This is automatically set by GitLab CI during deployment:
# GitLab CI sets this during deploy
- kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
All pods in the deployment share the same timestamp (when the deployment was initiated), regardless of when individual pods were created. This is displayed in the footer as "Last deployed X ago · YYYY-MM-DD HH:MM:SS UTC".
For manual deployments without GitLab CI, set the timestamp:
kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
Deploying
Apply all resources using kustomize:
kubectl apply -k k8s/
Or individually:
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/secret.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/service-headless.yaml
kubectl apply -f k8s/certificate.yaml
kubectl apply -f k8s/ingressroute.yaml