Graham McIntire
5ee241fc16
Addresses production Redis disconnection issues by implementing a highly
resilient Valkey (Redis) setup with automatic failover capabilities.
Infrastructure Changes:
- Add Valkey ConfigMap with optimized connection and memory settings
- TCP keepalive (60s), connection limits (10k clients)
- Memory management (256MB with LRU eviction)
- Separate configs for master, replica, and sentinel
- Update Valkey StatefulSet to 3 replicas (1 master + 2 replicas)
- Auto-configuration via init container (master vs replica)
- Increased memory limits: 256Mi → 1Gi
- Improved readiness probes with replication status checks
- Add Valkey Sentinel StatefulSet (3 instances for quorum)
- Automatic failover detection (5s down-after-milliseconds)
- Fast failover execution (10s timeout)
- Monitors master and promotes replicas automatically
- Add Sentinel headless service for pod discovery
Application Resilience:
- Update Phoenix.PubSub.Redis with TCP keepalive and reconnection
- Connection timeout: 5s
- Exponential backoff: 500ms → 30s
- exit_on_disconnection: false
- Update Exq background jobs with same resilience settings
- TCP keepalive enabled
- Better connection pool management
Benefits:
- Automatic failover when Valkey pod dies (no manual intervention)
- Zero data loss with replica synchronization
- Fast failure detection and recovery (5-10s total)
- Survives Flannel CNI networking issues
- Apps reconnect automatically during disconnections
Testing:
- All 3,686 tests passing
- Kustomize manifest validated
🤖 Generated with Claude Code
|
||
|---|---|---|
| .. | ||
| certificate.yaml | ||
| deployment.yaml | ||
| Dockerfile | ||
| etcd-statefulset.yaml | ||
| ingressroute.yaml | ||
| kustomization.yaml | ||
| namespace.yaml | ||
| poddisruptionbudget.yaml | ||
| README.md | ||
| service-headless.yaml | ||
| service.yaml | ||
| valkey-configmap.yaml | ||
| valkey-sentinel-service.yaml | ||
| valkey-sentinel-statefulset.yaml | ||
| valkey-service.yaml | ||
| valkey-statefulset.yaml | ||
Kubernetes Deployment
Secrets Management
Secrets are managed directly in the cluster and must be created before deploying the application.
Required secrets in the towerops namespace:
gitlab-registry- Docker registry credentials for pulling imagestowerops-secrets- Application secrets (RELEASE_COOKIE, SECRET_KEY_BASE)towerops-db- Database connection credentialstowerops-aws- AWS credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION)
For local development, the project root .envrc is used by direnv.
Deployment Timestamp
The application footer displays the deployment timestamp to track when the current version was deployed. This is automatically set by GitLab CI during deployment:
# GitLab CI sets this during deploy
- kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
All pods in the deployment share the same timestamp (when the deployment was initiated), regardless of when individual pods were created. This is displayed in the footer as "Last deployed X ago · YYYY-MM-DD HH:MM:SS UTC".
For manual deployments without GitLab CI, set the timestamp:
kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
Deploying
Apply all resources using kustomize:
kubectl apply -k k8s/
Or individually:
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/secret.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/service-headless.yaml
kubectl apply -f k8s/certificate.yaml
kubectl apply -f k8s/ingressroute.yaml