Graham McIntire
5ee241fc16
Addresses production Redis disconnection issues by implementing a highly
resilient Valkey (Redis) setup with automatic failover capabilities.
Infrastructure Changes:
- Add Valkey ConfigMap with optimized connection and memory settings
- TCP keepalive (60s), connection limits (10k clients)
- Memory management (256MB with LRU eviction)
- Separate configs for master, replica, and sentinel
- Update Valkey StatefulSet to 3 replicas (1 master + 2 replicas)
- Auto-configuration via init container (master vs replica)
- Increased memory limits: 256Mi → 1Gi
- Improved readiness probes with replication status checks
- Add Valkey Sentinel StatefulSet (3 instances for quorum)
- Automatic failover detection (5s down-after-milliseconds)
- Fast failover execution (10s timeout)
- Monitors master and promotes replicas automatically
- Add Sentinel headless service for pod discovery
Application Resilience:
- Update Phoenix.PubSub.Redis with TCP keepalive and reconnection
- Connection timeout: 5s
- Exponential backoff: 500ms → 30s
- exit_on_disconnection: false
- Update Exq background jobs with same resilience settings
- TCP keepalive enabled
- Better connection pool management
Benefits:
- Automatic failover when Valkey pod dies (no manual intervention)
- Zero data loss with replica synchronization
- Fast failure detection and recovery (5-10s total)
- Survives Flannel CNI networking issues
- Apps reconnect automatically during disconnections
Testing:
- All 3,686 tests passing
- Kustomize manifest validated
🤖 Generated with Claude Code
17 lines
423 B
YAML
17 lines
423 B
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
namespace: towerops
|
|
resources:
|
|
- namespace.yaml
|
|
- deployment.yaml
|
|
- service.yaml
|
|
- service-headless.yaml
|
|
- valkey-configmap.yaml
|
|
- valkey-statefulset.yaml
|
|
- valkey-service.yaml
|
|
- valkey-sentinel-statefulset.yaml
|
|
- valkey-sentinel-service.yaml
|
|
- etcd-statefulset.yaml
|
|
- certificate.yaml
|
|
- ingressroute.yaml
|
|
- poddisruptionbudget.yaml
|