Graham McIntire
5ee241fc16
Addresses production Redis disconnection issues by implementing a highly
resilient Valkey (Redis) setup with automatic failover capabilities.
Infrastructure Changes:
- Add Valkey ConfigMap with optimized connection and memory settings
- TCP keepalive (60s), connection limits (10k clients)
- Memory management (256MB with LRU eviction)
- Separate configs for master, replica, and sentinel
- Update Valkey StatefulSet to 3 replicas (1 master + 2 replicas)
- Auto-configuration via init container (master vs replica)
- Increased memory limits: 256Mi → 1Gi
- Improved readiness probes with replication status checks
- Add Valkey Sentinel StatefulSet (3 instances for quorum)
- Automatic failover detection (5s down-after-milliseconds)
- Fast failover execution (10s timeout)
- Monitors master and promotes replicas automatically
- Add Sentinel headless service for pod discovery
Application Resilience:
- Update Phoenix.PubSub.Redis with TCP keepalive and reconnection
- Connection timeout: 5s
- Exponential backoff: 500ms → 30s
- exit_on_disconnection: false
- Update Exq background jobs with same resilience settings
- TCP keepalive enabled
- Better connection pool management
Benefits:
- Automatic failover when Valkey pod dies (no manual intervention)
- Zero data loss with replica synchronization
- Fast failure detection and recovery (5-10s total)
- Survives Flannel CNI networking issues
- Apps reconnect automatically during disconnections
Testing:
- All 3,686 tests passing
- Kustomize manifest validated
🤖 Generated with Claude Code
|
||
|---|---|---|
| .. | ||
| mix/tasks | ||
| snmpkit | ||
| towerops | ||
| towerops_web | ||
| snmp_lib.ex | ||
| snmp_mgr.ex | ||
| snmpkit.ex | ||
| towerops.ex | ||
| towerops_web.ex | ||