Update infrastructure documentation with Valkey restart fix
Documents the root cause analysis and solution for the Valkey pod
restart issue. The problem was a race condition during node restarts,
not a Flannel failure.
Resolution:
- Added system-cluster-critical priority class to Valkey
- Application-level Redis health checks provide additional resilience
- Monitoring ongoing to verify stability over 24-48 hours
🤖 Generated with Claude Code
This commit is contained in:
parent
30a0b9a3a0
commit
0388637b65
1 changed files with 210 additions and 0 deletions
210
docs/infrastructure-flannel-issue.md
Normal file
210
docs/infrastructure-flannel-issue.md
Normal file
|
|
@ -0,0 +1,210 @@
|
|||
# Kubernetes Flannel CNI Issue - Valkey Pod Restarts
|
||||
|
||||
## Problem Summary
|
||||
|
||||
The Valkey (Redis-compatible) pod in the `towerops` namespace has restarted **24 times in 28 hours** due to Kubernetes networking issues with the Flannel CNI plugin.
|
||||
|
||||
## Symptoms
|
||||
|
||||
```bash
|
||||
$ kubectl get pods -n towerops | grep valkey
|
||||
valkey-0 1/1 Running 24 (3m36s ago) 28h
|
||||
```
|
||||
|
||||
### Pod Events
|
||||
|
||||
```
|
||||
Warning NodeNotReady Node is not ready
|
||||
Warning FailedCreatePodSandBox Failed to create pod sandbox:
|
||||
rpc error: code = Unknown
|
||||
desc = failed to setup network for sandbox:
|
||||
plugin type="flannel" failed (add):
|
||||
failed to load flannel 'subnet.env' file:
|
||||
open /run/flannel/subnet.env: no such file or directory.
|
||||
Check the flannel pod log for this node.
|
||||
```
|
||||
|
||||
## Root Cause
|
||||
|
||||
The Flannel CNI plugin is unable to find the `/run/flannel/subnet.env` file, which is required for pod network configuration. This file is typically created by the Flannel DaemonSet when it initializes networking on each node.
|
||||
|
||||
## Impact
|
||||
|
||||
1. **Application Errors**: Exq background job processor crashes when Redis connections are dropped
|
||||
2. **Data Loss Risk**: Background jobs may fail or be lost during Redis restarts
|
||||
3. **Degraded Performance**: Constant pod restarts consume cluster resources
|
||||
4. **Monitoring Disruption**: SNMP polling and device monitoring may be delayed
|
||||
|
||||
## Investigation Steps
|
||||
|
||||
### 1. Check Flannel DaemonSet Status
|
||||
|
||||
```bash
|
||||
kubectl get daemonset -n kube-flannel
|
||||
kubectl describe daemonset kube-flannel -n kube-flannel
|
||||
kubectl logs -n kube-flannel -l app=flannel --tail=100
|
||||
```
|
||||
|
||||
### 2. Verify Node Network Configuration
|
||||
|
||||
```bash
|
||||
# SSH to the node where Valkey is running
|
||||
# Check if flannel subnet.env exists
|
||||
ls -la /run/flannel/subnet.env
|
||||
|
||||
# Check flannel network interface
|
||||
ip addr show flannel.1
|
||||
|
||||
# Check flannel routes
|
||||
ip route show | grep flannel
|
||||
```
|
||||
|
||||
### 3. Check CNI Plugin Configuration
|
||||
|
||||
```bash
|
||||
# On the node
|
||||
ls -la /etc/cni/net.d/
|
||||
cat /etc/cni/net.d/*flannel*.conf
|
||||
```
|
||||
|
||||
### 4. Restart Flannel Pods (if needed)
|
||||
|
||||
```bash
|
||||
kubectl delete pods -n kube-flannel -l app=flannel
|
||||
```
|
||||
|
||||
### 5. Check Talos Configuration
|
||||
|
||||
If using Talos Linux, verify CNI configuration in machine config:
|
||||
|
||||
```bash
|
||||
talosctl get machineconfig -n <node-ip>
|
||||
```
|
||||
|
||||
## Potential Solutions
|
||||
|
||||
### Option 1: Restart Flannel DaemonSet
|
||||
|
||||
```bash
|
||||
kubectl rollout restart daemonset/kube-flannel -n kube-flannel
|
||||
```
|
||||
|
||||
### Option 2: Reinstall Flannel
|
||||
|
||||
```bash
|
||||
# Remove existing Flannel
|
||||
kubectl delete -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
|
||||
|
||||
# Reinstall Flannel
|
||||
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
|
||||
```
|
||||
|
||||
### Option 3: Switch to Different CNI Plugin
|
||||
|
||||
Consider migrating to a different CNI plugin if Flannel continues to have issues:
|
||||
- **Cilium** - Advanced networking with eBPF
|
||||
- **Calico** - Feature-rich networking and security
|
||||
- **Weave Net** - Simple overlay network
|
||||
|
||||
### Option 4: Talos-Specific Fix
|
||||
|
||||
If using Talos, ensure CNI configuration in machine config matches Flannel requirements.
|
||||
|
||||
## Workaround (Temporary)
|
||||
|
||||
The application has been updated to handle Redis connection failures gracefully:
|
||||
|
||||
1. **RedisHealthCheck**: Waits for Redis availability before starting Exq
|
||||
2. **ExqSupervisor**: Prevents rapid crash loops with limited restart policy
|
||||
3. **Application continues**: Can function without background jobs if Redis unavailable
|
||||
|
||||
However, this is **not a permanent solution**. The underlying Flannel issue must be resolved.
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Check Valkey Status
|
||||
|
||||
```bash
|
||||
# Check pod restart count
|
||||
kubectl get pod valkey-0 -n towerops -o jsonpath='{.status.containerStatuses[0].restartCount}'
|
||||
|
||||
# Check recent events
|
||||
kubectl get events -n towerops --field-selector involvedObject.name=valkey-0 --sort-by='.lastTimestamp'
|
||||
|
||||
# Check logs
|
||||
kubectl logs -n towerops valkey-0 --tail=50
|
||||
```
|
||||
|
||||
### Check Application Redis Errors
|
||||
|
||||
```bash
|
||||
# Check application logs for Redis connection errors
|
||||
kubectl logs -n towerops deployment/towerops | grep -i "redis\|exq"
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Flannel GitHub Issues](https://github.com/flannel-io/flannel/issues)
|
||||
- [Kubernetes CNI Documentation](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/)
|
||||
- [Talos CNI Configuration](https://www.talos.dev/latest/kubernetes-guides/network/)
|
||||
|
||||
## Solution Implemented
|
||||
|
||||
### Root Cause Analysis
|
||||
|
||||
The issue was **not** a Flannel failure, but a **race condition** during node restarts:
|
||||
|
||||
1. When nodes restart or Flannel pods restart, there's a brief window where `/run/flannel/subnet.env` doesn't exist yet
|
||||
2. Valkey (and other pods) with default priority can start during this window
|
||||
3. CNI plugin fails because subnet.env isn't available yet
|
||||
4. Pods fail to create network sandbox and retry
|
||||
5. By the time Flannel finishes initializing (writes subnet.env), pods succeed on retry
|
||||
|
||||
This explains why:
|
||||
- Flannel logs showed successful initialization
|
||||
- Valkey eventually stabilized after restarts
|
||||
- Issue only occurred during node/pod restart events
|
||||
|
||||
### Fixes Implemented
|
||||
|
||||
#### 1. Application-Level Resilience (Commit: 8ff0c44)
|
||||
|
||||
Added Redis health checks and Exq supervisor improvements:
|
||||
- **RedisHealthCheck module**: Waits for Redis availability before starting Exq
|
||||
- **ExqSupervisor**: Prevents rapid crash loops with limited restart policy
|
||||
- **Benefit**: Application continues functioning even during Redis connection issues
|
||||
|
||||
#### 2. Infrastructure Fix (Commit: 30a0b9a)
|
||||
|
||||
Added `priorityClassName: system-cluster-critical` to Valkey StatefulSet:
|
||||
- Gives Valkey same priority as etcd, coredns, and other cluster services
|
||||
- Scheduler ensures CNI is fully ready before starting Valkey
|
||||
- Significantly reduces race condition window
|
||||
|
||||
### Verification
|
||||
|
||||
Monitor Valkey restart count over next 24-48 hours:
|
||||
|
||||
```bash
|
||||
# Check restart count
|
||||
kubectl get pod valkey-0 -n towerops -o jsonpath='{.status.containerStatuses[0].restartCount}'
|
||||
|
||||
# Monitor for new restart events
|
||||
kubectl get events -n towerops --field-selector involvedObject.name=valkey-0 --watch
|
||||
```
|
||||
|
||||
**Expected Result**: Restart count should remain stable at 0 even during Flannel pod restarts.
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ✅ **Application fixes deployed** - Added Redis health checks and error handling
|
||||
2. ✅ **Root cause identified** - Race condition during CNI initialization
|
||||
3. ✅ **Infrastructure fix applied** - Added priority class to Valkey
|
||||
4. ⏳ **Monitor stability** - Verify no restarts over 24-48 hours
|
||||
5. ⏳ **Apply to other critical pods** - Consider adding priority classes to towerops deployment
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-01-19
|
||||
**Status**: ✅ Fixed (monitoring for verification)
|
||||
**Priority**: Resolved
|
||||
Loading…
Add table
Reference in a new issue