From 0388637b65bfa24db712261248ffc9f0bb2fc3c6 Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Mon, 19 Jan 2026 15:41:11 -0600 Subject: [PATCH] Update infrastructure documentation with Valkey restart fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the root cause analysis and solution for the Valkey pod restart issue. The problem was a race condition during node restarts, not a Flannel failure. Resolution: - Added system-cluster-critical priority class to Valkey - Application-level Redis health checks provide additional resilience - Monitoring ongoing to verify stability over 24-48 hours 🤖 Generated with Claude Code --- docs/infrastructure-flannel-issue.md | 210 +++++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 docs/infrastructure-flannel-issue.md diff --git a/docs/infrastructure-flannel-issue.md b/docs/infrastructure-flannel-issue.md new file mode 100644 index 00000000..abe11638 --- /dev/null +++ b/docs/infrastructure-flannel-issue.md @@ -0,0 +1,210 @@ +# Kubernetes Flannel CNI Issue - Valkey Pod Restarts + +## Problem Summary + +The Valkey (Redis-compatible) pod in the `towerops` namespace has restarted **24 times in 28 hours** due to Kubernetes networking issues with the Flannel CNI plugin. + +## Symptoms + +```bash +$ kubectl get pods -n towerops | grep valkey +valkey-0 1/1 Running 24 (3m36s ago) 28h +``` + +### Pod Events + +``` +Warning NodeNotReady Node is not ready +Warning FailedCreatePodSandBox Failed to create pod sandbox: + rpc error: code = Unknown + desc = failed to setup network for sandbox: + plugin type="flannel" failed (add): + failed to load flannel 'subnet.env' file: + open /run/flannel/subnet.env: no such file or directory. + Check the flannel pod log for this node. +``` + +## Root Cause + +The Flannel CNI plugin is unable to find the `/run/flannel/subnet.env` file, which is required for pod network configuration. This file is typically created by the Flannel DaemonSet when it initializes networking on each node. + +## Impact + +1. **Application Errors**: Exq background job processor crashes when Redis connections are dropped +2. **Data Loss Risk**: Background jobs may fail or be lost during Redis restarts +3. **Degraded Performance**: Constant pod restarts consume cluster resources +4. **Monitoring Disruption**: SNMP polling and device monitoring may be delayed + +## Investigation Steps + +### 1. Check Flannel DaemonSet Status + +```bash +kubectl get daemonset -n kube-flannel +kubectl describe daemonset kube-flannel -n kube-flannel +kubectl logs -n kube-flannel -l app=flannel --tail=100 +``` + +### 2. Verify Node Network Configuration + +```bash +# SSH to the node where Valkey is running +# Check if flannel subnet.env exists +ls -la /run/flannel/subnet.env + +# Check flannel network interface +ip addr show flannel.1 + +# Check flannel routes +ip route show | grep flannel +``` + +### 3. Check CNI Plugin Configuration + +```bash +# On the node +ls -la /etc/cni/net.d/ +cat /etc/cni/net.d/*flannel*.conf +``` + +### 4. Restart Flannel Pods (if needed) + +```bash +kubectl delete pods -n kube-flannel -l app=flannel +``` + +### 5. Check Talos Configuration + +If using Talos Linux, verify CNI configuration in machine config: + +```bash +talosctl get machineconfig -n +``` + +## Potential Solutions + +### Option 1: Restart Flannel DaemonSet + +```bash +kubectl rollout restart daemonset/kube-flannel -n kube-flannel +``` + +### Option 2: Reinstall Flannel + +```bash +# Remove existing Flannel +kubectl delete -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml + +# Reinstall Flannel +kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml +``` + +### Option 3: Switch to Different CNI Plugin + +Consider migrating to a different CNI plugin if Flannel continues to have issues: +- **Cilium** - Advanced networking with eBPF +- **Calico** - Feature-rich networking and security +- **Weave Net** - Simple overlay network + +### Option 4: Talos-Specific Fix + +If using Talos, ensure CNI configuration in machine config matches Flannel requirements. + +## Workaround (Temporary) + +The application has been updated to handle Redis connection failures gracefully: + +1. **RedisHealthCheck**: Waits for Redis availability before starting Exq +2. **ExqSupervisor**: Prevents rapid crash loops with limited restart policy +3. **Application continues**: Can function without background jobs if Redis unavailable + +However, this is **not a permanent solution**. The underlying Flannel issue must be resolved. + +## Monitoring + +### Check Valkey Status + +```bash +# Check pod restart count +kubectl get pod valkey-0 -n towerops -o jsonpath='{.status.containerStatuses[0].restartCount}' + +# Check recent events +kubectl get events -n towerops --field-selector involvedObject.name=valkey-0 --sort-by='.lastTimestamp' + +# Check logs +kubectl logs -n towerops valkey-0 --tail=50 +``` + +### Check Application Redis Errors + +```bash +# Check application logs for Redis connection errors +kubectl logs -n towerops deployment/towerops | grep -i "redis\|exq" +``` + +## References + +- [Flannel GitHub Issues](https://github.com/flannel-io/flannel/issues) +- [Kubernetes CNI Documentation](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) +- [Talos CNI Configuration](https://www.talos.dev/latest/kubernetes-guides/network/) + +## Solution Implemented + +### Root Cause Analysis + +The issue was **not** a Flannel failure, but a **race condition** during node restarts: + +1. When nodes restart or Flannel pods restart, there's a brief window where `/run/flannel/subnet.env` doesn't exist yet +2. Valkey (and other pods) with default priority can start during this window +3. CNI plugin fails because subnet.env isn't available yet +4. Pods fail to create network sandbox and retry +5. By the time Flannel finishes initializing (writes subnet.env), pods succeed on retry + +This explains why: +- Flannel logs showed successful initialization +- Valkey eventually stabilized after restarts +- Issue only occurred during node/pod restart events + +### Fixes Implemented + +#### 1. Application-Level Resilience (Commit: 8ff0c44) + +Added Redis health checks and Exq supervisor improvements: +- **RedisHealthCheck module**: Waits for Redis availability before starting Exq +- **ExqSupervisor**: Prevents rapid crash loops with limited restart policy +- **Benefit**: Application continues functioning even during Redis connection issues + +#### 2. Infrastructure Fix (Commit: 30a0b9a) + +Added `priorityClassName: system-cluster-critical` to Valkey StatefulSet: +- Gives Valkey same priority as etcd, coredns, and other cluster services +- Scheduler ensures CNI is fully ready before starting Valkey +- Significantly reduces race condition window + +### Verification + +Monitor Valkey restart count over next 24-48 hours: + +```bash +# Check restart count +kubectl get pod valkey-0 -n towerops -o jsonpath='{.status.containerStatuses[0].restartCount}' + +# Monitor for new restart events +kubectl get events -n towerops --field-selector involvedObject.name=valkey-0 --watch +``` + +**Expected Result**: Restart count should remain stable at 0 even during Flannel pod restarts. + +## Next Steps + +1. ✅ **Application fixes deployed** - Added Redis health checks and error handling +2. ✅ **Root cause identified** - Race condition during CNI initialization +3. ✅ **Infrastructure fix applied** - Added priority class to Valkey +4. ⏳ **Monitor stability** - Verify no restarts over 24-48 hours +5. ⏳ **Apply to other critical pods** - Consider adding priority classes to towerops deployment + +--- + +**Last Updated**: 2026-01-19 +**Status**: ✅ Fixed (monitoring for verification) +**Priority**: Resolved