towerops/docs/infrastructure-flannel-issue.md

Kubernetes Flannel CNI Issue - Valkey Pod Restarts

Problem Summary

The Valkey (Redis-compatible) pod in the towerops namespace has restarted 24 times in 28 hours due to Kubernetes networking issues with the Flannel CNI plugin.

Symptoms

$ kubectl get pods -n towerops | grep valkey
valkey-0  1/1  Running  24 (3m36s ago)  28h

Pod Events

Warning  NodeNotReady            Node is not ready
Warning  FailedCreatePodSandBox  Failed to create pod sandbox:
         rpc error: code = Unknown
         desc = failed to setup network for sandbox:
         plugin type="flannel" failed (add):
         failed to load flannel 'subnet.env' file:
         open /run/flannel/subnet.env: no such file or directory.
         Check the flannel pod log for this node.

Root Cause

The Flannel CNI plugin is unable to find the /run/flannel/subnet.env file, which is required for pod network configuration. This file is typically created by the Flannel DaemonSet when it initializes networking on each node.

Impact

1. Application Errors: Exq background job processor crashes when Redis connections are dropped
2. Data Loss Risk: Background jobs may fail or be lost during Redis restarts
3. Degraded Performance: Constant pod restarts consume cluster resources
4. Monitoring Disruption: SNMP polling and device monitoring may be delayed

Investigation Steps

1. Check Flannel DaemonSet Status

kubectl get daemonset -n kube-flannel
kubectl describe daemonset kube-flannel -n kube-flannel
kubectl logs -n kube-flannel -l app=flannel --tail=100

2. Verify Node Network Configuration

# SSH to the node where Valkey is running
# Check if flannel subnet.env exists
ls -la /run/flannel/subnet.env

# Check flannel network interface
ip addr show flannel.1

# Check flannel routes
ip route show | grep flannel

3. Check CNI Plugin Configuration

# On the node
ls -la /etc/cni/net.d/
cat /etc/cni/net.d/*flannel*.conf

4. Restart Flannel Pods (if needed)

kubectl delete pods -n kube-flannel -l app=flannel

5. Check Talos Configuration

If using Talos Linux, verify CNI configuration in machine config:

talosctl get machineconfig -n <node-ip>

Potential Solutions

Option 1: Restart Flannel DaemonSet

kubectl rollout restart daemonset/kube-flannel -n kube-flannel

Option 2: Reinstall Flannel

# Remove existing Flannel
kubectl delete -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

# Reinstall Flannel
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Option 3: Switch to Different CNI Plugin

Consider migrating to a different CNI plugin if Flannel continues to have issues:

• Cilium - Advanced networking with eBPF
• Calico - Feature-rich networking and security
• Weave Net - Simple overlay network

Option 4: Talos-Specific Fix

If using Talos, ensure CNI configuration in machine config matches Flannel requirements.

Workaround (Temporary)

The application has been updated to handle Redis connection failures gracefully:

1. RedisHealthCheck: Waits for Redis availability before starting Exq
2. ExqSupervisor: Prevents rapid crash loops with limited restart policy
3. Application continues: Can function without background jobs if Redis unavailable

However, this is not a permanent solution. The underlying Flannel issue must be resolved.

Monitoring

Check Valkey Status

# Check pod restart count
kubectl get pod valkey-0 -n towerops -o jsonpath='{.status.containerStatuses[0].restartCount}'

# Check recent events
kubectl get events -n towerops --field-selector involvedObject.name=valkey-0 --sort-by='.lastTimestamp'

# Check logs
kubectl logs -n towerops valkey-0 --tail=50

Check Application Redis Errors

# Check application logs for Redis connection errors
kubectl logs -n towerops deployment/towerops | grep -i "redis\|exq"

References

• Flannel GitHub Issues
• Kubernetes CNI Documentation
• Talos CNI Configuration

Solution Implemented

Root Cause Analysis

The issue was not a Flannel failure, but a race condition during node restarts:

1. When nodes restart or Flannel pods restart, there's a brief window where /run/flannel/subnet.env doesn't exist yet
2. Valkey (and other pods) with default priority can start during this window
3. CNI plugin fails because subnet.env isn't available yet
4. Pods fail to create network sandbox and retry
5. By the time Flannel finishes initializing (writes subnet.env), pods succeed on retry

This explains why:

• Flannel logs showed successful initialization
• Valkey eventually stabilized after restarts
• Issue only occurred during node/pod restart events

Fixes Implemented

1. Application-Level Resilience (Commit: 8ff0c44)

Added Redis health checks and Exq supervisor improvements:

• RedisHealthCheck module: Waits for Redis availability before starting Exq
• ExqSupervisor: Prevents rapid crash loops with limited restart policy
• Benefit: Application continues functioning even during Redis connection issues

2. Infrastructure Fix (Commit: 30a0b9a)

Added priorityClassName: system-cluster-critical to Valkey StatefulSet:

• Gives Valkey same priority as etcd, coredns, and other cluster services
• Scheduler ensures CNI is fully ready before starting Valkey
• Significantly reduces race condition window

Verification

Monitor Valkey restart count over next 24-48 hours:

# Check restart count
kubectl get pod valkey-0 -n towerops -o jsonpath='{.status.containerStatuses[0].restartCount}'

# Monitor for new restart events
kubectl get events -n towerops --field-selector involvedObject.name=valkey-0 --watch

Expected Result: Restart count should remain stable at 0 even during Flannel pod restarts.

Next Steps

1. ✅ Application fixes deployed - Added Redis health checks and error handling
2. ✅ Root cause identified - Race condition during CNI initialization
3. ✅ Infrastructure fix applied - Added priority class to Valkey
4. ⏳ Monitor stability - Verify no restarts over 24-48 hours
5. ⏳ Apply to other critical pods - Consider adding priority classes to towerops deployment

---

Last Updated: 2026-01-19 Status: ✅ Fixed (monitoring for verification) Priority: Resolved