towerops/docs/infrastructure-flannel-issue.md

# Kubernetes Flannel CNI Issue - Valkey Pod Restarts

## Problem Summary

The Valkey (Redis-compatible) pod in the `towerops` namespace has restarted **24 times in 28 hours** due to Kubernetes networking issues with the Flannel CNI plugin.

## Symptoms

```bash
$ kubectl get pods -n towerops | grep valkey
valkey-0  1/1  Running  24 (3m36s ago)  28h
```

### Pod Events

```
Warning  NodeNotReady            Node is not ready
Warning  FailedCreatePodSandBox  Failed to create pod sandbox:
         rpc error: code = Unknown
         desc = failed to setup network for sandbox:
         plugin type="flannel" failed (add):
         failed to load flannel 'subnet.env' file:
         open /run/flannel/subnet.env: no such file or directory.
         Check the flannel pod log for this node.
```

## Root Cause

The Flannel CNI plugin is unable to find the `/run/flannel/subnet.env` file, which is required for pod network configuration. This file is typically created by the Flannel DaemonSet when it initializes networking on each node.

## Impact

1. **Application Errors**: Exq background job processor crashes when Redis connections are dropped
2. **Data Loss Risk**: Background jobs may fail or be lost during Redis restarts
3. **Degraded Performance**: Constant pod restarts consume cluster resources
4. **Monitoring Disruption**: SNMP polling and device monitoring may be delayed

## Investigation Steps

### 1. Check Flannel DaemonSet Status

```bash
kubectl get daemonset -n kube-flannel
kubectl describe daemonset kube-flannel -n kube-flannel
kubectl logs -n kube-flannel -l app=flannel --tail=100
```

### 2. Verify Node Network Configuration

```bash
# SSH to the node where Valkey is running
# Check if flannel subnet.env exists
ls -la /run/flannel/subnet.env

# Check flannel network interface
ip addr show flannel.1

# Check flannel routes
ip route show | grep flannel
```

### 3. Check CNI Plugin Configuration

```bash
# On the node
ls -la /etc/cni/net.d/
cat /etc/cni/net.d/*flannel*.conf
```

### 4. Restart Flannel Pods (if needed)

```bash
kubectl delete pods -n kube-flannel -l app=flannel
```

### 5. Check Talos Configuration

If using Talos Linux, verify CNI configuration in machine config:

```bash
talosctl get machineconfig -n <node-ip>
```

## Potential Solutions

### Option 1: Restart Flannel DaemonSet

```bash
kubectl rollout restart daemonset/kube-flannel -n kube-flannel
```

### Option 2: Reinstall Flannel

```bash
# Remove existing Flannel
kubectl delete -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

# Reinstall Flannel
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
```

### Option 3: Switch to Different CNI Plugin

Consider migrating to a different CNI plugin if Flannel continues to have issues:
- **Cilium** - Advanced networking with eBPF
- **Calico** - Feature-rich networking and security
- **Weave Net** - Simple overlay network

### Option 4: Talos-Specific Fix

If using Talos, ensure CNI configuration in machine config matches Flannel requirements.

## Workaround (Temporary)

The application has been updated to handle Redis connection failures gracefully:

1. **RedisHealthCheck**: Waits for Redis availability before starting Exq
2. **ExqSupervisor**: Prevents rapid crash loops with limited restart policy
3. **Application continues**: Can function without background jobs if Redis unavailable

However, this is **not a permanent solution**. The underlying Flannel issue must be resolved.

## Monitoring

### Check Valkey Status

```bash
# Check pod restart count
kubectl get pod valkey-0 -n towerops -o jsonpath='{.status.containerStatuses[0].restartCount}'

# Check recent events
kubectl get events -n towerops --field-selector involvedObject.name=valkey-0 --sort-by='.lastTimestamp'

# Check logs
kubectl logs -n towerops valkey-0 --tail=50
```

### Check Application Redis Errors

```bash
# Check application logs for Redis connection errors
kubectl logs -n towerops deployment/towerops | grep -i "redis\|exq"
```

## References

- [Flannel GitHub Issues](https://github.com/flannel-io/flannel/issues)
- [Kubernetes CNI Documentation](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/)
- [Talos CNI Configuration](https://www.talos.dev/latest/kubernetes-guides/network/)

## Solution Implemented

### Root Cause Analysis

The issue was **not** a Flannel failure, but a **race condition** during node restarts:

1. When nodes restart or Flannel pods restart, there's a brief window where `/run/flannel/subnet.env` doesn't exist yet
2. Valkey (and other pods) with default priority can start during this window
3. CNI plugin fails because subnet.env isn't available yet
4. Pods fail to create network sandbox and retry
5. By the time Flannel finishes initializing (writes subnet.env), pods succeed on retry

This explains why:
- Flannel logs showed successful initialization
- Valkey eventually stabilized after restarts
- Issue only occurred during node/pod restart events

### Fixes Implemented

#### 1. Application-Level Resilience (Commit: 8ff0c44)

Added Redis health checks and Exq supervisor improvements:
- **RedisHealthCheck module**: Waits for Redis availability before starting Exq
- **ExqSupervisor**: Prevents rapid crash loops with limited restart policy
- **Benefit**: Application continues functioning even during Redis connection issues

#### 2. Infrastructure Fix (Commit: 30a0b9a)

Added `priorityClassName: system-cluster-critical` to Valkey StatefulSet:
- Gives Valkey same priority as etcd, coredns, and other cluster services
- Scheduler ensures CNI is fully ready before starting Valkey
- Significantly reduces race condition window

### Verification

Monitor Valkey restart count over next 24-48 hours:

```bash
# Check restart count
kubectl get pod valkey-0 -n towerops -o jsonpath='{.status.containerStatuses[0].restartCount}'

# Monitor for new restart events
kubectl get events -n towerops --field-selector involvedObject.name=valkey-0 --watch
```

**Expected Result**: Restart count should remain stable at 0 even during Flannel pod restarts.

## Next Steps

1. ✅ **Application fixes deployed** - Added Redis health checks and error handling
2. ✅ **Root cause identified** - Race condition during CNI initialization
3. ✅ **Infrastructure fix applied** - Added priority class to Valkey
4. ⏳ **Monitor stability** - Verify no restarts over 24-48 hours
5. ⏳ **Apply to other critical pods** - Consider adding priority classes to towerops deployment

---

**Last Updated**: 2026-01-19
**Status**: ✅ Fixed (monitoring for verification)
**Priority**: Resolved