infra/home/cluster/high-availability-setup.md

High Availability Setup for K3s Cluster

Current Situation

• node1 (10.0.101.21): Has public IP 204.110.191.2, currently being updated
• node2 (10.0.101.22): No public IP
• node3 (10.0.101.23): No public IP

Solution Options

Option 1: Floating IP (Recommended if your provider supports it)

Use a floating/virtual IP that can be moved between nodes:

1. Get a floating IP from your provider
2. Use Keepalived to manage automatic failover
3. Install MetalLB or Kube-VIP for in-cluster load balancing

Option 2: External Load Balancer (Best for production)

Use your hosting provider's load balancer or set up HAProxy/Nginx:

1. Set up an external load balancer pointing to all 3 nodes
2. Configure health checks on port 6443 (K3s API)
3. For services, use NodePort and configure LB to forward to all nodes

Option 3: DNS Round-Robin with Health Checks (Simple but less reliable)

Configure multiple A records with health checking:

1. Add A records for all node IPs to your domain
2. Use a DNS provider that supports health checks (like Route53)
3. Automatically remove unhealthy nodes from rotation

Option 4: Tailscale + Caddy on Separate Host (Quick solution)

Since you have Tailscale, use a separate small VPS as a reverse proxy:

1. Get a small VPS with public IP
2. Install Tailscale and join your network
3. Install Caddy to reverse proxy to cluster services
4. Point DNS to this VPS instead of directly to cluster

Immediate Solution: Configure Services for High Availability

While node1 is down, ensure your apps remain accessible:

1. Configure Pod Anti-Affinity

Ensure critical pods run on different nodes: