infra/home/cluster/update-k3s-cluster.md

K3s Cluster Update Instructions

Update Order for Multi-Master Cluster

When updating a multi-master K3s cluster with embedded etcd, follow this order:

1. Update node2 first (secondary master)
2. Update node1 second (primary master)
3. Update node3 last (if it's successfully added to etcd cluster)

Steps for Each Node

1. Copy the update script to the node:

   scp update-k3s-node.sh node2:~/
   

2. SSH to the node and run the script:

   ssh node2
   chmod +x update-k3s-node.sh
   ./update-k3s-node.sh
   

3. When prompted about draining:

   • Answer 'y' for production safety (moves pods to other nodes)
   • Answer 'n' if you want faster update and can tolerate brief downtime

4. Monitor the update progress and verify the node comes back online

Verification Commands

From your local machine:

# Check all nodes are ready and on correct version
kubectl get nodes -o wide

# Check etcd cluster health (run on any master node)
ssh node1 'sudo k3s etcdctl endpoint health'

# Check for any pods having issues
kubectl get pods -A | grep -v Running

# Check cluster info
kubectl cluster-info

Rollback (if needed)

If something goes wrong, you can rollback to the previous version:

# On the affected node
sudo systemctl stop k3s
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.33.4+k3s1" INSTALL_K3S_SKIP_START=true sh -
sudo systemctl start k3s

Important Notes

• The update process will cause a brief disruption to pods on that node
• Embedded etcd cluster requires majority of nodes to be online
• Always update secondary masters before primary master
• Monitor etcd logs during update: sudo journalctl -u k3s -f | grep etcd