infra/talos/CLAUDE.md

Talos Cluster Setup Guide

This document contains notes on how the Talos home cluster was created and how to recreate it in the future.

Cluster Information

• Cluster Name: home-cluster
• Kubernetes Version: v1.36.1
• Talos Version: v1.13.5
• Control Plane Nodes: 3 (10.0.19.1-3)
• Worker Nodes: 3 (10.0.19.4-6)
• API Endpoint: https://10.0.19.10:6443

Node Details

Hostname	Role	IP	Proxmox Node	VMID	MAC Address
talos-cp1	Control Plane	10.0.19.1	node1	200	BC:24:11:9B:48:92
talos-cp2	Control Plane	10.0.19.2	node2	201	BC:24:11:62:7B:3F
talos-cp3	Control Plane	10.0.19.3	node3	202	BC:24:11:D4:2F:ED
talos-worker1	Worker	10.0.19.4	node1	210	BC:24:11:43:3F:FF
talos-worker2	Worker	10.0.19.5	node2	211	BC:24:11:62:C4:8F
talos-worker3	Worker	10.0.19.6	node3	212	BC:24:11:3F:8E:1A

Prerequisites

• Proxmox home cluster with nodes: node1, node2, node3 (10.0.19.101-103)
• Talos Linux template at VMID 9001 on node1
• DHCP static leases configured for all MAC addresses above
• Terraform/OpenTofu configured with Proxmox provider
• talosctl CLI installed

Complete Cluster Creation Process

1. Generate New Cluster Secrets

cd /Users/graham/dev/infra/talos
talosctl gen secrets -o secrets.yaml --force

2. Generate Base Configurations

Generate the base control plane and worker configs. Per-node settings (hostname, disk) are applied via --config-patch at deploy time, not baked into the configs.

# Generate base control plane config (used for all CP nodes with per-node patches)
talosctl gen config home-cluster https://10.0.19.10:6443 \
  --with-secrets secrets.yaml \
  --kubernetes-version v1.36.1 \
  --output-types controlplane \
  --output controlplane.yaml

# Generate base worker config (used for all worker nodes with per-node patches)
talosctl gen config home-cluster https://10.0.19.10:6443 \
  --with-secrets secrets.yaml \
  --kubernetes-version v1.36.1 \
  --output-types worker \
  --output worker.yaml

Do NOT hardcode hostnames in the base configs. Use the per-node patch files in patches/ instead. After regeneration, remove machine.network.hostname from the base configs and ensure HostnameConfig section is blank (hostnames come from patches).

3. Apply Configurations with Patches

Apply the base config with the appropriate per-node patch:

# Control plane nodes
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.1 --file controlplane.yaml --config-patch @patches/cp1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.2 --file controlplane.yaml --config-patch @patches/cp2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.3 --file controlplane.yaml --config-patch @patches/cp3.yaml --insecure

# Worker nodes
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.4 --file worker.yaml --config-patch @patches/worker1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.5 --file worker.yaml --config-patch @patches/worker2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.6 --file worker.yaml --config-patch @patches/worker3.yaml --insecure

4. Deploy VMs via Terraform

cd /Users/graham/dev/infra/terraform
source .envrc  # Load environment variables

# If recreating, destroy old VMs first
tofu destroy \
  -target=proxmox_virtual_environment_vm.talos_cp1 \
  -target=proxmox_virtual_environment_vm.talos_cp2 \
  -target=proxmox_virtual_environment_vm.talos_cp3 \
  -target=proxmox_virtual_environment_vm.talos_worker1 \
  -target=proxmox_virtual_environment_vm.talos_worker2 \
  -target=proxmox_virtual_environment_vm.talos_worker3 \
  -auto-approve

# Create new VMs
tofu apply \
  -target=proxmox_virtual_environment_vm.talos_cp1 \
  -target=proxmox_virtual_environment_vm.talos_cp2 \
  -target=proxmox_virtual_environment_vm.talos_cp3 \
  -target=proxmox_virtual_environment_vm.talos_worker1 \
  -target=proxmox_virtual_environment_vm.talos_worker2 \
  -target=proxmox_virtual_environment_vm.talos_worker3 \
  -auto-approve

5. Wait for VMs to Boot

cd /Users/graham/dev/infra/talos
sleep 30

# Verify all nodes are up
for ip in 10.0.19.{1..6}; do
  echo -n "$ip: "
  ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down"
done

5. Wait for VMs to Boot

cd /Users/graham/dev/infra/talos
sleep 30

# Verify all nodes are up
for ip in 10.0.19.{1..6}; do
  echo -n "$ip: "
  ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down"
done

6. Bootstrap the Cluster (after configs are applied per step 3)

# Set talosctl endpoints
talosctl --talosconfig talosconfig config endpoint 10.0.19.1 10.0.19.2 10.0.19.3

# Bootstrap Kubernetes on the first control plane node
talosctl --talosconfig talosconfig bootstrap --nodes 10.0.19.1

8. Retrieve Kubeconfig

# Wait for cluster to initialize (about 60 seconds)
sleep 60

# Retrieve and merge kubeconfig
talosctl --talosconfig talosconfig kubeconfig --nodes 10.0.19.1 --force

# Switch to the cluster
kubectl config use-context admin@home-cluster

9. Verify Cluster

# Wait for all nodes to be Ready
kubectl get nodes -o wide

# Check all system pods are running
kubectl get pods -A

# Verify node names
kubectl get nodes
# Expected output:
# NAME            STATUS   ROLES           AGE   VERSION
# talos-cp1       Ready    control-plane   ...   v1.36.1
# talos-cp2       Ready    control-plane   ...   v1.36.1
# talos-cp3       Ready    control-plane   ...   v1.36.1
# talos-worker1   Ready    <none>          ...   v1.36.1
# talos-worker2   Ready    <none>          ...   v1.36.1
# talos-worker3   Ready    <none>          ...   v1.36.1

Important Files

• secrets.yaml - Cluster secrets (CRITICAL - keep secure and backed up!)
• talosconfig - Talosctl client configuration
• controlplane.yaml / worker.yaml - Base node configs (with cluster secrets)
• patches/cp*.yaml - Per-node control plane patches (hostname only)
• patches/worker*.yaml - Per-node worker patches (hostname, disk)
• ~/.kube/config - Contains merged kubeconfig with admin@home-cluster context

Key Lessons Learned

1. Hostname Configuration: Talos uses two places for hostname:

   • machine.network.hostname (for machine config)
   • HostnameConfig document (v1alpha1 config)
   • Only ONE should be set. Use HostnameConfig document with hostname: field for static names.

2. Template Considerations: The Talos template (VMID 9001) has HostnameConfig.auto: stable which auto-generates hostnames. When applying custom configs, this needs to be overridden.

3. Terraform Environment: Always source .envrc before running terraform commands to load backend configuration.

4. Bootstrap Process: Bootstrap must happen AFTER all control plane nodes have configs applied, but BEFORE retrieving kubeconfig.

5. Node Naming: Kubernetes node names come from the Talos hostname configuration, not VM names in Proxmox.

6. Recreating Individual VMs from Template: The OpenTofu VM definitions have been removed. To recreate a single VM directly on Proxmox:

   ssh root@<proxmox-node> "qm clone 9001 <VMID> --name <hostname> --full true --storage local-lvm"
   ssh root@<proxmox-node> "qm set <VMID> --cpu host --balloon 0 --numa 0 --sockets 1 --net0 virtio=<MAC>,bridge=vmbr0,firewall=0 --onboot 1 --agent enabled=1,fstrim_cloned_disks=0,type=virtio --vga std,memory=16"
   ssh root@<proxmox-node> "qm resize <VMID> scsi0 64G"
   ssh root@<proxmox-node> "qm start <VMID>"
   # Wait for DHCP, then apply config with the per-node patch:
   talosctl --talosconfig talosconfig apply-config --nodes <IP> --file worker.yaml --config-patch @patches/worker<N>.yaml --insecure
   # Worker VMs use the factory image (with iscsi-tools, util-linux-tools) by default,
   # so no upgrade step is needed. Control plane nodes use the stock installer.
   

   Critical VM settings: cpu: host is required — without it, Talos boots but never gets a network address. Also set balloon: 0, firewall=0 on NIC.

Troubleshooting

Nodes won't accept new config

If seeing "static hostname is already set" error:

• Ensure only HostnameConfig document has hostname, not machine.network.hostname
• Use --mode=reboot or recreate the VM entirely

Can't connect with talosctl

Set endpoints explicitly:

talosctl --talosconfig talosconfig config endpoint 10.0.19.1 10.0.19.2 10.0.19.3

Nodes not appearing in kubectl

Wait 1-2 minutes after bootstrap. Check:

talosctl --talosconfig talosconfig service --nodes 10.0.19.1
# Look for kubelet, etcd, apid as "Running"

Networking

• CNI: Flannel (deployed automatically)
• Service CIDR: 10.96.0.0/12
• Pod CIDR: 10.244.0.0/16
• DNS: CoreDNS (cluster.local domain)

Storage

• VM boot disks: local-lvm on each Proxmox node (no ceph storage available)

Maintenance

Adding a Node

1. Add VM definition to terraform/talos_vms.tf
2. Run tofu apply
3. Create a per-node patch in patches/ with the appropriate hostname
4. Apply config with patch: talosctl --talosconfig talosconfig apply-config --nodes <IP> --file worker.yaml --config-patch @patches/<new-node>.yaml --insecure

Upgrading Talos

Pre-upgrade checklist:

1. Check current versions: for ip in 10.0.19.{1..6}; do echo -n "$ip: "; talosctl --talosconfig talosconfig version --nodes $ip 2>&1 | grep "Tag:" | grep -v Client; done
2. Check latest release: curl -sL https://github.com/siderolabs/talos/releases/latest | grep -o 'v[0-9]\+\.[0-9]\+\.[0-9]\+' | head -1
3. Verify cluster health: kubectl get nodes -o wide
4. Check extensions on workers: talosctl --talosconfig talosconfig get extensions -n 10.0.19.4

Important: CP nodes use the stock installer image. Worker nodes use the factory image (with iscsi-tools + util-linux-tools extensions). They MUST be upgraded separately with their respective images.

# 1. Upgrade control plane nodes (stock image)
talosctl --talosconfig talosconfig upgrade \
  --nodes 10.0.19.1,10.0.19.2,10.0.19.3 \
  --image ghcr.io/siderolabs/installer:v<VERSION>

# 2. Upgrade worker nodes (factory image with extensions)
#    The schematic ID can be found via: talosctl --talosconfig talosconfig get extensions -n <worker-ip>
talosctl --talosconfig talosconfig upgrade \
  --nodes 10.0.19.4,10.0.19.5,10.0.19.6 \
  --image factory.talos.dev/installer/<SCHEMATIC_ID>:v<VERSION>

# 3. If workers fail to auto-reboot/uncordon, do it manually:
talosctl --talosconfig talosconfig reboot --nodes 10.0.19.4,10.0.19.5,10.0.19.6
# Wait ~90s for reboot, then:
kubectl uncordon talos-worker1 talos-worker2 talos-worker3

Factory schematic ID (as of v1.13.5): 613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245 (extensions: iscsi-tools v0.2.0, util-linux-tools 2.41.4)

Upgrading Kubernetes

talosctl --talosconfig talosconfig upgrade-k8s --to 1.36.1

Backup Strategy

Essential files to backup:

1. secrets.yaml - Cannot recover cluster without this
2. talosconfig - Required for talosctl access
3. controlplane.yaml / worker.yaml - Base node configs
4. Patch files in patches/ directory
5. Terraform state (stored in GitLab HTTP backend)

Store securely in password manager or encrypted backup location.