# Talos Cluster Setup Guide This document contains notes on how the Talos home cluster was created and how to recreate it in the future. ## Cluster Information - **Cluster Name**: home-cluster - **Kubernetes Version**: v1.36.1 - **Talos Version**: v1.13.5 - **Control Plane Nodes**: 3 (10.0.19.1-3) - **Worker Nodes**: 3 (10.0.19.4-6) - **API Endpoint**: https://10.0.19.10:6443 ## Node Details | Hostname | Role | IP | Proxmox Node | VMID | MAC Address | |----------|------|-----|--------------|------|-------------| | talos-cp1 | Control Plane | 10.0.19.1 | node1 | 200 | BC:24:11:9B:48:92 | | talos-cp2 | Control Plane | 10.0.19.2 | node2 | 201 | BC:24:11:62:7B:3F | | talos-cp3 | Control Plane | 10.0.19.3 | node3 | 202 | BC:24:11:D4:2F:ED | | talos-worker1 | Worker | 10.0.19.4 | node1 | 210 | BC:24:11:43:3F:FF | | talos-worker2 | Worker | 10.0.19.5 | node2 | 211 | BC:24:11:62:C4:8F | | talos-worker3 | Worker | 10.0.19.6 | node3 | 212 | BC:24:11:3F:8E:1A | ## Prerequisites - Proxmox home cluster with nodes: node1, node2, node3 (10.0.19.101-103) - Talos Linux template at VMID 9001 on node1 - DHCP static leases configured for all MAC addresses above - Terraform/OpenTofu configured with Proxmox provider - talosctl CLI installed ## Complete Cluster Creation Process ### 1. Generate New Cluster Secrets ```bash cd /Users/graham/dev/infra/talos talosctl gen secrets -o secrets.yaml --force ``` ### 2. Generate Base Configurations Generate the base control plane and worker configs. Per-node settings (hostname, disk) are applied via `--config-patch` at deploy time, not baked into the configs. ```bash # Generate base control plane config (used for all CP nodes with per-node patches) talosctl gen config home-cluster https://10.0.19.10:6443 \ --with-secrets secrets.yaml \ --kubernetes-version v1.36.1 \ --output-types controlplane \ --output controlplane.yaml # Generate base worker config (used for all worker nodes with per-node patches) talosctl gen config home-cluster https://10.0.19.10:6443 \ --with-secrets secrets.yaml \ --kubernetes-version v1.36.1 \ --output-types worker \ --output worker.yaml ``` **Do NOT hardcode hostnames in the base configs.** Use the per-node patch files in `patches/` instead. After regeneration, remove `machine.network.hostname` from the base configs and ensure `HostnameConfig` section is blank (hostnames come from patches). ### 3. Apply Configurations with Patches Apply the base config with the appropriate per-node patch: ```bash # Control plane nodes talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.1 --file controlplane.yaml --config-patch @patches/cp1.yaml --insecure talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.2 --file controlplane.yaml --config-patch @patches/cp2.yaml --insecure talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.3 --file controlplane.yaml --config-patch @patches/cp3.yaml --insecure # Worker nodes talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.4 --file worker.yaml --config-patch @patches/worker1.yaml --insecure talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.5 --file worker.yaml --config-patch @patches/worker2.yaml --insecure talosctl --talosconfig talosconfig apply-config --nodes 10.0.19.6 --file worker.yaml --config-patch @patches/worker3.yaml --insecure ``` ### 4. Deploy VMs via Terraform ```bash cd /Users/graham/dev/infra/terraform source .envrc # Load environment variables # If recreating, destroy old VMs first tofu destroy \ -target=proxmox_virtual_environment_vm.talos_cp1 \ -target=proxmox_virtual_environment_vm.talos_cp2 \ -target=proxmox_virtual_environment_vm.talos_cp3 \ -target=proxmox_virtual_environment_vm.talos_worker1 \ -target=proxmox_virtual_environment_vm.talos_worker2 \ -target=proxmox_virtual_environment_vm.talos_worker3 \ -auto-approve # Create new VMs tofu apply \ -target=proxmox_virtual_environment_vm.talos_cp1 \ -target=proxmox_virtual_environment_vm.talos_cp2 \ -target=proxmox_virtual_environment_vm.talos_cp3 \ -target=proxmox_virtual_environment_vm.talos_worker1 \ -target=proxmox_virtual_environment_vm.talos_worker2 \ -target=proxmox_virtual_environment_vm.talos_worker3 \ -auto-approve ``` ### 5. Wait for VMs to Boot ```bash cd /Users/graham/dev/infra/talos sleep 30 # Verify all nodes are up for ip in 10.0.19.{1..6}; do echo -n "$ip: " ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down" done ``` ### 5. Wait for VMs to Boot ```bash cd /Users/graham/dev/infra/talos sleep 30 # Verify all nodes are up for ip in 10.0.19.{1..6}; do echo -n "$ip: " ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down" done ``` ### 6. Bootstrap the Cluster (after configs are applied per step 3) ```bash # Set talosctl endpoints talosctl --talosconfig talosconfig config endpoint 10.0.19.1 10.0.19.2 10.0.19.3 # Bootstrap Kubernetes on the first control plane node talosctl --talosconfig talosconfig bootstrap --nodes 10.0.19.1 ``` ### 8. Retrieve Kubeconfig ```bash # Wait for cluster to initialize (about 60 seconds) sleep 60 # Retrieve and merge kubeconfig talosctl --talosconfig talosconfig kubeconfig --nodes 10.0.19.1 --force # Switch to the cluster kubectl config use-context admin@home-cluster ``` ### 9. Verify Cluster ```bash # Wait for all nodes to be Ready kubectl get nodes -o wide # Check all system pods are running kubectl get pods -A # Verify node names kubectl get nodes # Expected output: # NAME STATUS ROLES AGE VERSION # talos-cp1 Ready control-plane ... v1.36.1 # talos-cp2 Ready control-plane ... v1.36.1 # talos-cp3 Ready control-plane ... v1.36.1 # talos-worker1 Ready ... v1.36.1 # talos-worker2 Ready ... v1.36.1 # talos-worker3 Ready ... v1.36.1 ``` ## Important Files - `secrets.yaml` - Cluster secrets (CRITICAL - keep secure and backed up!) - `talosconfig` - Talosctl client configuration - `controlplane.yaml` / `worker.yaml` - Base node configs (with cluster secrets) - `patches/cp*.yaml` - Per-node control plane patches (hostname only) - `patches/worker*.yaml` - Per-node worker patches (hostname, disk) - `~/.kube/config` - Contains merged kubeconfig with admin@home-cluster context ## Key Lessons Learned 1. **Hostname Configuration**: Talos uses two places for hostname: - `machine.network.hostname` (for machine config) - `HostnameConfig` document (v1alpha1 config) - Only ONE should be set. Use `HostnameConfig` document with `hostname:` field for static names. 2. **Template Considerations**: The Talos template (VMID 9001) has `HostnameConfig.auto: stable` which auto-generates hostnames. When applying custom configs, this needs to be overridden. 3. **Terraform Environment**: Always `source .envrc` before running terraform commands to load backend configuration. 4. **Bootstrap Process**: Bootstrap must happen AFTER all control plane nodes have configs applied, but BEFORE retrieving kubeconfig. 5. **Node Naming**: Kubernetes node names come from the Talos hostname configuration, not VM names in Proxmox. 6. **Recreating Individual VMs from Template**: The OpenTofu VM definitions have been removed. To recreate a single VM directly on Proxmox: ```bash ssh root@ "qm clone 9001 --name --full true --storage local-lvm" ssh root@ "qm set --cpu host --balloon 0 --numa 0 --sockets 1 --net0 virtio=,bridge=vmbr0,firewall=0 --onboot 1 --agent enabled=1,fstrim_cloned_disks=0,type=virtio --vga std,memory=16" ssh root@ "qm resize scsi0 64G" ssh root@ "qm start " # Wait for DHCP, then apply config with the per-node patch: talosctl --talosconfig talosconfig apply-config --nodes --file worker.yaml --config-patch @patches/worker.yaml --insecure # Worker VMs use the factory image (with iscsi-tools, util-linux-tools) by default, # so no upgrade step is needed. Control plane nodes use the stock installer. ``` **Critical VM settings**: `cpu: host` is required — without it, Talos boots but never gets a network address. Also set `balloon: 0`, `firewall=0` on NIC. ## Troubleshooting ### Nodes won't accept new config If seeing "static hostname is already set" error: - Ensure only `HostnameConfig` document has hostname, not `machine.network.hostname` - Use `--mode=reboot` or recreate the VM entirely ### Can't connect with talosctl Set endpoints explicitly: ```bash talosctl --talosconfig talosconfig config endpoint 10.0.19.1 10.0.19.2 10.0.19.3 ``` ### Nodes not appearing in kubectl Wait 1-2 minutes after bootstrap. Check: ```bash talosctl --talosconfig talosconfig service --nodes 10.0.19.1 # Look for kubelet, etcd, apid as "Running" ``` ## Networking - **CNI**: Flannel (deployed automatically) - **Service CIDR**: 10.96.0.0/12 - **Pod CIDR**: 10.244.0.0/16 - **DNS**: CoreDNS (cluster.local domain) ## Storage - **VM boot disks**: local-lvm on each Proxmox node (no ceph storage available) ## Maintenance ### Adding a Node 1. Add VM definition to `terraform/talos_vms.tf` 2. Run `tofu apply` 3. Create a per-node patch in `patches/` with the appropriate hostname 4. Apply config with patch: `talosctl --talosconfig talosconfig apply-config --nodes --file worker.yaml --config-patch @patches/.yaml --insecure` ### Upgrading Talos **Pre-upgrade checklist:** 1. Check current versions: `for ip in 10.0.19.{1..6}; do echo -n "$ip: "; talosctl --talosconfig talosconfig version --nodes $ip 2>&1 | grep "Tag:" | grep -v Client; done` 2. Check latest release: `curl -sL https://github.com/siderolabs/talos/releases/latest | grep -o 'v[0-9]\+\.[0-9]\+\.[0-9]\+' | head -1` 3. Verify cluster health: `kubectl get nodes -o wide` 4. Check extensions on workers: `talosctl --talosconfig talosconfig get extensions -n 10.0.19.4` **Important:** CP nodes use the **stock** installer image. Worker nodes use the **factory** image (with iscsi-tools + util-linux-tools extensions). They MUST be upgraded separately with their respective images. ```bash # 1. Upgrade control plane nodes (stock image) talosctl --talosconfig talosconfig upgrade \ --nodes 10.0.19.1,10.0.19.2,10.0.19.3 \ --image ghcr.io/siderolabs/installer:v # 2. Upgrade worker nodes (factory image with extensions) # The schematic ID can be found via: talosctl --talosconfig talosconfig get extensions -n talosctl --talosconfig talosconfig upgrade \ --nodes 10.0.19.4,10.0.19.5,10.0.19.6 \ --image factory.talos.dev/installer/:v # 3. If workers fail to auto-reboot/uncordon, do it manually: talosctl --talosconfig talosconfig reboot --nodes 10.0.19.4,10.0.19.5,10.0.19.6 # Wait ~90s for reboot, then: kubectl uncordon talos-worker1 talos-worker2 talos-worker3 ``` **Factory schematic ID (as of v1.13.5):** `613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245` (extensions: iscsi-tools v0.2.0, util-linux-tools 2.41.4) ### Upgrading Kubernetes ```bash talosctl --talosconfig talosconfig upgrade-k8s --to 1.36.1 ``` ## Backup Strategy Essential files to backup: 1. `secrets.yaml` - Cannot recover cluster without this 2. `talosconfig` - Required for talosctl access 3. `controlplane.yaml` / `worker.yaml` - Base node configs 4. Patch files in `patches/` directory 5. Terraform state (stored in GitLab HTTP backend) Store securely in password manager or encrypted backup location.