infra/talos/CLAUDE.md

Talos Cluster Setup Guide

This document contains notes on how the Talos home cluster was created and how to recreate it in the future.

Cluster Information

• Cluster Name: home-cluster
• Kubernetes Version: v1.36.1
• Talos Version: v1.13.3
• Control Plane Nodes: 3 (10.0.15.1-3)
• Worker Nodes: 4 (10.0.15.4-7)
• API Endpoint: https://10.0.15.1:6443

Node Details

Hostname	Role	IP	Proxmox Node	VMID	MAC Address
talos-cp1	Control Plane	10.0.15.1	node1	200	BC:24:11:9B:48:92
talos-cp2	Control Plane	10.0.15.2	node2	201	BC:24:11:62:7B:3F
talos-cp3	Control Plane	10.0.15.3	node3	202	BC:24:11:D4:2F:ED
talos-worker1	Worker	10.0.15.4	node1	210	BC:24:11:43:3F:FF
talos-worker2	Worker	10.0.15.5	node2	211	BC:24:11:62:C4:8F
talos-worker3	Worker	10.0.15.6	node3	212	BC:24:11:3F:8E:1A
talos4	Worker	10.0.15.7	bare-metal (HP t620)	—	7C:D3:0A:12:C8:7E

Prerequisites

• Proxmox home cluster with nodes: node1, node2, node3 (10.0.15.101-103)
• Talos Linux template at VMID 9001 on node1
• DHCP static leases configured for all MAC addresses above
• Terraform/OpenTofu configured with Proxmox provider
• talosctl CLI installed

Complete Cluster Creation Process

1. Generate New Cluster Secrets

cd /Users/graham/dev/infra/talos
talosctl gen secrets -o secrets.yaml --force

2. Generate Base Configurations

Generate the base control plane and worker configs. Per-node settings (hostname, disk) are applied via --config-patch at deploy time, not baked into the configs.

# Generate base control plane config (used for all CP nodes with per-node patches)
talosctl gen config home-cluster https://10.0.15.1:6443 \
  --with-secrets secrets.yaml \
  --kubernetes-version v1.36.1 \
  --output-types controlplane \
  --output controlplane.yaml

# Generate base worker config (used for all worker nodes with per-node patches)
talosctl gen config home-cluster https://10.0.15.1:6443 \
  --with-secrets secrets.yaml \
  --kubernetes-version v1.36.1 \
  --output-types worker \
  --output worker.yaml

Do NOT hardcode hostnames in the base configs. Use the per-node patch files in patches/ instead. After regeneration, remove machine.network.hostname from the base configs and ensure HostnameConfig section is blank (hostnames come from patches).

3. Apply Configurations with Patches

Apply the base config with the appropriate per-node patch:

# Control plane nodes
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.1 --file controlplane.yaml --config-patch @patches/cp1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.2 --file controlplane.yaml --config-patch @patches/cp2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.3 --file controlplane.yaml --config-patch @patches/cp3.yaml --insecure

# Worker nodes
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.4 --file worker.yaml --config-patch @patches/worker1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.5 --file worker.yaml --config-patch @patches/worker2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.6 --file worker.yaml --config-patch @patches/worker3.yaml --insecure

# Bare-metal worker (different install disk)
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.7 --file worker.yaml --config-patch @patches/worker4.yaml --insecure

4. Deploy VMs via Terraform

cd /Users/graham/dev/infra/terraform
source .envrc  # Load environment variables

# If recreating, destroy old VMs first
tofu destroy \
  -target=proxmox_virtual_environment_vm.talos_cp1 \
  -target=proxmox_virtual_environment_vm.talos_cp2 \
  -target=proxmox_virtual_environment_vm.talos_cp3 \
  -target=proxmox_virtual_environment_vm.talos_worker1 \
  -target=proxmox_virtual_environment_vm.talos_worker2 \
  -target=proxmox_virtual_environment_vm.talos_worker3 \
  -auto-approve

# Create new VMs
tofu apply \
  -target=proxmox_virtual_environment_vm.talos_cp1 \
  -target=proxmox_virtual_environment_vm.talos_cp2 \
  -target=proxmox_virtual_environment_vm.talos_cp3 \
  -target=proxmox_virtual_environment_vm.talos_worker1 \
  -target=proxmox_virtual_environment_vm.talos_worker2 \
  -target=proxmox_virtual_environment_vm.talos_worker3 \
  -auto-approve

5. Wait for VMs to Boot

cd /Users/graham/dev/infra/talos
sleep 30

# Verify all nodes are up
for ip in 10.0.15.{1..6}; do
  echo -n "$ip: "
  ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down"
done

5. Wait for VMs to Boot

cd /Users/graham/dev/infra/talos
sleep 30

# Verify all nodes are up
for ip in 10.0.15.{1..6}; do
  echo -n "$ip: "
  ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down"
done

6. Bootstrap the Cluster (after configs are applied per step 3)

# Set talosctl endpoints
talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3

# Bootstrap Kubernetes on the first control plane node
talosctl --talosconfig talosconfig bootstrap --nodes 10.0.15.1

8. Retrieve Kubeconfig

# Wait for cluster to initialize (about 60 seconds)
sleep 60

# Retrieve and merge kubeconfig
talosctl --talosconfig talosconfig kubeconfig --nodes 10.0.15.1 --force

# Switch to the cluster
kubectl config use-context admin@home-cluster

9. Verify Cluster

# Wait for all nodes to be Ready
kubectl get nodes -o wide

# Check all system pods are running
kubectl get pods -A

# Verify node names
kubectl get nodes
# Expected output:
# NAME            STATUS   ROLES           AGE   VERSION
# talos-cp1       Ready    control-plane   ...   v1.36.1
# talos-cp2       Ready    control-plane   ...   v1.36.1
# talos-cp3       Ready    control-plane   ...   v1.36.1
# talos-worker1   Ready    <none>          ...   v1.36.1
# talos-worker2   Ready    <none>          ...   v1.36.1
# talos-worker3   Ready    <none>          ...   v1.36.1

Important Files

• secrets.yaml - Cluster secrets (CRITICAL - keep secure and backed up!)
• talosconfig - Talosctl client configuration
• controlplane.yaml / worker.yaml - Base node configs (with cluster secrets)
• patches/cp*.yaml - Per-node control plane patches (hostname only)
• patches/worker*.yaml - Per-node worker patches (hostname, disk)
• ~/.kube/config - Contains merged kubeconfig with admin@home-cluster context

Key Lessons Learned

1. Hostname Configuration: Talos uses two places for hostname:

   • machine.network.hostname (for machine config)
   • HostnameConfig document (v1alpha1 config)
   • Only ONE should be set. Use HostnameConfig document with hostname: field for static names.

2. Template Considerations: The Talos template (VMID 9001) has HostnameConfig.auto: stable which auto-generates hostnames. When applying custom configs, this needs to be overridden.

3. Terraform Environment: Always source .envrc before running terraform commands to load backend configuration.

4. Bootstrap Process: Bootstrap must happen AFTER all control plane nodes have configs applied, but BEFORE retrieving kubeconfig.

5. Node Naming: Kubernetes node names come from the Talos hostname configuration, not VM names in Proxmox.

6. Recreating Individual VMs from Template: The OpenTofu VM definitions have been removed. To recreate a single VM directly on Proxmox:

   ssh root@<proxmox-node> "qm clone 9001 <VMID> --name <hostname> --full true --storage local-lvm"
   ssh root@<proxmox-node> "qm set <VMID> --cpu host --balloon 0 --numa 0 --sockets 1 --net0 virtio=<MAC>,bridge=vmbr0,firewall=0 --onboot 1 --agent enabled=1,fstrim_cloned_disks=0,type=virtio --vga std,memory=16"
   ssh root@<proxmox-node> "qm resize <VMID> scsi0 64G"
   ssh root@<proxmox-node> "qm start <VMID>"
   # Wait for DHCP, then apply config with the per-node patch:
   talosctl --talosconfig talosconfig apply-config --nodes <IP> --file worker.yaml --config-patch @patches/worker<N>.yaml --insecure
   # Worker VMs use the factory image (with iscsi-tools, util-linux-tools) by default,
   # so no upgrade step is needed. Control plane nodes use the stock installer.
   

   Critical VM settings: cpu: host is required — without it, Talos boots but never gets a network address. Also set balloon: 0, firewall=0 on NIC.

7. Bare-Metal Install Disk Enumeration: On bare-metal nodes booted from a Talos USB, the USB enumerates as /dev/sda and the internal drive as /dev/sdb. Set machine.install.disk: /dev/sdb in the config so Talos installs to the internal drive. After install, when the USB is removed, the internal drive becomes /dev/sda in the running system — this is expected and does not require reconfiguration. Example: talos4 (HP t620 with 128GB SK hynix M.2 SATA) was installed this way.

Troubleshooting

Nodes won't accept new config

If seeing "static hostname is already set" error:

• Ensure only HostnameConfig document has hostname, not machine.network.hostname
• Use --mode=reboot or recreate the VM entirely

Can't connect with talosctl

Set endpoints explicitly:

talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3

Nodes not appearing in kubectl

Wait 1-2 minutes after bootstrap. Check:

talosctl --talosconfig talosconfig service --nodes 10.0.15.1
# Look for kubelet, etcd, apid as "Running"

Networking

• CNI: Flannel (deployed automatically)
• Service CIDR: 10.96.0.0/12
• Pod CIDR: 10.244.0.0/16
• DNS: CoreDNS (cluster.local domain)

Storage

• VM boot disks: local-lvm on each Proxmox node (no ceph storage available)

Maintenance

Adding a Node

1. Add VM definition to terraform/talos_vms.tf
2. Run tofu apply
3. Create a per-node patch in patches/ with the appropriate hostname
4. Apply config with patch: talosctl --talosconfig talosconfig apply-config --nodes <IP> --file worker.yaml --config-patch @patches/<new-node>.yaml --insecure

Upgrading Talos

talosctl --talosconfig talosconfig upgrade --image ghcr.io/siderolabs/installer:v1.13.3

Upgrading Kubernetes

talosctl --talosconfig talosconfig upgrade-k8s --to 1.36.1

Backup Strategy

Essential files to backup:

1. secrets.yaml - Cannot recover cluster without this
2. talosconfig - Required for talosctl access
3. controlplane.yaml / worker.yaml - Base node configs
4. Patch files in patches/ directory
5. Terraform state (stored in GitLab HTTP backend)

Store securely in password manager or encrypted backup location.