infra/talos/CLAUDE.md

Talos Cluster Setup Guide

This document contains notes on how the Talos home cluster was created and how to recreate it in the future.

Cluster Information

• Cluster Name: home-cluster
• Kubernetes Version: v1.36.1
• Talos Version: v1.13.3
• Control Plane Nodes: 3 (10.0.15.1-3)
• Worker Nodes: 4 (10.0.15.4-7)
• API Endpoint: https://10.0.15.1:6443

Node Details

Hostname	Role	IP	Proxmox Node	VMID	MAC Address
talos-cp1	Control Plane	10.0.15.1	node1	200	BC:24:11:9B:48:92
talos-cp2	Control Plane	10.0.15.2	node2	201	BC:24:11:62:7B:3F
talos-cp3	Control Plane	10.0.15.3	node3	202	BC:24:11:D4:2F:ED
talos-worker1	Worker	10.0.15.4	node1	210	BC:24:11:43:3F:FF
talos-worker2	Worker	10.0.15.5	node2	211	BC:24:11:62:C4:8F
talos-worker3	Worker	10.0.15.6	node3	212	BC:24:11:3F:8E:1A
talos4	Worker	10.0.15.7	bare-metal (HP t620)	—	7C:D3:0A:12:C8:7E

Prerequisites

• Proxmox home cluster with nodes: node1, node2, node3 (10.0.15.101-103)
• Talos Linux template at VMID 9001 on node1
• DHCP static leases configured for all MAC addresses above
• Terraform/OpenTofu configured with Proxmox provider
• talosctl CLI installed

Complete Cluster Creation Process

1. Generate New Cluster Secrets

cd /Users/graham/dev/infra/talos
talosctl gen secrets -o secrets.yaml --force

2. Generate Node-Specific Configurations

Each node needs a configuration with its specific hostname. Use patch files to set hostnames:

mkdir -p configs

# Generate control plane configs
for i in 1 2 3; do
  cat > /tmp/cp${i}-patch.yaml <<EOF
machine:
  network:
    hostname: talos-cp${i}
EOF
  talosctl gen config home-cluster https://10.0.15.1:6443 \
    --with-secrets secrets.yaml \
    --kubernetes-version v1.36.1 \
    --output-types controlplane \
    --output configs/talos-cp${i}.yaml \
    --config-patch /tmp/cp${i}-patch.yaml
done

# Generate worker configs
for i in 1 2 3; do
  cat > /tmp/w${i}-patch.yaml <<EOF
machine:
  network:
    hostname: talos-worker${i}
EOF
  talosctl gen config home-cluster https://10.0.15.1:6443 \
    --with-secrets secrets.yaml \
    --kubernetes-version v1.36.1 \
    --output-types worker \
    --output configs/talos-worker${i}.yaml \
    --config-patch /tmp/w${i}-patch.yaml
done

3. Fix HostnameConfig Sections

The generated configs will have both machine.network.hostname and a HostnameConfig document. Fix this:

# Remove hostname from machine.network section
for file in configs/*.yaml; do
  sed -i '' '/^    network:$/,/^        # # Configures KubeSpan/{
    /^        hostname:/d
  }' "$file"
done

# Update HostnameConfig sections
sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-cp1
}' configs/talos-cp1.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-cp2
}' configs/talos-cp2.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-cp3
}' configs/talos-cp3.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-worker1
}' configs/talos-worker1.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-worker2
}' configs/talos-worker2.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-worker3
}' configs/talos-worker3.yaml

4. Deploy VMs via Terraform

cd /Users/graham/dev/infra/terraform
source .envrc  # Load environment variables

# If recreating, destroy old VMs first
tofu destroy \
  -target=proxmox_virtual_environment_vm.talos_cp1 \
  -target=proxmox_virtual_environment_vm.talos_cp2 \
  -target=proxmox_virtual_environment_vm.talos_cp3 \
  -target=proxmox_virtual_environment_vm.talos_worker1 \
  -target=proxmox_virtual_environment_vm.talos_worker2 \
  -target=proxmox_virtual_environment_vm.talos_worker3 \
  -auto-approve

# Create new VMs
tofu apply \
  -target=proxmox_virtual_environment_vm.talos_cp1 \
  -target=proxmox_virtual_environment_vm.talos_cp2 \
  -target=proxmox_virtual_environment_vm.talos_cp3 \
  -target=proxmox_virtual_environment_vm.talos_worker1 \
  -target=proxmox_virtual_environment_vm.talos_worker2 \
  -target=proxmox_virtual_environment_vm.talos_worker3 \
  -auto-approve

5. Wait for VMs to Boot

cd /Users/graham/dev/infra/talos
sleep 30

# Verify all nodes are up
for ip in 10.0.15.{1..6}; do
  echo -n "$ip: "
  ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down"
done

6. Apply Talos Configuration to All Nodes

talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.1 --file configs/talos-cp1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.2 --file configs/talos-cp2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.3 --file configs/talos-cp3.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.4 --file configs/talos-worker1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.5 --file configs/talos-worker2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.6 --file configs/talos-worker3.yaml --insecure

7. Bootstrap the Cluster

# Set talosctl endpoints
talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3

# Bootstrap Kubernetes on the first control plane node
talosctl --talosconfig talosconfig bootstrap --nodes 10.0.15.1

8. Retrieve Kubeconfig

# Wait for cluster to initialize (about 60 seconds)
sleep 60

# Retrieve and merge kubeconfig
talosctl --talosconfig talosconfig kubeconfig --nodes 10.0.15.1 --force

# Switch to the cluster
kubectl config use-context admin@home-cluster

9. Verify Cluster

# Wait for all nodes to be Ready
kubectl get nodes -o wide

# Check all system pods are running
kubectl get pods -A

# Verify node names
kubectl get nodes
# Expected output:
# NAME            STATUS   ROLES           AGE   VERSION
# talos-cp1       Ready    control-plane   ...   v1.36.1
# talos-cp2       Ready    control-plane   ...   v1.36.1
# talos-cp3       Ready    control-plane   ...   v1.36.1
# talos-worker1   Ready    <none>          ...   v1.36.1
# talos-worker2   Ready    <none>          ...   v1.36.1
# talos-worker3   Ready    <none>          ...   v1.36.1

Important Files

• secrets.yaml - Cluster secrets (CRITICAL - keep secure and backed up!)
• talosconfig - Talosctl client configuration
• configs/talos-cp*.yaml - Control plane node configurations
• configs/talos-worker*.yaml - Worker node configurations
• ~/.kube/config - Contains merged kubeconfig with admin@home-cluster context

Key Lessons Learned

1. Hostname Configuration: Talos uses two places for hostname:

   • machine.network.hostname (for machine config)
   • HostnameConfig document (v1alpha1 config)
   • Only ONE should be set. Use HostnameConfig document with hostname: field for static names.

2. Template Considerations: The Talos template (VMID 9001) has HostnameConfig.auto: stable which auto-generates hostnames. When applying custom configs, this needs to be overridden.

3. Terraform Environment: Always source .envrc before running terraform commands to load backend configuration.

4. Bootstrap Process: Bootstrap must happen AFTER all control plane nodes have configs applied, but BEFORE retrieving kubeconfig.

5. Node Naming: Kubernetes node names come from the Talos hostname configuration, not VM names in Proxmox.

6. Recreating Individual VMs from Template: The OpenTofu VM definitions have been removed. To recreate a single VM directly on Proxmox:

   ssh root@<proxmox-node> "qm clone 9001 <VMID> --name <hostname> --full true --storage local-lvm"
   ssh root@<proxmox-node> "qm set <VMID> --cpu host --balloon 0 --numa 0 --sockets 1 --net0 virtio=<MAC>,bridge=vmbr0,firewall=0 --onboot 1 --agent enabled=1,fstrim_cloned_disks=0,type=virtio --vga std,memory=16"
   ssh root@<proxmox-node> "qm resize <VMID> scsi0 64G"
   ssh root@<proxmox-node> "qm start <VMID>"
   # Wait for DHCP, then apply config:
   talosctl --talosconfig talosconfig apply-config --nodes <IP> --file configs/<hostname>.yaml --insecure
   # Then upgrade to the factory image with extensions (iscsi-tools, util-linux-tools):
   talosctl --talosconfig talosconfig --nodes <IP> upgrade --image factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.13.3
   

   Critical VM settings: cpu: host is required — without it, Talos boots but never gets a network address. Also set balloon: 0, firewall=0 on NIC.

7. Bare-Metal Install Disk Enumeration: On bare-metal nodes booted from a Talos USB, the USB enumerates as /dev/sda and the internal drive as /dev/sdb. Set machine.install.disk: /dev/sdb in the config so Talos installs to the internal drive. After install, when the USB is removed, the internal drive becomes /dev/sda in the running system — this is expected and does not require reconfiguration. Example: talos4 (HP t620 with 128GB SK hynix M.2 SATA) was installed this way.

Troubleshooting

Nodes won't accept new config

If seeing "static hostname is already set" error:

• Ensure only HostnameConfig document has hostname, not machine.network.hostname
• Use --mode=reboot or recreate the VM entirely

Can't connect with talosctl

Set endpoints explicitly:

talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3

Nodes not appearing in kubectl

Wait 1-2 minutes after bootstrap. Check:

talosctl --talosconfig talosconfig service --nodes 10.0.15.1
# Look for kubelet, etcd, apid as "Running"

Networking

• CNI: Flannel (deployed automatically)
• Service CIDR: 10.96.0.0/12
• Pod CIDR: 10.244.0.0/16
• DNS: CoreDNS (cluster.local domain)

Storage

• VM boot disks: local-lvm on each Proxmox node (no ceph storage available)

Maintenance

Adding a Node

1. Add VM definition to terraform/talos_vms.tf
2. Run tofu apply
3. Generate config with appropriate hostname
4. Apply config: talosctl apply-config --nodes <IP> --file <config>.yaml --insecure

Upgrading Talos

talosctl --talosconfig talosconfig upgrade --image ghcr.io/siderolabs/installer:v1.13.3

Upgrading Kubernetes

talosctl --talosconfig talosconfig upgrade-k8s --to 1.36.1

Backup Strategy

Essential files to backup:

1. secrets.yaml - Cannot recover cluster without this
2. talosconfig - Required for talosctl access
3. Node configs in configs/ directory
4. Terraform state (stored in GitLab HTTP backend)

Store securely in password manager or encrypted backup location.