graham/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
infra/talos/CLAUDE.md
Graham McIntire b951c77015
fix vntx cnames
2026-01-16 09:25:46 -06:00

9.4 KiB
Raw Blame History

Talos Cluster Setup Guide

This document contains notes on how the Talos home cluster was created and how to recreate it in the future.

Cluster Information

  • Cluster Name: home-cluster
  • Kubernetes Version: v1.35.0
  • Talos Version: v1.12.1
  • Control Plane Nodes: 3 (10.0.15.1-3)
  • Worker Nodes: 3 (10.0.15.4-6)
  • API Endpoint: https://10.0.15.1:6443

Node Details

Hostname Role IP Proxmox Node VMID MAC Address
talos-cp1 Control Plane 10.0.15.1 node1 200 BC:24:11:9B:48:92
talos-cp2 Control Plane 10.0.15.2 node2 201 BC:24:11:62:7B:3F
talos-cp3 Control Plane 10.0.15.3 node3 202 BC:24:11:D4:2F:ED
talos-worker1 Worker 10.0.15.4 node1 210 BC:24:11:43:3F:FF
talos-worker2 Worker 10.0.15.5 node2 211 BC:24:11:62:C4:8F
talos-worker3 Worker 10.0.15.6 node3 212 BC:24:11:3F:8E:1A

Prerequisites

  • Proxmox home cluster with nodes: node1, node2, node3 (10.0.15.101-103)
  • Talos Linux template at VMID 9001 on node1
  • DHCP static leases configured for all MAC addresses above
  • Terraform/OpenTofu configured with Proxmox provider
  • talosctl CLI installed

Complete Cluster Creation Process

1. Generate New Cluster Secrets

cd /Users/graham/dev/infra/talos
talosctl gen secrets -o secrets.yaml --force

2. Generate Node-Specific Configurations

Each node needs a configuration with its specific hostname. Use patch files to set hostnames:

mkdir -p configs

# Generate control plane configs
for i in 1 2 3; do
  cat > /tmp/cp${i}-patch.yaml <<EOF
machine:
  network:
    hostname: talos-cp${i}
EOF
  talosctl gen config home-cluster https://10.0.15.1:6443 \
    --with-secrets secrets.yaml \
    --kubernetes-version v1.35.0 \
    --output-types controlplane \
    --output configs/talos-cp${i}.yaml \
    --config-patch /tmp/cp${i}-patch.yaml
done

# Generate worker configs
for i in 1 2 3; do
  cat > /tmp/w${i}-patch.yaml <<EOF
machine:
  network:
    hostname: talos-worker${i}
EOF
  talosctl gen config home-cluster https://10.0.15.1:6443 \
    --with-secrets secrets.yaml \
    --kubernetes-version v1.35.0 \
    --output-types worker \
    --output configs/talos-worker${i}.yaml \
    --config-patch /tmp/w${i}-patch.yaml
done

3. Fix HostnameConfig Sections

The generated configs will have both machine.network.hostname and a HostnameConfig document. Fix this:

# Remove hostname from machine.network section
for file in configs/*.yaml; do
  sed -i '' '/^    network:$/,/^        # # Configures KubeSpan/{
    /^        hostname:/d
  }' "$file"
done

# Update HostnameConfig sections
sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-cp1
}' configs/talos-cp1.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-cp2
}' configs/talos-cp2.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-cp3
}' configs/talos-cp3.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-worker1
}' configs/talos-worker1.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-worker2
}' configs/talos-worker2.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-worker3
}' configs/talos-worker3.yaml

4. Deploy VMs via Terraform

cd /Users/graham/dev/infra/terraform
source .envrc  # Load environment variables

# If recreating, destroy old VMs first
tofu destroy \
  -target=proxmox_virtual_environment_vm.talos_cp1 \
  -target=proxmox_virtual_environment_vm.talos_cp2 \
  -target=proxmox_virtual_environment_vm.talos_cp3 \
  -target=proxmox_virtual_environment_vm.talos_worker1 \
  -target=proxmox_virtual_environment_vm.talos_worker2 \
  -target=proxmox_virtual_environment_vm.talos_worker3 \
  -auto-approve

# Create new VMs
tofu apply \
  -target=proxmox_virtual_environment_vm.talos_cp1 \
  -target=proxmox_virtual_environment_vm.talos_cp2 \
  -target=proxmox_virtual_environment_vm.talos_cp3 \
  -target=proxmox_virtual_environment_vm.talos_worker1 \
  -target=proxmox_virtual_environment_vm.talos_worker2 \
  -target=proxmox_virtual_environment_vm.talos_worker3 \
  -auto-approve

5. Wait for VMs to Boot

cd /Users/graham/dev/infra/talos
sleep 30

# Verify all nodes are up
for ip in 10.0.15.{1..6}; do
  echo -n "$ip: "
  ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down"
done

6. Apply Talos Configuration to All Nodes

talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.1 --file configs/talos-cp1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.2 --file configs/talos-cp2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.3 --file configs/talos-cp3.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.4 --file configs/talos-worker1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.5 --file configs/talos-worker2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.6 --file configs/talos-worker3.yaml --insecure

7. Bootstrap the Cluster

# Set talosctl endpoints
talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3

# Bootstrap Kubernetes on the first control plane node
talosctl --talosconfig talosconfig bootstrap --nodes 10.0.15.1

8. Retrieve Kubeconfig

# Wait for cluster to initialize (about 60 seconds)
sleep 60

# Retrieve and merge kubeconfig
talosctl --talosconfig talosconfig kubeconfig --nodes 10.0.15.1 --force

# Switch to the cluster
kubectl config use-context admin@home-cluster

9. Verify Cluster

# Wait for all nodes to be Ready
kubectl get nodes -o wide

# Check all system pods are running
kubectl get pods -A

# Verify node names
kubectl get nodes
# Expected output:
# NAME            STATUS   ROLES           AGE   VERSION
# talos-cp1       Ready    control-plane   ...   v1.35.0
# talos-cp2       Ready    control-plane   ...   v1.35.0
# talos-cp3       Ready    control-plane   ...   v1.35.0
# talos-worker1   Ready    <none>          ...   v1.35.0
# talos-worker2   Ready    <none>          ...   v1.35.0
# talos-worker3   Ready    <none>          ...   v1.35.0

Important Files

  • secrets.yaml - Cluster secrets (CRITICAL - keep secure and backed up!)
  • talosconfig - Talosctl client configuration
  • configs/talos-cp*.yaml - Control plane node configurations
  • configs/talos-worker*.yaml - Worker node configurations
  • ~/.kube/config - Contains merged kubeconfig with admin@home-cluster context

Key Lessons Learned

  1. Hostname Configuration: Talos uses two places for hostname:

    • machine.network.hostname (for machine config)
    • HostnameConfig document (v1alpha1 config)
    • Only ONE should be set. Use HostnameConfig document with hostname: field for static names.

  2. Template Considerations: The Talos template (VMID 9001) has HostnameConfig.auto: stable which auto-generates hostnames. When applying custom configs, this needs to be overridden.

  3. Terraform Environment: Always source .envrc before running terraform commands to load backend configuration.

  4. Bootstrap Process: Bootstrap must happen AFTER all control plane nodes have configs applied, but BEFORE retrieving kubeconfig.

  5. Node Naming: Kubernetes node names come from the Talos hostname configuration, not VM names in Proxmox.

Troubleshooting

Nodes won't accept new config

If seeing "static hostname is already set" error:

  • Ensure only HostnameConfig document has hostname, not machine.network.hostname
  • Use --mode=reboot or recreate the VM entirely

Can't connect with talosctl

Set endpoints explicitly:

talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3

Nodes not appearing in kubectl

Wait 1-2 minutes after bootstrap. Check:

talosctl --talosconfig talosconfig service --nodes 10.0.15.1
# Look for kubelet, etcd, apid as "Running"

Networking

  • CNI: Flannel (deployed automatically)
  • Service CIDR: 10.96.0.0/12
  • Pod CIDR: 10.244.0.0/16
  • DNS: CoreDNS (cluster.local domain)

Storage

Nodes use Proxmox Ceph for VM disks. For Kubernetes persistent storage, deploy:

  • Ceph CSI driver (to use Proxmox Ceph directly)
  • Or Longhorn for cluster-managed storage

Maintenance

Adding a Node

  1. Add VM definition to terraform/talos_vms.tf
  2. Run tofu apply
  3. Generate config with appropriate hostname
  4. Apply config: talosctl apply-config --nodes <IP> --file <config>.yaml --insecure

Upgrading Talos

talosctl --talosconfig talosconfig upgrade --image ghcr.io/siderolabs/installer:v1.X.Y

Upgrading Kubernetes

talosctl --talosconfig talosconfig upgrade-k8s --to 1.XX.0

Backup Strategy

Essential files to backup:

  1. secrets.yaml - Cannot recover cluster without this
  2. talosconfig - Required for talosctl access
  3. Node configs in configs/ directory
  4. Terraform state (stored in GitLab HTTP backend)

Store securely in password manager or encrypted backup location.