infra/talos/CLAUDE.md

# Talos Cluster Setup Guide

This document contains notes on how the Talos home cluster was created and how to recreate it in the future.

## Cluster Information

- **Cluster Name**: home-cluster
- **Kubernetes Version**: v1.35.0
- **Talos Version**: v1.12.1
- **Control Plane Nodes**: 3 (10.0.15.1-3)
- **Worker Nodes**: 3 (10.0.15.4-6)
- **API Endpoint**: https://10.0.15.1:6443

## Node Details

| Hostname | Role | IP | Proxmox Node | VMID | MAC Address |
|----------|------|-----|--------------|------|-------------|
| talos-cp1 | Control Plane | 10.0.15.1 | node1 | 200 | BC:24:11:9B:48:92 |
| talos-cp2 | Control Plane | 10.0.15.2 | node2 | 201 | BC:24:11:62:7B:3F |
| talos-cp3 | Control Plane | 10.0.15.3 | node3 | 202 | BC:24:11:D4:2F:ED |
| talos-worker1 | Worker | 10.0.15.4 | node1 | 210 | BC:24:11:43:3F:FF |
| talos-worker2 | Worker | 10.0.15.5 | node2 | 211 | BC:24:11:62:C4:8F |
| talos-worker3 | Worker | 10.0.15.6 | node3 | 212 | BC:24:11:3F:8E:1A |

## Prerequisites

- Proxmox home cluster with nodes: node1, node2, node3 (10.0.15.101-103)
- Talos Linux template at VMID 9001 on node1
- DHCP static leases configured for all MAC addresses above
- Terraform/OpenTofu configured with Proxmox provider
- talosctl CLI installed

## Complete Cluster Creation Process

### 1. Generate New Cluster Secrets

```bash
cd /Users/graham/dev/infra/talos
talosctl gen secrets -o secrets.yaml --force
```

### 2. Generate Node-Specific Configurations

Each node needs a configuration with its specific hostname. Use patch files to set hostnames:

```bash
mkdir -p configs

# Generate control plane configs
for i in 1 2 3; do
  cat > /tmp/cp${i}-patch.yaml <<EOF
machine:
  network:
    hostname: talos-cp${i}
EOF
  talosctl gen config home-cluster https://10.0.15.1:6443 \
    --with-secrets secrets.yaml \
    --kubernetes-version v1.35.0 \
    --output-types controlplane \
    --output configs/talos-cp${i}.yaml \
    --config-patch /tmp/cp${i}-patch.yaml
done

# Generate worker configs
for i in 1 2 3; do
  cat > /tmp/w${i}-patch.yaml <<EOF
machine:
  network:
    hostname: talos-worker${i}
EOF
  talosctl gen config home-cluster https://10.0.15.1:6443 \
    --with-secrets secrets.yaml \
    --kubernetes-version v1.35.0 \
    --output-types worker \
    --output configs/talos-worker${i}.yaml \
    --config-patch /tmp/w${i}-patch.yaml
done
```

### 3. Fix HostnameConfig Sections

The generated configs will have both `machine.network.hostname` and a `HostnameConfig` document. Fix this:

```bash
# Remove hostname from machine.network section
for file in configs/*.yaml; do
  sed -i '' '/^    network:$/,/^        # # Configures KubeSpan/{
    /^        hostname:/d
  }' "$file"
done

# Update HostnameConfig sections
sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-cp1
}' configs/talos-cp1.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-cp2
}' configs/talos-cp2.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-cp3
}' configs/talos-cp3.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-worker1
}' configs/talos-worker1.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-worker2
}' configs/talos-worker2.yaml

sed -i '' '/^apiVersion: v1alpha1$/,/^# hostname:.*/{
  /^kind: HostnameConfig$/,/^# hostname:.*/c\
kind: HostnameConfig\
hostname: talos-worker3
}' configs/talos-worker3.yaml
```

### 4. Deploy VMs via Terraform

```bash
cd /Users/graham/dev/infra/terraform
source .envrc  # Load environment variables

# If recreating, destroy old VMs first
tofu destroy \
  -target=proxmox_virtual_environment_vm.talos_cp1 \
  -target=proxmox_virtual_environment_vm.talos_cp2 \
  -target=proxmox_virtual_environment_vm.talos_cp3 \
  -target=proxmox_virtual_environment_vm.talos_worker1 \
  -target=proxmox_virtual_environment_vm.talos_worker2 \
  -target=proxmox_virtual_environment_vm.talos_worker3 \
  -auto-approve

# Create new VMs
tofu apply \
  -target=proxmox_virtual_environment_vm.talos_cp1 \
  -target=proxmox_virtual_environment_vm.talos_cp2 \
  -target=proxmox_virtual_environment_vm.talos_cp3 \
  -target=proxmox_virtual_environment_vm.talos_worker1 \
  -target=proxmox_virtual_environment_vm.talos_worker2 \
  -target=proxmox_virtual_environment_vm.talos_worker3 \
  -auto-approve
```

### 5. Wait for VMs to Boot

```bash
cd /Users/graham/dev/infra/talos
sleep 30

# Verify all nodes are up
for ip in 10.0.15.{1..6}; do
  echo -n "$ip: "
  ping -c 1 -W 2 $ip > /dev/null 2>&1 && echo "up" || echo "down"
done
```

### 6. Apply Talos Configuration to All Nodes

```bash
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.1 --file configs/talos-cp1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.2 --file configs/talos-cp2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.3 --file configs/talos-cp3.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.4 --file configs/talos-worker1.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.5 --file configs/talos-worker2.yaml --insecure
talosctl --talosconfig talosconfig apply-config --nodes 10.0.15.6 --file configs/talos-worker3.yaml --insecure
```

### 7. Bootstrap the Cluster

```bash
# Set talosctl endpoints
talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3

# Bootstrap Kubernetes on the first control plane node
talosctl --talosconfig talosconfig bootstrap --nodes 10.0.15.1
```

### 8. Retrieve Kubeconfig

```bash
# Wait for cluster to initialize (about 60 seconds)
sleep 60

# Retrieve and merge kubeconfig
talosctl --talosconfig talosconfig kubeconfig --nodes 10.0.15.1 --force

# Switch to the cluster
kubectl config use-context admin@home-cluster
```

### 9. Verify Cluster

```bash
# Wait for all nodes to be Ready
kubectl get nodes -o wide

# Check all system pods are running
kubectl get pods -A

# Verify node names
kubectl get nodes
# Expected output:
# NAME            STATUS   ROLES           AGE   VERSION
# talos-cp1       Ready    control-plane   ...   v1.35.0
# talos-cp2       Ready    control-plane   ...   v1.35.0
# talos-cp3       Ready    control-plane   ...   v1.35.0
# talos-worker1   Ready    <none>          ...   v1.35.0
# talos-worker2   Ready    <none>          ...   v1.35.0
# talos-worker3   Ready    <none>          ...   v1.35.0
```

## Important Files

- `secrets.yaml` - Cluster secrets (CRITICAL - keep secure and backed up!)
- `talosconfig` - Talosctl client configuration
- `configs/talos-cp*.yaml` - Control plane node configurations
- `configs/talos-worker*.yaml` - Worker node configurations
- `~/.kube/config` - Contains merged kubeconfig with admin@home-cluster context

## Key Lessons Learned

1. **Hostname Configuration**: Talos uses two places for hostname:
   - `machine.network.hostname` (for machine config)
   - `HostnameConfig` document (v1alpha1 config)
   - Only ONE should be set. Use `HostnameConfig` document with `hostname:` field for static names.

2. **Template Considerations**: The Talos template (VMID 9001) has `HostnameConfig.auto: stable` which auto-generates hostnames. When applying custom configs, this needs to be overridden.

3. **Terraform Environment**: Always `source .envrc` before running terraform commands to load backend configuration.

4. **Bootstrap Process**: Bootstrap must happen AFTER all control plane nodes have configs applied, but BEFORE retrieving kubeconfig.

5. **Node Naming**: Kubernetes node names come from the Talos hostname configuration, not VM names in Proxmox.

## Troubleshooting

### Nodes won't accept new config

If seeing "static hostname is already set" error:
- Ensure only `HostnameConfig` document has hostname, not `machine.network.hostname`
- Use `--mode=reboot` or recreate the VM entirely

### Can't connect with talosctl

Set endpoints explicitly:
```bash
talosctl --talosconfig talosconfig config endpoint 10.0.15.1 10.0.15.2 10.0.15.3
```

### Nodes not appearing in kubectl

Wait 1-2 minutes after bootstrap. Check:
```bash
talosctl --talosconfig talosconfig service --nodes 10.0.15.1
# Look for kubelet, etcd, apid as "Running"
```

## Networking

- **CNI**: Flannel (deployed automatically)
- **Service CIDR**: 10.96.0.0/12
- **Pod CIDR**: 10.244.0.0/16
- **DNS**: CoreDNS (cluster.local domain)

## Storage

Nodes use Proxmox Ceph for VM disks. For Kubernetes persistent storage, deploy:
- Ceph CSI driver (to use Proxmox Ceph directly)
- Or Longhorn for cluster-managed storage

## Maintenance

### Adding a Node

1. Add VM definition to `terraform/talos_vms.tf`
2. Run `tofu apply`
3. Generate config with appropriate hostname
4. Apply config: `talosctl apply-config --nodes <IP> --file <config>.yaml --insecure`

### Upgrading Talos

```bash
talosctl --talosconfig talosconfig upgrade --image ghcr.io/siderolabs/installer:v1.X.Y
```

### Upgrading Kubernetes

```bash
talosctl --talosconfig talosconfig upgrade-k8s --to 1.XX.0
```

## Backup Strategy

Essential files to backup:
1. `secrets.yaml` - Cannot recover cluster without this
2. `talosconfig` - Required for talosctl access
3. Node configs in `configs/` directory
4. Terraform state (stored in GitLab HTTP backend)

Store securely in password manager or encrypted backup location.