graham/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
infra/talos/CLUSTER_CONFIG.md

3.7 KiB
Raw Blame History

Talos Cluster Configuration Notes

Proxmox Environment

Network Configuration

Private Network (Cluster Internal)

  • Network: 10.0.0.0/24
  • Control Plane Nodes:
    • cp-1: 10.0.0.101 (DHCP)
    • cp-2: 10.0.0.102 (DHCP)
    • cp-3: 10.0.0.103 (DHCP)
  • Interface: eth0 (net0 in Proxmox, no VLAN tag)
  • Usage: Kubernetes API, etcd, internal pod communication

Public Network (External Services)

  • Network: 204.110.191.192/26
  • Gateway: 204.110.191.254
  • Node IPs:
    • cp-1: 204.110.191.193/26 (static)
    • cp-2: 204.110.191.194/26 (static)
    • cp-3: 204.110.191.195/26 (static)
  • Interface: eth1 (net1 in Proxmox, VLAN tag 9)
  • Usage: MetalLB LoadBalancer services, external access

MetalLB IP Pool

  • Range: 204.110.191.193-215
  • Interface: eth1
  • Mode: L2 Advertisement
  • IMPORTANT: LoadBalancer services must use externalTrafficPolicy: Cluster (not Local) to avoid routing issues with L2 mode

Required Node Labels

All nodes must have the following labels for proper CSI and scheduling:

# Topology labels for Proxmox CSI
topology.kubernetes.io/region=380
topology.kubernetes.io/zone=vm2-380

# Platform label for CSI DaemonSet
node.cloudprovider.kubernetes.io/platform=nocloud

Proxmox VM IDs

Map Kubernetes nodes to Proxmox VM IDs for CSI provider IDs:

  • talos-if6-e42: VM 203 → proxmox://380/203
  • talos-sbl-bvx: VM 202 → proxmox://380/202
  • talos-xgb-7n7: VM 201 → proxmox://380/201

Note: These mappings may change when VMs are recreated. Update proxmox-csi/node-providerid-patches.yaml accordingly.

Storage Configuration

  • StorageClass: proxmox-local-lvm
  • Backend: Proxmox local-lvm storage
  • Filesystem: ext4
  • Binding Mode: WaitForFirstConsumer
  • Allowed Topology: region=380

Service Deployment Order

  1. MetalLB - LoadBalancer capability for public IPs
  2. NFS Provisioner - Persistent storage via NFS (10.0.0.253:/mnt/k8s/cluster)
  3. Tailscale Operator - Private network access to services
  4. DNS Services - PowerDNS with external PostgreSQL
  5. Other services - As needed

External Dependencies

PostgreSQL Database

  • Host: 10.0.0.252
  • Database: powerdns
  • User: powerdns
  • Running on: Proxmox (outside Kubernetes)
  • Credentials: Stored in 1Password (item: t3nqhjmeci6msx4qouu64yfr3q)
  • PowerDNS Version: 4.8 (uses Primary/Secondary instead of Master/Slave)
  • Zone Type: All zones must be type Primary to support AXFR zone transfers
  • AXFR Settings:
    • allow-axfr-ips=204.87.183.53 - Allow zone transfers from secondary
    • also-notify=204.87.183.53 - Send notifications on zone updates

NFS Storage

  • Server: 10.0.0.253
  • Export: /mnt/k8s/cluster
  • Provider: TrueNAS
  • Configuration: Maproot user/group set to root

Tailscale

  • OAuth Credentials: Stored in 1Password (item: almzem6nffphe3xv4pgxpkbaga)
  • Namespace: tailscale
  • PodSecurity: Must be set to 'privileged' for proxy pods to run

Bootstrap Checklist

After recreating cluster:

  • Apply node configs via talosctl
  • Bootstrap cluster: talosctl -n 10.0.0.101 bootstrap
  • Get kubeconfig: talosctl kubeconfig --force
  • Remove control-plane taint: kubectl taint nodes --all node-role.kubernetes.io/control-plane:NoSchedule-
  • Deploy MetalLB for public LoadBalancer IPs
  • Deploy NFS provisioner (10.0.0.253:/mnt/k8s/cluster)
  • Deploy Tailscale operator:
    • Set tailscale namespace to privileged PodSecurity
    • Install operator with OAuth credentials from 1Password
  • Verify external PostgreSQL is running (10.0.0.252)
  • Deploy DNS services (PowerDNS)