graham/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
infra/old/cluster/deploy.md

13 KiB
Raw Blame History

Cluster Deployment Guide

This document tracks the commands and process for provisioning the k3s cluster with Talos.

Initial Talos Setup

Fresh Talos cluster has been set up. Document deployment commands below as they are executed.

Deployment Commands Log

Latest APRS Deployment (2025-10-18)

# Generate kubeconfig from Talos
talosctl --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 kubeconfig

# Force re-deploy APRS.me with new GitHub pull token
cd /Users/graham/dev/infra/home/cluster/aprs
./deploy.sh

# Check pod status
kubectl get pods -n aprs -o wide

# Force restart of APRS StatefulSet to pull new image with GitHub token
kubectl delete pod aprs-0 -n aprs
kubectl rollout restart statefulset/aprs -n aprs

# Check persistent volume claim status
kubectl get pvc -n aprs
kubectl get storageclass

# Check cluster nodes
kubectl get nodes -o wide

Status: Image pull now working with GitHub token, but PostgreSQL stuck due to missing storage class

Storage Configuration (2025-10-18)

# Initially tried Longhorn but had iscsi-tools issues in Talos
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml
kubectl label namespace longhorn-system pod-security.kubernetes.io/enforce=privileged --overwrite
kubectl rollout restart ds/longhorn-manager -n longhorn-system

# Installed local-path-provisioner as alternative
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.28/deploy/local-path-storage.yaml
kubectl label namespace local-path-storage pod-security.kubernetes.io/enforce=privileged --overwrite
kubectl rollout restart deployment/local-path-provisioner -n local-path-storage

# Updated PostgreSQL to use local-path with 256GB storage
# Modified postgres-deployment.yaml: storageClassName: local-path, storage: 256Gi
kubectl delete pvc postgres-storage-postgres-aprs-0 -n aprs
kubectl delete statefulset postgres-aprs -n aprs
kubectl apply -f /Users/graham/dev/infra/home/cluster/aprs/postgres-deployment.yaml

# Verify storage configuration
kubectl get pvc -n aprs
kubectl get storageclass

Status: PostgreSQL with PostGIS now configured with 256GB local-path storage, APRS.me deploying with GitHub token

Longhorn Storage Final Configuration (2025-10-18)

# Updated controlplane.yaml to add kubelet extraMounts for Longhorn
# Added:
#   extraMounts:
#     - destination: /var/mnt/longhorn
#       type: bind
#       source: /var/mnt/longhorn
#       options: [bind, rshared, rw]

# Applied updated configuration to both nodes
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --file controlplane.yaml --config-patch @node1.patch.yaml --config-patch @tailscale.patch.yaml
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.22 --nodes 10.0.101.22 --file controlplane.yaml --config-patch @node2.patch.yaml --config-patch @tailscale.patch.yaml

# Generated custom Talos image with iscsi-tools extension
curl -X POST --data-binary @schematic.yaml https://factory.talos.dev/schematics
# Returned schematic ID: 077514df2c1b6436460bc60faabc976687b16193b8a1290fda4366c69024fec2

# Upgraded both nodes with custom image including iscsi-tools
talosctl upgrade --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --image factory.talos.dev/metal-installer/708747e350d604ae9e57227d8dcf274091453ddb1097b765d4ea8884f1992c1f:v1.11.3 --preserve --force
talosctl upgrade --talosconfig=./talosconfig -e 10.0.101.22 --nodes 10.0.101.22 --image factory.talos.dev/metal-installer/708747e350d604ae9e57227d8dcf274091453ddb1097b765d4ea8884f1992c1f:v1.11.3 --preserve --force

# Reinstalled Longhorn after node upgrade
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml
kubectl label namespace longhorn-system pod-security.kubernetes.io/enforce=privileged --overwrite

# Configured PostgreSQL to use Longhorn storage with 256GB
# Modified postgres-deployment.yaml: storageClassName: longhorn, storage: 256Gi
kubectl delete pvc postgres-storage-postgres-aprs-0 -n aprs
kubectl delete statefulset postgres-aprs -n aprs
kubectl apply -f /Users/graham/dev/infra/home/cluster/aprs/postgres-deployment.yaml

# Verify Longhorn storage
kubectl get storageclass
kubectl get nodes.longhorn.io -n longhorn-system
kubectl get pvc -n aprs

Status: Longhorn storage successfully configured on both nodes, PostgreSQL with PostGIS using 256GB Longhorn storage

APRS.me Application Deployment Success (2025-10-19)

# Fixed PostgreSQL service to use ClusterIP instead of headless service
# Modified postgres-deployment.yaml: type: ClusterIP
kubectl delete svc postgres-aprs -n aprs
kubectl apply -f /Users/graham/dev/infra/home/cluster/aprs/postgres-deployment.yaml

# Fixed PostgreSQL permissions with fsGroup
# Added to postgres-deployment.yaml:
#   securityContext:
#     fsGroup: 999
#     fsGroupChangePolicy: OnRootMismatch

# Shortened PostgreSQL readiness probe delays
# Updated readiness probe: initialDelaySeconds: 30, periodSeconds: 10, failureThreshold: 3

# Fixed APRS container permissions 
# Removed strict runAsUser constraints from aprs-statefulset-with-pullsecret.yaml
# Added writable volumes for tmp directories

# Set APRS namespace to privileged pod security
kubectl label namespace aprs pod-security.kubernetes.io/enforce=privileged --overwrite

# Final deployment verification
kubectl get pods -n aprs
kubectl get svc -n aprs
kubectl logs aprs-0 -n aprs -c aprs

Status: ✅ APRS.me application fully deployed and operational

  • PostgreSQL 17 with PostGIS running on 256GB Longhorn storage
  • Redis cache operational
  • APRS.me web application responding on port 4000
  • APRS-IS packet receiver active and processing
  • Multi-replica deployment with Erlang clustering
  • All services healthy and ready

MetalLB LoadBalancer Configuration (2025-10-19)

# Install MetalLB for LoadBalancer services
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml
kubectl wait --for=condition=ready pod -l app=metallb -n metallb-system --timeout=60s

# Create MetalLB configuration with public IP
cat > /Users/graham/dev/infra/home/cluster/metallb-config.yaml << 'EOF'
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: public-ip-pool
  namespace: metallb-system
spec:
  addresses:
  - 204.110.191.2/32
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: public-ip-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - public-ip-pool
EOF

# Apply MetalLB configuration
kubectl apply -f /Users/graham/dev/infra/home/cluster/metallb-config.yaml

Status: ✅ LoadBalancer services now working with MetalLB

  • Traefik assigned external IP 204.110.191.2
  • HTTPS access to aprs.me working correctly
  • HTTP redirects to HTTPS automatically

Attempted APRS Deployment

# Attempted to deploy APRS application
cd /Users/graham/dev/infra/home/cluster/aprs
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" ./deploy.sh

Issue: Kubernetes API at 10.0.101.21:6443 not accessible - connection timeout.
Current Context: admin@home
Status: Deployment blocked - need cluster connectivity

Network Configuration Fix

# Fixed node1.patch.yaml to remove conflicting default route from VLAN 10
# VLAN 10 now only has IP assignment, main interface handles routing

Issue: Dual default routes caused connectivity loss
Fix: Removed default route from VLAN 9 configuration to prevent dual route conflicts

Security Context Fixes for PodSecurity Policy

# Fixed PostgreSQL deployment security context
kubectl apply -f postgres-deployment.yaml

# Fixed Redis deployment security context  
kubectl apply -f redis.yaml

# Fixed APRS StatefulSet security context (migrate + aprs containers)
kubectl apply -f aprs-statefulset-with-pullsecret.yaml

Node1 Public IP Configuration

# Updated node1.patch.yaml with public IP 204.110.191.2/27
# Gateway: 204.110.191.30 on VLAN 9
# Applied node1 configuration with public IP
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --file controlplane.yaml --config-patch @node1.patch.yaml --config-patch @tailscale.patch.yaml

Cert-Manager Deployment

# Check if cert-manager is installed
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl get pods -n cert-manager

# Install cert-manager v1.16.1
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml

# Wait for cert-manager to be ready
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=cert-manager -n cert-manager --timeout=300s

# Deploy Let's Encrypt ClusterIssuer (had webhook timeout issues)
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl delete validatingwebhookconfiguration cert-manager-webhook
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f cert-manager/letsencrypt-prod.yaml
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml

Traefik Deployment

# Create traefik namespace
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl create namespace traefik-system

# Install Traefik via Helm with LoadBalancer
helm repo update
helm install traefik traefik/traefik --namespace traefik-system --set service.type=LoadBalancer

# Wait for Traefik to be ready
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=traefik -n traefik-system --timeout=300s

# Deploy APRS IngressRoute and Certificate (temporarily removed webhook)
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl delete validatingwebhookconfiguration cert-manager-webhook
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f aprs/aprs-ingressroute.yaml
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml

Longhorn Storage Deployment

# Install Longhorn v1.7.1
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.1/deploy/longhorn.yaml

# Set privileged security policy for Longhorn namespace (required for hostPath volumes)
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl label namespace longhorn-system pod-security.kubernetes.io/enforce=privileged

# Restart DaemonSet to apply new security policy
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl rollout restart ds/longhorn-manager -n longhorn-system

# Note: Longhorn may take several minutes to fully initialize and create storage class

DNS Configuration Update

# Updated aprs.me DNS records to point to new cluster IP
# File: home/terraform/dns_aprs_me_cloudflare.tf
# Changed from 204.110.191.3 to 204.110.191.2

# Verify DNS changes (records already applied correctly)
tofu -chdir=/Users/graham/dev/infra/home/terraform plan -target=cloudflare_record.aprs_me_root -target=cloudflare_record.aprs_me_www
tofu -chdir=/Users/graham/dev/infra/home/terraform state show cloudflare_record.aprs_me_root

# Confirmed: Both aprs.me and www.aprs.me point to 204.110.191.2

Node-RED Deployment (2025-10-19)

# Added 204.110.191.3 to node1 configuration
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --file controlplane.yaml --config-patch @node1.patch.yaml --config-patch @tailscale.patch.yaml

# Deployed nginx proxy for multiple services on 204.110.191.3:8080/8443
cd /Users/graham/dev/infra/home/cluster/nginx-proxy
./deploy.sh

# Deployed Node-RED to cluster
cd /Users/graham/dev/infra/home/cluster/nodered
./deploy.sh

# Updated DNS for nodered.w5isp.com to point to 204.110.191.3
tofu -chdir=/Users/graham/dev/infra/home/terraform apply -target=dnsimple_zone_record.w5isp_nodered -auto-approve

Status: ✅ Node-RED deployed and accessible

  • Traefik handles routing and SSL termination via host-based routing
  • APRS: https://aprs.me (204.110.191.2 via Traefik)
  • Node-RED: https://nodered.w5isp.com (204.110.191.2 via Traefik)
  • Both services use ClusterIP (internal only)
  • Uses Longhorn for persistent storage

Key Configuration Files

  • controlplane.yaml - Talos control plane configuration
  • worker.yaml - Talos worker node configuration
  • node1.patch.yaml, node2.patch.yaml, node3.patch.yaml - Node-specific patches
  • talosconfig - Talos client configuration

Notes

  • Cluster uses Talos Linux as the operating system
  • Applications deployed via kubectl with manifests in subdirectories
  • Each application has its own deploy.sh script for easy deployment

Reproduction Steps

  1. Apply Talos configurations to nodes
  2. Bootstrap cluster
  3. Deploy applications as needed

Detailed steps will be added as deployment progresses.