graham/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
infra/old/cluster/deploy.md

335 lines
No EOL
13 KiB
Markdown
Raw Blame History

# Cluster Deployment Guide
This document tracks the commands and process for provisioning the k3s cluster with Talos.
## Initial Talos Setup
Fresh Talos cluster has been set up. Document deployment commands below as they are executed.
## Deployment Commands Log
### Latest APRS Deployment (2025-10-18)
```bash
# Generate kubeconfig from Talos
talosctl --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 kubeconfig
# Force re-deploy APRS.me with new GitHub pull token
cd /Users/graham/dev/infra/home/cluster/aprs
./deploy.sh
# Check pod status
kubectl get pods -n aprs -o wide
# Force restart of APRS StatefulSet to pull new image with GitHub token
kubectl delete pod aprs-0 -n aprs
kubectl rollout restart statefulset/aprs -n aprs
# Check persistent volume claim status
kubectl get pvc -n aprs
kubectl get storageclass
# Check cluster nodes
kubectl get nodes -o wide
```
**Status**: Image pull now working with GitHub token, but PostgreSQL stuck due to missing storage class
### Storage Configuration (2025-10-18)
```bash
# Initially tried Longhorn but had iscsi-tools issues in Talos
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml
kubectl label namespace longhorn-system pod-security.kubernetes.io/enforce=privileged --overwrite
kubectl rollout restart ds/longhorn-manager -n longhorn-system
# Installed local-path-provisioner as alternative
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.28/deploy/local-path-storage.yaml
kubectl label namespace local-path-storage pod-security.kubernetes.io/enforce=privileged --overwrite
kubectl rollout restart deployment/local-path-provisioner -n local-path-storage
# Updated PostgreSQL to use local-path with 256GB storage
# Modified postgres-deployment.yaml: storageClassName: local-path, storage: 256Gi
kubectl delete pvc postgres-storage-postgres-aprs-0 -n aprs
kubectl delete statefulset postgres-aprs -n aprs
kubectl apply -f /Users/graham/dev/infra/home/cluster/aprs/postgres-deployment.yaml
# Verify storage configuration
kubectl get pvc -n aprs
kubectl get storageclass
```
**Status**: PostgreSQL with PostGIS now configured with 256GB local-path storage, APRS.me deploying with GitHub token
### Longhorn Storage Final Configuration (2025-10-18)
```bash
# Updated controlplane.yaml to add kubelet extraMounts for Longhorn
# Added:
# extraMounts:
# - destination: /var/mnt/longhorn
# type: bind
# source: /var/mnt/longhorn
# options: [bind, rshared, rw]
# Applied updated configuration to both nodes
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --file controlplane.yaml --config-patch @node1.patch.yaml --config-patch @tailscale.patch.yaml
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.22 --nodes 10.0.101.22 --file controlplane.yaml --config-patch @node2.patch.yaml --config-patch @tailscale.patch.yaml
# Generated custom Talos image with iscsi-tools extension
curl -X POST --data-binary @schematic.yaml https://factory.talos.dev/schematics
# Returned schematic ID: 077514df2c1b6436460bc60faabc976687b16193b8a1290fda4366c69024fec2
# Upgraded both nodes with custom image including iscsi-tools
talosctl upgrade --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --image factory.talos.dev/metal-installer/708747e350d604ae9e57227d8dcf274091453ddb1097b765d4ea8884f1992c1f:v1.11.3 --preserve --force
talosctl upgrade --talosconfig=./talosconfig -e 10.0.101.22 --nodes 10.0.101.22 --image factory.talos.dev/metal-installer/708747e350d604ae9e57227d8dcf274091453ddb1097b765d4ea8884f1992c1f:v1.11.3 --preserve --force
# Reinstalled Longhorn after node upgrade
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml
kubectl label namespace longhorn-system pod-security.kubernetes.io/enforce=privileged --overwrite
# Configured PostgreSQL to use Longhorn storage with 256GB
# Modified postgres-deployment.yaml: storageClassName: longhorn, storage: 256Gi
kubectl delete pvc postgres-storage-postgres-aprs-0 -n aprs
kubectl delete statefulset postgres-aprs -n aprs
kubectl apply -f /Users/graham/dev/infra/home/cluster/aprs/postgres-deployment.yaml
# Verify Longhorn storage
kubectl get storageclass
kubectl get nodes.longhorn.io -n longhorn-system
kubectl get pvc -n aprs
```
**Status**: Longhorn storage successfully configured on both nodes, PostgreSQL with PostGIS using 256GB Longhorn storage
### APRS.me Application Deployment Success (2025-10-19)
```bash
# Fixed PostgreSQL service to use ClusterIP instead of headless service
# Modified postgres-deployment.yaml: type: ClusterIP
kubectl delete svc postgres-aprs -n aprs
kubectl apply -f /Users/graham/dev/infra/home/cluster/aprs/postgres-deployment.yaml
# Fixed PostgreSQL permissions with fsGroup
# Added to postgres-deployment.yaml:
# securityContext:
# fsGroup: 999
# fsGroupChangePolicy: OnRootMismatch
# Shortened PostgreSQL readiness probe delays
# Updated readiness probe: initialDelaySeconds: 30, periodSeconds: 10, failureThreshold: 3
# Fixed APRS container permissions
# Removed strict runAsUser constraints from aprs-statefulset-with-pullsecret.yaml
# Added writable volumes for tmp directories
# Set APRS namespace to privileged pod security
kubectl label namespace aprs pod-security.kubernetes.io/enforce=privileged --overwrite
# Final deployment verification
kubectl get pods -n aprs
kubectl get svc -n aprs
kubectl logs aprs-0 -n aprs -c aprs
```
**Status**: ✅ APRS.me application fully deployed and operational
- PostgreSQL 17 with PostGIS running on 256GB Longhorn storage
- Redis cache operational
- APRS.me web application responding on port 4000
- APRS-IS packet receiver active and processing
- Multi-replica deployment with Erlang clustering
- All services healthy and ready
### MetalLB LoadBalancer Configuration (2025-10-19)
```bash
# Install MetalLB for LoadBalancer services
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml
kubectl wait --for=condition=ready pod -l app=metallb -n metallb-system --timeout=60s
# Create MetalLB configuration with public IP
cat > /Users/graham/dev/infra/home/cluster/metallb-config.yaml << 'EOF'
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: public-ip-pool
namespace: metallb-system
spec:
addresses:
- 204.110.191.2/32
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: public-ip-advertisement
namespace: metallb-system
spec:
ipAddressPools:
- public-ip-pool
EOF
# Apply MetalLB configuration
kubectl apply -f /Users/graham/dev/infra/home/cluster/metallb-config.yaml
```
**Status**: ✅ LoadBalancer services now working with MetalLB
- Traefik assigned external IP 204.110.191.2
- HTTPS access to aprs.me working correctly
- HTTP redirects to HTTPS automatically
### Attempted APRS Deployment
```bash
# Attempted to deploy APRS application
cd /Users/graham/dev/infra/home/cluster/aprs
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" ./deploy.sh
```
**Issue**: Kubernetes API at 10.0.101.21:6443 not accessible - connection timeout.
**Current Context**: admin@home
**Status**: Deployment blocked - need cluster connectivity
### Network Configuration Fix
```bash
# Fixed node1.patch.yaml to remove conflicting default route from VLAN 10
# VLAN 10 now only has IP assignment, main interface handles routing
```
**Issue**: Dual default routes caused connectivity loss
**Fix**: Removed default route from VLAN 9 configuration to prevent dual route conflicts
### Security Context Fixes for PodSecurity Policy
```bash
# Fixed PostgreSQL deployment security context
kubectl apply -f postgres-deployment.yaml
# Fixed Redis deployment security context
kubectl apply -f redis.yaml
# Fixed APRS StatefulSet security context (migrate + aprs containers)
kubectl apply -f aprs-statefulset-with-pullsecret.yaml
```
### Node1 Public IP Configuration
```bash
# Updated node1.patch.yaml with public IP 204.110.191.2/27
# Gateway: 204.110.191.30 on VLAN 9
# Applied node1 configuration with public IP
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --file controlplane.yaml --config-patch @node1.patch.yaml --config-patch @tailscale.patch.yaml
```
### Cert-Manager Deployment
```bash
# Check if cert-manager is installed
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl get pods -n cert-manager
# Install cert-manager v1.16.1
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml
# Wait for cert-manager to be ready
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=cert-manager -n cert-manager --timeout=300s
# Deploy Let's Encrypt ClusterIssuer (had webhook timeout issues)
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl delete validatingwebhookconfiguration cert-manager-webhook
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f cert-manager/letsencrypt-prod.yaml
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml
```
### Traefik Deployment
```bash
# Create traefik namespace
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl create namespace traefik-system
# Install Traefik via Helm with LoadBalancer
helm repo update
helm install traefik traefik/traefik --namespace traefik-system --set service.type=LoadBalancer
# Wait for Traefik to be ready
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=traefik -n traefik-system --timeout=300s
# Deploy APRS IngressRoute and Certificate (temporarily removed webhook)
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl delete validatingwebhookconfiguration cert-manager-webhook
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f aprs/aprs-ingressroute.yaml
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml
```
### Longhorn Storage Deployment
```bash
# Install Longhorn v1.7.1
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.1/deploy/longhorn.yaml
# Set privileged security policy for Longhorn namespace (required for hostPath volumes)
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl label namespace longhorn-system pod-security.kubernetes.io/enforce=privileged
# Restart DaemonSet to apply new security policy
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl rollout restart ds/longhorn-manager -n longhorn-system
# Note: Longhorn may take several minutes to fully initialize and create storage class
```
### DNS Configuration Update
```bash
# Updated aprs.me DNS records to point to new cluster IP
# File: home/terraform/dns_aprs_me_cloudflare.tf
# Changed from 204.110.191.3 to 204.110.191.2
# Verify DNS changes (records already applied correctly)
tofu -chdir=/Users/graham/dev/infra/home/terraform plan -target=cloudflare_record.aprs_me_root -target=cloudflare_record.aprs_me_www
tofu -chdir=/Users/graham/dev/infra/home/terraform state show cloudflare_record.aprs_me_root
# Confirmed: Both aprs.me and www.aprs.me point to 204.110.191.2
```
### Node-RED Deployment (2025-10-19)
```bash
# Added 204.110.191.3 to node1 configuration
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --file controlplane.yaml --config-patch @node1.patch.yaml --config-patch @tailscale.patch.yaml
# Deployed nginx proxy for multiple services on 204.110.191.3:8080/8443
cd /Users/graham/dev/infra/home/cluster/nginx-proxy
./deploy.sh
# Deployed Node-RED to cluster
cd /Users/graham/dev/infra/home/cluster/nodered
./deploy.sh
# Updated DNS for nodered.w5isp.com to point to 204.110.191.3
tofu -chdir=/Users/graham/dev/infra/home/terraform apply -target=dnsimple_zone_record.w5isp_nodered -auto-approve
```
**Status**: ✅ Node-RED deployed and accessible
- Traefik handles routing and SSL termination via host-based routing
- APRS: https://aprs.me (204.110.191.2 via Traefik)
- Node-RED: https://nodered.w5isp.com (204.110.191.2 via Traefik)
- Both services use ClusterIP (internal only)
- Uses Longhorn for persistent storage
## Key Configuration Files
- `controlplane.yaml` - Talos control plane configuration
- `worker.yaml` - Talos worker node configuration
- `node1.patch.yaml`, `node2.patch.yaml`, `node3.patch.yaml` - Node-specific patches
- `talosconfig` - Talos client configuration
## Notes
- Cluster uses Talos Linux as the operating system
- Applications deployed via kubectl with manifests in subdirectories
- Each application has its own deploy.sh script for easy deployment
## Reproduction Steps
1. Apply Talos configurations to nodes
2. Bootstrap cluster
3. Deploy applications as needed
_Detailed steps will be added as deployment progresses._
Reference in a new issue View git blame Copy permalink