# Cluster Deployment Guide

This document tracks the commands and process for provisioning the k3s cluster with Talos.

## Initial Talos Setup

Fresh Talos cluster has been set up. Document deployment commands below as they are executed.

## Deployment Commands Log

### Latest APRS Deployment (2025-10-18)

```bash
# Generate kubeconfig from Talos
talosctl --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 kubeconfig

# Force re-deploy APRS.me with new GitHub pull token
cd /Users/graham/dev/infra/home/cluster/aprs
./deploy.sh

# Check pod status
kubectl get pods -n aprs -o wide

# Force restart of APRS StatefulSet to pull new image with GitHub token
kubectl delete pod aprs-0 -n aprs
kubectl rollout restart statefulset/aprs -n aprs

# Check persistent volume claim status
kubectl get pvc -n aprs
kubectl get storageclass

# Check cluster nodes
kubectl get nodes -o wide
```

**Status**: Image pull now working with GitHub token, but PostgreSQL stuck due to missing storage class

### Storage Configuration (2025-10-18)

```bash
# Initially tried Longhorn but had iscsi-tools issues in Talos
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml
kubectl label namespace longhorn-system pod-security.kubernetes.io/enforce=privileged --overwrite
kubectl rollout restart ds/longhorn-manager -n longhorn-system

# Installed local-path-provisioner as alternative
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.28/deploy/local-path-storage.yaml
kubectl label namespace local-path-storage pod-security.kubernetes.io/enforce=privileged --overwrite
kubectl rollout restart deployment/local-path-provisioner -n local-path-storage

# Updated PostgreSQL to use local-path with 256GB storage
# Modified postgres-deployment.yaml: storageClassName: local-path, storage: 256Gi
kubectl delete pvc postgres-storage-postgres-aprs-0 -n aprs
kubectl delete statefulset postgres-aprs -n aprs
kubectl apply -f /Users/graham/dev/infra/home/cluster/aprs/postgres-deployment.yaml

# Verify storage configuration
kubectl get pvc -n aprs
kubectl get storageclass
```

**Status**: PostgreSQL with PostGIS now configured with 256GB local-path storage, APRS.me deploying with GitHub token

### Longhorn Storage Final Configuration (2025-10-18)

```bash
# Updated controlplane.yaml to add kubelet extraMounts for Longhorn
# Added:
#   extraMounts:
#     - destination: /var/mnt/longhorn
#       type: bind
#       source: /var/mnt/longhorn
#       options: [bind, rshared, rw]

# Applied updated configuration to both nodes
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --file controlplane.yaml --config-patch @node1.patch.yaml --config-patch @tailscale.patch.yaml
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.22 --nodes 10.0.101.22 --file controlplane.yaml --config-patch @node2.patch.yaml --config-patch @tailscale.patch.yaml

# Generated custom Talos image with iscsi-tools extension
curl -X POST --data-binary @schematic.yaml https://factory.talos.dev/schematics
# Returned schematic ID: 077514df2c1b6436460bc60faabc976687b16193b8a1290fda4366c69024fec2

# Upgraded both nodes with custom image including iscsi-tools
talosctl upgrade --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --image factory.talos.dev/metal-installer/708747e350d604ae9e57227d8dcf274091453ddb1097b765d4ea8884f1992c1f:v1.11.3 --preserve --force
talosctl upgrade --talosconfig=./talosconfig -e 10.0.101.22 --nodes 10.0.101.22 --image factory.talos.dev/metal-installer/708747e350d604ae9e57227d8dcf274091453ddb1097b765d4ea8884f1992c1f:v1.11.3 --preserve --force

# Reinstalled Longhorn after node upgrade
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml
kubectl label namespace longhorn-system pod-security.kubernetes.io/enforce=privileged --overwrite

# Configured PostgreSQL to use Longhorn storage with 256GB
# Modified postgres-deployment.yaml: storageClassName: longhorn, storage: 256Gi
kubectl delete pvc postgres-storage-postgres-aprs-0 -n aprs
kubectl delete statefulset postgres-aprs -n aprs
kubectl apply -f /Users/graham/dev/infra/home/cluster/aprs/postgres-deployment.yaml

# Verify Longhorn storage
kubectl get storageclass
kubectl get nodes.longhorn.io -n longhorn-system
kubectl get pvc -n aprs
```

**Status**: Longhorn storage successfully configured on both nodes, PostgreSQL with PostGIS using 256GB Longhorn storage

### APRS.me Application Deployment Success (2025-10-19)

```bash
# Fixed PostgreSQL service to use ClusterIP instead of headless service
# Modified postgres-deployment.yaml: type: ClusterIP
kubectl delete svc postgres-aprs -n aprs
kubectl apply -f /Users/graham/dev/infra/home/cluster/aprs/postgres-deployment.yaml

# Fixed PostgreSQL permissions with fsGroup
# Added to postgres-deployment.yaml:
#   securityContext:
#     fsGroup: 999
#     fsGroupChangePolicy: OnRootMismatch

# Shortened PostgreSQL readiness probe delays
# Updated readiness probe: initialDelaySeconds: 30, periodSeconds: 10, failureThreshold: 3

# Fixed APRS container permissions 
# Removed strict runAsUser constraints from aprs-statefulset-with-pullsecret.yaml
# Added writable volumes for tmp directories

# Set APRS namespace to privileged pod security
kubectl label namespace aprs pod-security.kubernetes.io/enforce=privileged --overwrite

# Final deployment verification
kubectl get pods -n aprs
kubectl get svc -n aprs
kubectl logs aprs-0 -n aprs -c aprs
```

**Status**: ✅ APRS.me application fully deployed and operational
- PostgreSQL 17 with PostGIS running on 256GB Longhorn storage
- Redis cache operational
- APRS.me web application responding on port 4000
- APRS-IS packet receiver active and processing
- Multi-replica deployment with Erlang clustering
- All services healthy and ready

### MetalLB LoadBalancer Configuration (2025-10-19)

```bash
# Install MetalLB for LoadBalancer services
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml
kubectl wait --for=condition=ready pod -l app=metallb -n metallb-system --timeout=60s

# Create MetalLB configuration with public IP
cat > /Users/graham/dev/infra/home/cluster/metallb-config.yaml << 'EOF'
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: public-ip-pool
  namespace: metallb-system
spec:
  addresses:
  - 204.110.191.2/32
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: public-ip-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - public-ip-pool
EOF

# Apply MetalLB configuration
kubectl apply -f /Users/graham/dev/infra/home/cluster/metallb-config.yaml
```

**Status**: ✅ LoadBalancer services now working with MetalLB
- Traefik assigned external IP 204.110.191.2
- HTTPS access to aprs.me working correctly
- HTTP redirects to HTTPS automatically

### Attempted APRS Deployment

```bash
# Attempted to deploy APRS application
cd /Users/graham/dev/infra/home/cluster/aprs
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" ./deploy.sh
```

**Issue**: Kubernetes API at 10.0.101.21:6443 not accessible - connection timeout.  
**Current Context**: admin@home  
**Status**: Deployment blocked - need cluster connectivity

### Network Configuration Fix

```bash
# Fixed node1.patch.yaml to remove conflicting default route from VLAN 10
# VLAN 10 now only has IP assignment, main interface handles routing
```

**Issue**: Dual default routes caused connectivity loss  
**Fix**: Removed default route from VLAN 9 configuration to prevent dual route conflicts

### Security Context Fixes for PodSecurity Policy

```bash
# Fixed PostgreSQL deployment security context
kubectl apply -f postgres-deployment.yaml

# Fixed Redis deployment security context  
kubectl apply -f redis.yaml

# Fixed APRS StatefulSet security context (migrate + aprs containers)
kubectl apply -f aprs-statefulset-with-pullsecret.yaml
```

### Node1 Public IP Configuration

```bash
# Updated node1.patch.yaml with public IP 204.110.191.2/27
# Gateway: 204.110.191.30 on VLAN 9
# Applied node1 configuration with public IP
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --file controlplane.yaml --config-patch @node1.patch.yaml --config-patch @tailscale.patch.yaml
```

### Cert-Manager Deployment

```bash
# Check if cert-manager is installed
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl get pods -n cert-manager

# Install cert-manager v1.16.1
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml

# Wait for cert-manager to be ready
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=cert-manager -n cert-manager --timeout=300s

# Deploy Let's Encrypt ClusterIssuer (had webhook timeout issues)
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl delete validatingwebhookconfiguration cert-manager-webhook
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f cert-manager/letsencrypt-prod.yaml
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml
```

### Traefik Deployment

```bash
# Create traefik namespace
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl create namespace traefik-system

# Install Traefik via Helm with LoadBalancer
helm repo update
helm install traefik traefik/traefik --namespace traefik-system --set service.type=LoadBalancer

# Wait for Traefik to be ready
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=traefik -n traefik-system --timeout=300s

# Deploy APRS IngressRoute and Certificate (temporarily removed webhook)
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl delete validatingwebhookconfiguration cert-manager-webhook
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f aprs/aprs-ingressroute.yaml
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml
```

### Longhorn Storage Deployment

```bash
# Install Longhorn v1.7.1
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.1/deploy/longhorn.yaml

# Set privileged security policy for Longhorn namespace (required for hostPath volumes)
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl label namespace longhorn-system pod-security.kubernetes.io/enforce=privileged

# Restart DaemonSet to apply new security policy
PATH="/Users/graham/google-cloud-sdk/bin:$PATH" kubectl rollout restart ds/longhorn-manager -n longhorn-system

# Note: Longhorn may take several minutes to fully initialize and create storage class
```

### DNS Configuration Update

```bash
# Updated aprs.me DNS records to point to new cluster IP
# File: home/terraform/dns_aprs_me_cloudflare.tf
# Changed from 204.110.191.3 to 204.110.191.2

# Verify DNS changes (records already applied correctly)
tofu -chdir=/Users/graham/dev/infra/home/terraform plan -target=cloudflare_record.aprs_me_root -target=cloudflare_record.aprs_me_www
tofu -chdir=/Users/graham/dev/infra/home/terraform state show cloudflare_record.aprs_me_root

# Confirmed: Both aprs.me and www.aprs.me point to 204.110.191.2
```

### Node-RED Deployment (2025-10-19)

```bash
# Added 204.110.191.3 to node1 configuration
talosctl apply-config --talosconfig=./talosconfig -e 10.0.101.21 --nodes 10.0.101.21 --file controlplane.yaml --config-patch @node1.patch.yaml --config-patch @tailscale.patch.yaml

# Deployed nginx proxy for multiple services on 204.110.191.3:8080/8443
cd /Users/graham/dev/infra/home/cluster/nginx-proxy
./deploy.sh

# Deployed Node-RED to cluster
cd /Users/graham/dev/infra/home/cluster/nodered
./deploy.sh

# Updated DNS for nodered.w5isp.com to point to 204.110.191.3
tofu -chdir=/Users/graham/dev/infra/home/terraform apply -target=dnsimple_zone_record.w5isp_nodered -auto-approve
```

**Status**: ✅ Node-RED deployed and accessible
- Traefik handles routing and SSL termination via host-based routing
- APRS: https://aprs.me (204.110.191.2 via Traefik)
- Node-RED: https://nodered.w5isp.com (204.110.191.2 via Traefik)
- Both services use ClusterIP (internal only)
- Uses Longhorn for persistent storage

## Key Configuration Files

- `controlplane.yaml` - Talos control plane configuration
- `worker.yaml` - Talos worker node configuration  
- `node1.patch.yaml`, `node2.patch.yaml`, `node3.patch.yaml` - Node-specific patches
- `talosconfig` - Talos client configuration

## Notes

- Cluster uses Talos Linux as the operating system
- Applications deployed via kubectl with manifests in subdirectories
- Each application has its own deploy.sh script for easy deployment

## Reproduction Steps

1. Apply Talos configurations to nodes
2. Bootstrap cluster
3. Deploy applications as needed

_Detailed steps will be added as deployment progresses._