From eb7dbb2d0e13401c0d4e78d5e51f43b60ab30249 Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Wed, 14 Jan 2026 13:14:10 -0600 Subject: [PATCH] Add comprehensive Kubernetes deployment documentation Document complete setup process for deploying Towerops to a Talos Kubernetes cluster, including: - All required infrastructure prerequisites - GitLab agent for CI/CD - Newt for Pangolin forwarding - Secrets management with 1Password - FluxCD GitOps configuration - Deployment process and troubleshooting --- CLAUDE.md | 215 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 215 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index de4daa0b..0f01eac3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -182,6 +182,221 @@ Key constraints from AGENTS.md (see that file for complete details): - be sure to always run mix format after you modify an elixir file - when you run mix format, don't specify a file, let it format everything +## Kubernetes Deployment + +### Prerequisites for Talos Kubernetes Cluster + +The application requires the following infrastructure components to be installed in the cluster before deployment: + +#### 1. Core Infrastructure + +**cert-manager** - Manages SSL/TLS certificates +- Required for automatic Let's Encrypt certificate provisioning +- Must have ClusterIssuers configured: `letsencrypt-prod` and `letsencrypt-staging` + +**Traefik** - Ingress controller +- Handles HTTP/HTTPS traffic routing +- Must be configured with `web` (HTTP) and `websecure` (HTTPS) entrypoints +- The application uses IngressRoute CRDs (not standard Ingress resources) + +**MetalLB** - Load balancer for bare metal clusters +- Provides LoadBalancer service type support in on-premises environments +- Required for Traefik to get an external IP + +**FluxCD** - GitOps continuous delivery +- Automatically syncs Kubernetes resources from Git repository +- Required components: source-controller, kustomize-controller, helm-controller, notification-controller +- Must have GitRepository resource pointing to this repository +- Must have Kustomization resource for the `k8s/` directory + +#### 2. Storage + +**NFS Provisioner** - Dynamic persistent volume provisioning +- Provides StorageClass for persistent volume claims +- Required for stateful workloads (if needed in the future) + +#### 3. CI/CD Integration + +**GitLab Agent** - Connects cluster to GitLab for CI/CD +- Enables GitLab CI/CD pipelines to deploy to the cluster +- Required for automated deployments from GitLab CI +- Agent configuration managed in `.gitlab/agents/` directory +- Agent must be registered in GitLab project settings + +#### 4. Network Services + +**Newt** - Pangolin tunnel forwarding +- Enables external access to internal services +- Required for Pangolin protocol forwarding +- Configuration specific to network topology + +**CoreDNS** - Cluster DNS +- Standard Kubernetes DNS service (included in Talos by default) + +**Tailscale Operator** - VPN mesh networking (optional) +- Provides secure access to cluster services +- Useful for development and administrative access + +### Secrets Management + +All secrets are stored in 1Password and must be created manually in the cluster before deployment. + +#### Required Secrets in `towerops` Namespace + +1. **`gitlab-registry`** - Docker registry credentials + - Type: `kubernetes.io/dockerconfigjson` + - Used to pull images from GitLab Container Registry + - Retrieve from 1Password or GitLab project settings + +2. **`towerops-secrets`** - Application secrets + - Type: `Opaque` + - Fields: + - `RELEASE_COOKIE` - Erlang distributed cookie for node clustering + - `SECRET_KEY_BASE` - Phoenix secret key base for session encryption + - Stored in 1Password: "Towerops K8s Secrets" + +3. **`towerops-db`** - Database connection + - Type: `Opaque` + - Fields: + - `POSTGRES_HOST` - Database hostname + - `POSTGRES_PORT` - Database port + - `POSTGRES_DB` - Database name + - `POSTGRES_USER` - Database username + - `POSTGRES_PASSWORD` - Database password + - `DATABASE_URL` - Full connection URL + - Stored in 1Password: "Towerops K8s Database" + +4. **`towerops-aws`** - AWS credentials + - Type: `Opaque` + - Fields: + - `AWS_ACCESS_KEY_ID` - AWS access key + - `AWS_SECRET_ACCESS_KEY` - AWS secret key + - `AWS_REGION` - AWS region (us-east-1) + - Stored in 1Password: "Towerops K8s AWS" + +#### Creating Secrets from 1Password + +```bash +# Set 1Password account +export OP_ACCOUNT=YOOATCZZSVGH7AD6VABUVPORLI + +# Create towerops-secrets +kubectl create secret generic towerops-secrets -n towerops \ + --from-literal=RELEASE_COOKIE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=RELEASE_COOKIE)" \ + --from-literal=SECRET_KEY_BASE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=SECRET_KEY_BASE)" + +# Create towerops-db +kubectl create secret generic towerops-db -n towerops \ + --from-literal=POSTGRES_HOST="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=server)" \ + --from-literal=POSTGRES_PORT="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=port)" \ + --from-literal=POSTGRES_DB="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=database)" \ + --from-literal=POSTGRES_USER="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=username)" \ + --from-literal=POSTGRES_PASSWORD="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=password)" \ + --from-literal=DATABASE_URL="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=DATABASE_URL)" + +# Create towerops-aws +kubectl create secret generic towerops-aws -n towerops \ + --from-literal=AWS_ACCESS_KEY_ID="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_ACCESS_KEY_ID)" \ + --from-literal=AWS_SECRET_ACCESS_KEY="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_SECRET_ACCESS_KEY)" \ + --from-literal=AWS_REGION="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_REGION)" +``` + +### Deployment Process + +#### FluxCD Automatic Deployment + +The application is automatically deployed via FluxCD GitOps: + +1. FluxCD monitors the Git repository for changes +2. When changes are detected, it applies the manifests in `k8s/` directory +3. Kustomize builds the manifests and applies them to the cluster + +**GitRepository Configuration:** +```yaml +apiVersion: source.toolkit.fluxcd.io/v1 +kind: GitRepository +metadata: + name: towerops + namespace: flux-system +spec: + interval: 1m0s + url: ssh://git@gitlab.com/graham/towerops.git + ref: + branch: main + secretRef: + name: towerops-git +``` + +**Kustomization Configuration:** +```yaml +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: towerops-app + namespace: flux-system +spec: + interval: 1m0s + path: ./k8s + prune: true + sourceRef: + kind: GitRepository + name: towerops + targetNamespace: towerops +``` + +#### Manual Deployment + +If FluxCD is not available or for testing: + +```bash +# Apply all resources using kustomize +kubectl apply -k k8s/ + +# Or apply individually +kubectl apply -f k8s/namespace.yaml +kubectl apply -f k8s/deployment.yaml +kubectl apply -f k8s/service.yaml +kubectl apply -f k8s/service-headless.yaml +kubectl apply -f k8s/certificate.yaml +kubectl apply -f k8s/ingressroute.yaml +``` + +### Cluster Setup Checklist + +When recreating a Talos cluster from scratch: + +- [ ] Install Talos on nodes and bootstrap cluster +- [ ] Install MetalLB and configure IP address pool +- [ ] Install Traefik ingress controller +- [ ] Install cert-manager and configure ClusterIssuers +- [ ] Install FluxCD and configure GitRepository source +- [ ] Install GitLab Agent and connect to GitLab project +- [ ] Install Newt for Pangolin forwarding +- [ ] Install NFS provisioner (if using persistent storage) +- [ ] Create `towerops` namespace +- [ ] Create all required secrets from 1Password +- [ ] Create FluxCD Kustomization resource for `k8s/` directory +- [ ] Verify deployment with `kubectl get pods -n towerops` +- [ ] Check certificate provisioning with `kubectl get certificate -n towerops` +- [ ] Test ingress with `curl https://towerops.net/health` + +### Troubleshooting + +**FluxCD reconciliation errors:** +- Check kustomization status: `kubectl get kustomization -n flux-system` +- View detailed status: `kubectl describe kustomization towerops-app -n flux-system` +- Force reconciliation: `kubectl annotate kustomization towerops-app -n flux-system reconcile.fluxcd.io/requestedAt="$(date +%Y-%m-%dT%H:%M:%S%z)" --overwrite` + +**Certificate issues:** +- Check certificate status: `kubectl describe certificate towerops-net-cert -n towerops` +- View cert-manager logs: `kubectl logs -n cert-manager deployment/cert-manager` +- Verify ClusterIssuer: `kubectl get clusterissuer letsencrypt-prod -o yaml` + +**Pod not starting:** +- Check pod status: `kubectl describe pod -n towerops -l app=towerops` +- View logs: `kubectl logs -n towerops deployment/towerops` +- Verify secrets exist: `kubectl get secrets -n towerops` + ## Testing Patterns ### SNMP Mocking with Mox