Add comprehensive Kubernetes deployment documentation
Document complete setup process for deploying Towerops to a Talos Kubernetes cluster, including: - All required infrastructure prerequisites - GitLab agent for CI/CD - Newt for Pangolin forwarding - Secrets management with 1Password - FluxCD GitOps configuration - Deployment process and troubleshooting
This commit is contained in:
parent
ae949b4a0b
commit
eb7dbb2d0e
1 changed files with 215 additions and 0 deletions
215
CLAUDE.md
215
CLAUDE.md
|
|
@ -182,6 +182,221 @@ Key constraints from AGENTS.md (see that file for complete details):
|
||||||
- be sure to always run mix format after you modify an elixir file
|
- be sure to always run mix format after you modify an elixir file
|
||||||
- when you run mix format, don't specify a file, let it format everything
|
- when you run mix format, don't specify a file, let it format everything
|
||||||
|
|
||||||
|
## Kubernetes Deployment
|
||||||
|
|
||||||
|
### Prerequisites for Talos Kubernetes Cluster
|
||||||
|
|
||||||
|
The application requires the following infrastructure components to be installed in the cluster before deployment:
|
||||||
|
|
||||||
|
#### 1. Core Infrastructure
|
||||||
|
|
||||||
|
**cert-manager** - Manages SSL/TLS certificates
|
||||||
|
- Required for automatic Let's Encrypt certificate provisioning
|
||||||
|
- Must have ClusterIssuers configured: `letsencrypt-prod` and `letsencrypt-staging`
|
||||||
|
|
||||||
|
**Traefik** - Ingress controller
|
||||||
|
- Handles HTTP/HTTPS traffic routing
|
||||||
|
- Must be configured with `web` (HTTP) and `websecure` (HTTPS) entrypoints
|
||||||
|
- The application uses IngressRoute CRDs (not standard Ingress resources)
|
||||||
|
|
||||||
|
**MetalLB** - Load balancer for bare metal clusters
|
||||||
|
- Provides LoadBalancer service type support in on-premises environments
|
||||||
|
- Required for Traefik to get an external IP
|
||||||
|
|
||||||
|
**FluxCD** - GitOps continuous delivery
|
||||||
|
- Automatically syncs Kubernetes resources from Git repository
|
||||||
|
- Required components: source-controller, kustomize-controller, helm-controller, notification-controller
|
||||||
|
- Must have GitRepository resource pointing to this repository
|
||||||
|
- Must have Kustomization resource for the `k8s/` directory
|
||||||
|
|
||||||
|
#### 2. Storage
|
||||||
|
|
||||||
|
**NFS Provisioner** - Dynamic persistent volume provisioning
|
||||||
|
- Provides StorageClass for persistent volume claims
|
||||||
|
- Required for stateful workloads (if needed in the future)
|
||||||
|
|
||||||
|
#### 3. CI/CD Integration
|
||||||
|
|
||||||
|
**GitLab Agent** - Connects cluster to GitLab for CI/CD
|
||||||
|
- Enables GitLab CI/CD pipelines to deploy to the cluster
|
||||||
|
- Required for automated deployments from GitLab CI
|
||||||
|
- Agent configuration managed in `.gitlab/agents/` directory
|
||||||
|
- Agent must be registered in GitLab project settings
|
||||||
|
|
||||||
|
#### 4. Network Services
|
||||||
|
|
||||||
|
**Newt** - Pangolin tunnel forwarding
|
||||||
|
- Enables external access to internal services
|
||||||
|
- Required for Pangolin protocol forwarding
|
||||||
|
- Configuration specific to network topology
|
||||||
|
|
||||||
|
**CoreDNS** - Cluster DNS
|
||||||
|
- Standard Kubernetes DNS service (included in Talos by default)
|
||||||
|
|
||||||
|
**Tailscale Operator** - VPN mesh networking (optional)
|
||||||
|
- Provides secure access to cluster services
|
||||||
|
- Useful for development and administrative access
|
||||||
|
|
||||||
|
### Secrets Management
|
||||||
|
|
||||||
|
All secrets are stored in 1Password and must be created manually in the cluster before deployment.
|
||||||
|
|
||||||
|
#### Required Secrets in `towerops` Namespace
|
||||||
|
|
||||||
|
1. **`gitlab-registry`** - Docker registry credentials
|
||||||
|
- Type: `kubernetes.io/dockerconfigjson`
|
||||||
|
- Used to pull images from GitLab Container Registry
|
||||||
|
- Retrieve from 1Password or GitLab project settings
|
||||||
|
|
||||||
|
2. **`towerops-secrets`** - Application secrets
|
||||||
|
- Type: `Opaque`
|
||||||
|
- Fields:
|
||||||
|
- `RELEASE_COOKIE` - Erlang distributed cookie for node clustering
|
||||||
|
- `SECRET_KEY_BASE` - Phoenix secret key base for session encryption
|
||||||
|
- Stored in 1Password: "Towerops K8s Secrets"
|
||||||
|
|
||||||
|
3. **`towerops-db`** - Database connection
|
||||||
|
- Type: `Opaque`
|
||||||
|
- Fields:
|
||||||
|
- `POSTGRES_HOST` - Database hostname
|
||||||
|
- `POSTGRES_PORT` - Database port
|
||||||
|
- `POSTGRES_DB` - Database name
|
||||||
|
- `POSTGRES_USER` - Database username
|
||||||
|
- `POSTGRES_PASSWORD` - Database password
|
||||||
|
- `DATABASE_URL` - Full connection URL
|
||||||
|
- Stored in 1Password: "Towerops K8s Database"
|
||||||
|
|
||||||
|
4. **`towerops-aws`** - AWS credentials
|
||||||
|
- Type: `Opaque`
|
||||||
|
- Fields:
|
||||||
|
- `AWS_ACCESS_KEY_ID` - AWS access key
|
||||||
|
- `AWS_SECRET_ACCESS_KEY` - AWS secret key
|
||||||
|
- `AWS_REGION` - AWS region (us-east-1)
|
||||||
|
- Stored in 1Password: "Towerops K8s AWS"
|
||||||
|
|
||||||
|
#### Creating Secrets from 1Password
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Set 1Password account
|
||||||
|
export OP_ACCOUNT=YOOATCZZSVGH7AD6VABUVPORLI
|
||||||
|
|
||||||
|
# Create towerops-secrets
|
||||||
|
kubectl create secret generic towerops-secrets -n towerops \
|
||||||
|
--from-literal=RELEASE_COOKIE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=RELEASE_COOKIE)" \
|
||||||
|
--from-literal=SECRET_KEY_BASE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=SECRET_KEY_BASE)"
|
||||||
|
|
||||||
|
# Create towerops-db
|
||||||
|
kubectl create secret generic towerops-db -n towerops \
|
||||||
|
--from-literal=POSTGRES_HOST="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=server)" \
|
||||||
|
--from-literal=POSTGRES_PORT="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=port)" \
|
||||||
|
--from-literal=POSTGRES_DB="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=database)" \
|
||||||
|
--from-literal=POSTGRES_USER="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=username)" \
|
||||||
|
--from-literal=POSTGRES_PASSWORD="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=password)" \
|
||||||
|
--from-literal=DATABASE_URL="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=DATABASE_URL)"
|
||||||
|
|
||||||
|
# Create towerops-aws
|
||||||
|
kubectl create secret generic towerops-aws -n towerops \
|
||||||
|
--from-literal=AWS_ACCESS_KEY_ID="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_ACCESS_KEY_ID)" \
|
||||||
|
--from-literal=AWS_SECRET_ACCESS_KEY="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_SECRET_ACCESS_KEY)" \
|
||||||
|
--from-literal=AWS_REGION="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_REGION)"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deployment Process
|
||||||
|
|
||||||
|
#### FluxCD Automatic Deployment
|
||||||
|
|
||||||
|
The application is automatically deployed via FluxCD GitOps:
|
||||||
|
|
||||||
|
1. FluxCD monitors the Git repository for changes
|
||||||
|
2. When changes are detected, it applies the manifests in `k8s/` directory
|
||||||
|
3. Kustomize builds the manifests and applies them to the cluster
|
||||||
|
|
||||||
|
**GitRepository Configuration:**
|
||||||
|
```yaml
|
||||||
|
apiVersion: source.toolkit.fluxcd.io/v1
|
||||||
|
kind: GitRepository
|
||||||
|
metadata:
|
||||||
|
name: towerops
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
interval: 1m0s
|
||||||
|
url: ssh://git@gitlab.com/graham/towerops.git
|
||||||
|
ref:
|
||||||
|
branch: main
|
||||||
|
secretRef:
|
||||||
|
name: towerops-git
|
||||||
|
```
|
||||||
|
|
||||||
|
**Kustomization Configuration:**
|
||||||
|
```yaml
|
||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: towerops-app
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
interval: 1m0s
|
||||||
|
path: ./k8s
|
||||||
|
prune: true
|
||||||
|
sourceRef:
|
||||||
|
kind: GitRepository
|
||||||
|
name: towerops
|
||||||
|
targetNamespace: towerops
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Manual Deployment
|
||||||
|
|
||||||
|
If FluxCD is not available or for testing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Apply all resources using kustomize
|
||||||
|
kubectl apply -k k8s/
|
||||||
|
|
||||||
|
# Or apply individually
|
||||||
|
kubectl apply -f k8s/namespace.yaml
|
||||||
|
kubectl apply -f k8s/deployment.yaml
|
||||||
|
kubectl apply -f k8s/service.yaml
|
||||||
|
kubectl apply -f k8s/service-headless.yaml
|
||||||
|
kubectl apply -f k8s/certificate.yaml
|
||||||
|
kubectl apply -f k8s/ingressroute.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Cluster Setup Checklist
|
||||||
|
|
||||||
|
When recreating a Talos cluster from scratch:
|
||||||
|
|
||||||
|
- [ ] Install Talos on nodes and bootstrap cluster
|
||||||
|
- [ ] Install MetalLB and configure IP address pool
|
||||||
|
- [ ] Install Traefik ingress controller
|
||||||
|
- [ ] Install cert-manager and configure ClusterIssuers
|
||||||
|
- [ ] Install FluxCD and configure GitRepository source
|
||||||
|
- [ ] Install GitLab Agent and connect to GitLab project
|
||||||
|
- [ ] Install Newt for Pangolin forwarding
|
||||||
|
- [ ] Install NFS provisioner (if using persistent storage)
|
||||||
|
- [ ] Create `towerops` namespace
|
||||||
|
- [ ] Create all required secrets from 1Password
|
||||||
|
- [ ] Create FluxCD Kustomization resource for `k8s/` directory
|
||||||
|
- [ ] Verify deployment with `kubectl get pods -n towerops`
|
||||||
|
- [ ] Check certificate provisioning with `kubectl get certificate -n towerops`
|
||||||
|
- [ ] Test ingress with `curl https://towerops.net/health`
|
||||||
|
|
||||||
|
### Troubleshooting
|
||||||
|
|
||||||
|
**FluxCD reconciliation errors:**
|
||||||
|
- Check kustomization status: `kubectl get kustomization -n flux-system`
|
||||||
|
- View detailed status: `kubectl describe kustomization towerops-app -n flux-system`
|
||||||
|
- Force reconciliation: `kubectl annotate kustomization towerops-app -n flux-system reconcile.fluxcd.io/requestedAt="$(date +%Y-%m-%dT%H:%M:%S%z)" --overwrite`
|
||||||
|
|
||||||
|
**Certificate issues:**
|
||||||
|
- Check certificate status: `kubectl describe certificate towerops-net-cert -n towerops`
|
||||||
|
- View cert-manager logs: `kubectl logs -n cert-manager deployment/cert-manager`
|
||||||
|
- Verify ClusterIssuer: `kubectl get clusterissuer letsencrypt-prod -o yaml`
|
||||||
|
|
||||||
|
**Pod not starting:**
|
||||||
|
- Check pod status: `kubectl describe pod -n towerops -l app=towerops`
|
||||||
|
- View logs: `kubectl logs -n towerops deployment/towerops`
|
||||||
|
- Verify secrets exist: `kubectl get secrets -n towerops`
|
||||||
|
|
||||||
## Testing Patterns
|
## Testing Patterns
|
||||||
|
|
||||||
### SNMP Mocking with Mox
|
### SNMP Mocking with Mox
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue