Add comprehensive Kubernetes deployment documentation
Document complete setup process for deploying Towerops to a Talos Kubernetes cluster, including: - All required infrastructure prerequisites - GitLab agent for CI/CD - Newt for Pangolin forwarding - Secrets management with 1Password - FluxCD GitOps configuration - Deployment process and troubleshooting
This commit is contained in:
parent
ae949b4a0b
commit
eb7dbb2d0e
1 changed files with 215 additions and 0 deletions
215
CLAUDE.md
215
CLAUDE.md
|
|
@ -182,6 +182,221 @@ Key constraints from AGENTS.md (see that file for complete details):
|
|||
- be sure to always run mix format after you modify an elixir file
|
||||
- when you run mix format, don't specify a file, let it format everything
|
||||
|
||||
## Kubernetes Deployment
|
||||
|
||||
### Prerequisites for Talos Kubernetes Cluster
|
||||
|
||||
The application requires the following infrastructure components to be installed in the cluster before deployment:
|
||||
|
||||
#### 1. Core Infrastructure
|
||||
|
||||
**cert-manager** - Manages SSL/TLS certificates
|
||||
- Required for automatic Let's Encrypt certificate provisioning
|
||||
- Must have ClusterIssuers configured: `letsencrypt-prod` and `letsencrypt-staging`
|
||||
|
||||
**Traefik** - Ingress controller
|
||||
- Handles HTTP/HTTPS traffic routing
|
||||
- Must be configured with `web` (HTTP) and `websecure` (HTTPS) entrypoints
|
||||
- The application uses IngressRoute CRDs (not standard Ingress resources)
|
||||
|
||||
**MetalLB** - Load balancer for bare metal clusters
|
||||
- Provides LoadBalancer service type support in on-premises environments
|
||||
- Required for Traefik to get an external IP
|
||||
|
||||
**FluxCD** - GitOps continuous delivery
|
||||
- Automatically syncs Kubernetes resources from Git repository
|
||||
- Required components: source-controller, kustomize-controller, helm-controller, notification-controller
|
||||
- Must have GitRepository resource pointing to this repository
|
||||
- Must have Kustomization resource for the `k8s/` directory
|
||||
|
||||
#### 2. Storage
|
||||
|
||||
**NFS Provisioner** - Dynamic persistent volume provisioning
|
||||
- Provides StorageClass for persistent volume claims
|
||||
- Required for stateful workloads (if needed in the future)
|
||||
|
||||
#### 3. CI/CD Integration
|
||||
|
||||
**GitLab Agent** - Connects cluster to GitLab for CI/CD
|
||||
- Enables GitLab CI/CD pipelines to deploy to the cluster
|
||||
- Required for automated deployments from GitLab CI
|
||||
- Agent configuration managed in `.gitlab/agents/` directory
|
||||
- Agent must be registered in GitLab project settings
|
||||
|
||||
#### 4. Network Services
|
||||
|
||||
**Newt** - Pangolin tunnel forwarding
|
||||
- Enables external access to internal services
|
||||
- Required for Pangolin protocol forwarding
|
||||
- Configuration specific to network topology
|
||||
|
||||
**CoreDNS** - Cluster DNS
|
||||
- Standard Kubernetes DNS service (included in Talos by default)
|
||||
|
||||
**Tailscale Operator** - VPN mesh networking (optional)
|
||||
- Provides secure access to cluster services
|
||||
- Useful for development and administrative access
|
||||
|
||||
### Secrets Management
|
||||
|
||||
All secrets are stored in 1Password and must be created manually in the cluster before deployment.
|
||||
|
||||
#### Required Secrets in `towerops` Namespace
|
||||
|
||||
1. **`gitlab-registry`** - Docker registry credentials
|
||||
- Type: `kubernetes.io/dockerconfigjson`
|
||||
- Used to pull images from GitLab Container Registry
|
||||
- Retrieve from 1Password or GitLab project settings
|
||||
|
||||
2. **`towerops-secrets`** - Application secrets
|
||||
- Type: `Opaque`
|
||||
- Fields:
|
||||
- `RELEASE_COOKIE` - Erlang distributed cookie for node clustering
|
||||
- `SECRET_KEY_BASE` - Phoenix secret key base for session encryption
|
||||
- Stored in 1Password: "Towerops K8s Secrets"
|
||||
|
||||
3. **`towerops-db`** - Database connection
|
||||
- Type: `Opaque`
|
||||
- Fields:
|
||||
- `POSTGRES_HOST` - Database hostname
|
||||
- `POSTGRES_PORT` - Database port
|
||||
- `POSTGRES_DB` - Database name
|
||||
- `POSTGRES_USER` - Database username
|
||||
- `POSTGRES_PASSWORD` - Database password
|
||||
- `DATABASE_URL` - Full connection URL
|
||||
- Stored in 1Password: "Towerops K8s Database"
|
||||
|
||||
4. **`towerops-aws`** - AWS credentials
|
||||
- Type: `Opaque`
|
||||
- Fields:
|
||||
- `AWS_ACCESS_KEY_ID` - AWS access key
|
||||
- `AWS_SECRET_ACCESS_KEY` - AWS secret key
|
||||
- `AWS_REGION` - AWS region (us-east-1)
|
||||
- Stored in 1Password: "Towerops K8s AWS"
|
||||
|
||||
#### Creating Secrets from 1Password
|
||||
|
||||
```bash
|
||||
# Set 1Password account
|
||||
export OP_ACCOUNT=YOOATCZZSVGH7AD6VABUVPORLI
|
||||
|
||||
# Create towerops-secrets
|
||||
kubectl create secret generic towerops-secrets -n towerops \
|
||||
--from-literal=RELEASE_COOKIE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=RELEASE_COOKIE)" \
|
||||
--from-literal=SECRET_KEY_BASE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=SECRET_KEY_BASE)"
|
||||
|
||||
# Create towerops-db
|
||||
kubectl create secret generic towerops-db -n towerops \
|
||||
--from-literal=POSTGRES_HOST="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=server)" \
|
||||
--from-literal=POSTGRES_PORT="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=port)" \
|
||||
--from-literal=POSTGRES_DB="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=database)" \
|
||||
--from-literal=POSTGRES_USER="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=username)" \
|
||||
--from-literal=POSTGRES_PASSWORD="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=password)" \
|
||||
--from-literal=DATABASE_URL="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=DATABASE_URL)"
|
||||
|
||||
# Create towerops-aws
|
||||
kubectl create secret generic towerops-aws -n towerops \
|
||||
--from-literal=AWS_ACCESS_KEY_ID="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_ACCESS_KEY_ID)" \
|
||||
--from-literal=AWS_SECRET_ACCESS_KEY="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_SECRET_ACCESS_KEY)" \
|
||||
--from-literal=AWS_REGION="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_REGION)"
|
||||
```
|
||||
|
||||
### Deployment Process
|
||||
|
||||
#### FluxCD Automatic Deployment
|
||||
|
||||
The application is automatically deployed via FluxCD GitOps:
|
||||
|
||||
1. FluxCD monitors the Git repository for changes
|
||||
2. When changes are detected, it applies the manifests in `k8s/` directory
|
||||
3. Kustomize builds the manifests and applies them to the cluster
|
||||
|
||||
**GitRepository Configuration:**
|
||||
```yaml
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: GitRepository
|
||||
metadata:
|
||||
name: towerops
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 1m0s
|
||||
url: ssh://git@gitlab.com/graham/towerops.git
|
||||
ref:
|
||||
branch: main
|
||||
secretRef:
|
||||
name: towerops-git
|
||||
```
|
||||
|
||||
**Kustomization Configuration:**
|
||||
```yaml
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: towerops-app
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 1m0s
|
||||
path: ./k8s
|
||||
prune: true
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: towerops
|
||||
targetNamespace: towerops
|
||||
```
|
||||
|
||||
#### Manual Deployment
|
||||
|
||||
If FluxCD is not available or for testing:
|
||||
|
||||
```bash
|
||||
# Apply all resources using kustomize
|
||||
kubectl apply -k k8s/
|
||||
|
||||
# Or apply individually
|
||||
kubectl apply -f k8s/namespace.yaml
|
||||
kubectl apply -f k8s/deployment.yaml
|
||||
kubectl apply -f k8s/service.yaml
|
||||
kubectl apply -f k8s/service-headless.yaml
|
||||
kubectl apply -f k8s/certificate.yaml
|
||||
kubectl apply -f k8s/ingressroute.yaml
|
||||
```
|
||||
|
||||
### Cluster Setup Checklist
|
||||
|
||||
When recreating a Talos cluster from scratch:
|
||||
|
||||
- [ ] Install Talos on nodes and bootstrap cluster
|
||||
- [ ] Install MetalLB and configure IP address pool
|
||||
- [ ] Install Traefik ingress controller
|
||||
- [ ] Install cert-manager and configure ClusterIssuers
|
||||
- [ ] Install FluxCD and configure GitRepository source
|
||||
- [ ] Install GitLab Agent and connect to GitLab project
|
||||
- [ ] Install Newt for Pangolin forwarding
|
||||
- [ ] Install NFS provisioner (if using persistent storage)
|
||||
- [ ] Create `towerops` namespace
|
||||
- [ ] Create all required secrets from 1Password
|
||||
- [ ] Create FluxCD Kustomization resource for `k8s/` directory
|
||||
- [ ] Verify deployment with `kubectl get pods -n towerops`
|
||||
- [ ] Check certificate provisioning with `kubectl get certificate -n towerops`
|
||||
- [ ] Test ingress with `curl https://towerops.net/health`
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
**FluxCD reconciliation errors:**
|
||||
- Check kustomization status: `kubectl get kustomization -n flux-system`
|
||||
- View detailed status: `kubectl describe kustomization towerops-app -n flux-system`
|
||||
- Force reconciliation: `kubectl annotate kustomization towerops-app -n flux-system reconcile.fluxcd.io/requestedAt="$(date +%Y-%m-%dT%H:%M:%S%z)" --overwrite`
|
||||
|
||||
**Certificate issues:**
|
||||
- Check certificate status: `kubectl describe certificate towerops-net-cert -n towerops`
|
||||
- View cert-manager logs: `kubectl logs -n cert-manager deployment/cert-manager`
|
||||
- Verify ClusterIssuer: `kubectl get clusterissuer letsencrypt-prod -o yaml`
|
||||
|
||||
**Pod not starting:**
|
||||
- Check pod status: `kubectl describe pod -n towerops -l app=towerops`
|
||||
- View logs: `kubectl logs -n towerops deployment/towerops`
|
||||
- Verify secrets exist: `kubectl get secrets -n towerops`
|
||||
|
||||
## Testing Patterns
|
||||
|
||||
### SNMP Mocking with Mox
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue