Graham McIntire
8a54f9a866
Pods were running at 500m/500m (100% of limit) with 44-47% of scheduling periods being CPU-throttled. SSL decryption is CPU-intensive — when the BEAM gets throttled mid ssl_gen_statem:call/2, the SSL recv stalls until the cgroup allows CPU again, causing the 15s DBConnection timeout. This was the actual root cause of "client timed out because it queued and checked out the connection for longer than 15000ms" errors. Previous fixes (TCP keepalive, pool tuning, oban pruner) were treating symptoms. - CPU limit: 500m → 2000m (4 cores burst capacity) - CPU request: 100m → 500m (guarantee adequate baseline) Reviewed-on: graham/towerops-web#96 |
||
|---|---|---|
| .. | ||
| base-image | ||
| certificate.yaml | ||
| deployment.yaml | ||
| Dockerfile | ||
| ingressroute.yaml | ||
| kustomization.yaml | ||
| namespace.yaml | ||
| poddisruptionbudget.yaml | ||
| README.md | ||
| service-headless.yaml | ||
| service.yaml | ||
| traefik-security-headers-middleware.yaml | ||
Kubernetes Deployment
Secrets Management
Secrets are managed directly in the cluster and must be created before deploying the application.
Required secrets in the towerops namespace:
gitlab-registry- Docker registry credentials for pulling imagestowerops-secrets- Application secrets (RELEASE_COOKIE, SECRET_KEY_BASE)towerops-db- Database connection credentialstowerops-aws- AWS credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION)
For local development, the project root .envrc is used by direnv.
Deployment Timestamp
The application footer displays the deployment timestamp to track when the current version was deployed. This is automatically set by GitLab CI during deployment:
# GitLab CI sets this during deploy
- kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
All pods in the deployment share the same timestamp (when the deployment was initiated), regardless of when individual pods were created. This is displayed in the footer as "Last deployed X ago · YYYY-MM-DD HH:MM:SS UTC".
For manual deployments without GitLab CI, set the timestamp:
kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
Deploying
Apply all resources using kustomize:
kubectl apply -k k8s/
Or individually:
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/secret.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/service-headless.yaml
kubectl apply -f k8s/certificate.yaml
kubectl apply -f k8s/ingressroute.yaml