Graham McIntire
b5e3c1f977
Three changes targeting the root cause of production up/down alerting: 1. Skip BruteForceProtection for /health and /health/live paths (removes unnecessary Repo.get_by(IpBlock) DB query from every k8s probe, matching the existing /socket/agent exemption). 2. Relax k8s readiness probe: periodSeconds 5→10, timeoutSeconds 2→5, failureThreshold 2→3, successThreshold 3→2 so transient DB/Redis blips don't cascade into full service unavailability. 3. Use the persistent Towerops.Redix connection for health checks instead of opening a brand-new TCP connection (handshake→AUTH→ PING→close) on every probe. Towerops.Redix.Fake gains a set_ping_response/2 toggle for testing failure paths. |
||
|---|---|---|
| .. | ||
| base-image | ||
| certificate.yaml | ||
| deployment.yaml | ||
| Dockerfile | ||
| Dockerfile.base | ||
| ingressroute.yaml | ||
| kustomization.yaml | ||
| namespace.yaml | ||
| poddisruptionbudget.yaml | ||
| README.md | ||
| secrets.example.yaml | ||
| service-headless.yaml | ||
| service.yaml | ||
| traefik-security-headers-middleware.yaml | ||
Kubernetes Deployment
Secrets Management
Secrets are managed directly in the cluster and must be created before deploying the application.
Required secrets in the towerops namespace:
gitlab-registry- Docker registry credentials for pulling imagestowerops-secrets- Application secrets (RELEASE_COOKIE, SECRET_KEY_BASE)towerops-db- Database connection credentialstowerops-aws- AWS credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION)
Optional secrets:
towerops-llm- DeepSeek API credentials for LLM-powered insight enrichment. Optional — when missing, insights still display without an AI summary.towerops-billing- Stripe credentials. Every key is optional in the deployment.
Local secrets workflow
k8s/secrets.yaml is gitignored. Use it to keep every Secret manifest
the deployment needs in one place for hand-application against the
cluster. Bootstrap from the unified template:
cp k8s/secrets.example.yaml k8s/secrets.yaml
# edit k8s/secrets.yaml — fill in DATABASE_URL, AWS keys, RELEASE_COOKIE,
# SECRET_KEY_BASE, CLOAK_KEY, and any optional ones (DEEPSEEK_API_KEY,
# Stripe keys, etc.)
kubectl apply -f k8s/secrets.yaml
kubectl rollout restart deployment/towerops -n towerops
k8s/secrets.yaml is one multi-document YAML covering every Secret the
deployment references (towerops-secrets, towerops-db, towerops-aws,
towerops-redis, towerops-billing, towerops-llm). It is excluded from git
via .gitignore so real values never accidentally land in source.
For local development, the project root .envrc is used by direnv.
Deployment Timestamp
The application footer displays the deployment timestamp to track when the current version was deployed. This is automatically set by GitLab CI during deployment:
# GitLab CI sets this during deploy
- kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
All pods in the deployment share the same timestamp (when the deployment was initiated), regardless of when individual pods were created. This is displayed in the footer as "Last deployed X ago · YYYY-MM-DD HH:MM:SS UTC".
For manual deployments without GitLab CI, set the timestamp:
kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
Deploying
Apply all resources using kustomize:
kubectl apply -k k8s/
Or individually:
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/secret.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/service-headless.yaml
kubectl apply -f k8s/certificate.yaml
kubectl apply -f k8s/ingressroute.yaml