Graham McIntire
9ffa49a8f2
Prod paired the Oban Pro Smart engine + unique workers with core Oban.Plugins.Lifeline, whose rescue has no unique_violation handling. An orphaned executing CheckExecutorWorker job rescued back to available collides with its already-scheduled successor on oban_jobs_unique_index (23505), crashing the plugin every 60s and leaving ~150 orphans stuck. Switch prod to Oban.Pro.Plugins.DynamicLifeline, which repairs the conflict via Smart.clear_uniq_violation. Dev keeps core Lifeline (Basic engine). Add a shallow /health/live endpoint that does not touch db/redis. k8s liveness and startup probes now target it so a transient dependency outage can't kill or block boot of a healthy pod; readiness keeps the deep /health to gate the load balancer. (deployment.yaml probe/replica change pushed separately, after the image carrying /health/live is live.) Also drop unused POSTGRES_* keys from the secrets example. |
||
|---|---|---|
| .. | ||
| base-image | ||
| certificate.yaml | ||
| deployment.yaml | ||
| Dockerfile | ||
| Dockerfile.base | ||
| ingressroute.yaml | ||
| kustomization.yaml | ||
| namespace.yaml | ||
| poddisruptionbudget.yaml | ||
| README.md | ||
| secrets.example.yaml | ||
| service-headless.yaml | ||
| service.yaml | ||
| traefik-security-headers-middleware.yaml | ||
Kubernetes Deployment
Secrets Management
Secrets are managed directly in the cluster and must be created before deploying the application.
Required secrets in the towerops namespace:
gitlab-registry- Docker registry credentials for pulling imagestowerops-secrets- Application secrets (RELEASE_COOKIE, SECRET_KEY_BASE)towerops-db- Database connection credentialstowerops-aws- AWS credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION)
Optional secrets:
towerops-llm- DeepSeek API credentials for LLM-powered insight enrichment. Optional — when missing, insights still display without an AI summary.towerops-billing- Stripe credentials. Every key is optional in the deployment.
Local secrets workflow
k8s/secrets.yaml is gitignored. Use it to keep every Secret manifest
the deployment needs in one place for hand-application against the
cluster. Bootstrap from the unified template:
cp k8s/secrets.example.yaml k8s/secrets.yaml
# edit k8s/secrets.yaml — fill in DATABASE_URL, AWS keys, RELEASE_COOKIE,
# SECRET_KEY_BASE, CLOAK_KEY, and any optional ones (DEEPSEEK_API_KEY,
# Stripe keys, etc.)
kubectl apply -f k8s/secrets.yaml
kubectl rollout restart deployment/towerops -n towerops
k8s/secrets.yaml is one multi-document YAML covering every Secret the
deployment references (towerops-secrets, towerops-db, towerops-aws,
towerops-redis, towerops-billing, towerops-llm). It is excluded from git
via .gitignore so real values never accidentally land in source.
For local development, the project root .envrc is used by direnv.
Deployment Timestamp
The application footer displays the deployment timestamp to track when the current version was deployed. This is automatically set by GitLab CI during deployment:
# GitLab CI sets this during deploy
- kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
All pods in the deployment share the same timestamp (when the deployment was initiated), regardless of when individual pods were created. This is displayed in the footer as "Last deployed X ago · YYYY-MM-DD HH:MM:SS UTC".
For manual deployments without GitLab CI, set the timestamp:
kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
Deploying
Apply all resources using kustomize:
kubectl apply -k k8s/
Or individually:
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/secret.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/service-headless.yaml
kubectl apply -f k8s/certificate.yaml
kubectl apply -f k8s/ingressroute.yaml