Graham McIntire
29593ac734
Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue. This simplifies the architecture by eliminating the need for a separate etcd cluster while providing better reliability and observability. Changes: - Add Oban dependency and migration (oban_jobs table) - Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers - Remove EtcdCoordinator and EtcdLock modules - Update application supervisor to start Oban - Configure Oban with pollers (50 workers) and monitors (50 workers) queues - Remove etcd StatefulSet from Kubernetes manifests - Update monitoring supervisor documentation Benefits: - Simpler architecture (no etcd cluster to manage) - PostgreSQL-based (uses existing database) - Built-in uniqueness prevents duplicate jobs cluster-wide - Better observability with Oban Web UI - Automatic job rescue on node crashes - Easier local development (no etcd required) What was removed: - etcd StatefulSet (3 pods) - EtcdCoordinator module (320 lines) - EtcdLock module (158 lines) - eetcd dependency All 3,686 tests passing. |
||
|---|---|---|
| .. | ||
| certificate.yaml | ||
| deployment.yaml | ||
| Dockerfile | ||
| ingressroute.yaml | ||
| kustomization.yaml | ||
| namespace.yaml | ||
| poddisruptionbudget.yaml | ||
| README.md | ||
| service-headless.yaml | ||
| service.yaml | ||
Kubernetes Deployment
Secrets Management
Secrets are managed directly in the cluster and must be created before deploying the application.
Required secrets in the towerops namespace:
gitlab-registry- Docker registry credentials for pulling imagestowerops-secrets- Application secrets (RELEASE_COOKIE, SECRET_KEY_BASE)towerops-db- Database connection credentialstowerops-aws- AWS credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION)
For local development, the project root .envrc is used by direnv.
Deployment Timestamp
The application footer displays the deployment timestamp to track when the current version was deployed. This is automatically set by GitLab CI during deployment:
# GitLab CI sets this during deploy
- kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
All pods in the deployment share the same timestamp (when the deployment was initiated), regardless of when individual pods were created. This is displayed in the footer as "Last deployed X ago · YYYY-MM-DD HH:MM:SS UTC".
For manual deployments without GitLab CI, set the timestamp:
kubectl set env deployment/towerops DEPLOY_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -n towerops
Deploying
Apply all resources using kustomize:
kubectl apply -k k8s/
Or individually:
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/secret.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/service-headless.yaml
kubectl apply -f k8s/certificate.yaml
kubectl apply -f k8s/ingressroute.yaml