Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
Removing all Valkey (Redis) resources from Kubernetes due to instability
caused by Flannel CNI networking issues. Redis will now run on Proxmox
hosts for better stability and performance.
Changes:
- Delete Valkey StatefulSet (master + 2 replicas)
- Delete Valkey Sentinel StatefulSet (3 instances)
- Delete Valkey services (headless and sentinel)
- Delete Valkey ConfigMap
- Remove Valkey resources from kustomization.yaml
- Update deployment to use towerops-redis secret for connection
Next Steps:
- Set up Redis Sentinel on 3 Proxmox hosts/LXC containers
- Create towerops-redis secret with REDIS_HOST and REDIS_PORT
- Test failover and application connectivity
Benefits:
- Not affected by K8s networking issues (Flannel failures)
- More stable (no restarts from node issues)
- Better performance (no K8s overhead)
- Independent lifecycle from K8s cluster
Previously Valkey ran as a sidecar container in the main deployment,
causing it to restart every time the web app deployed. This resulted
in connection errors and cache loss during deployments.
Changes:
- Created valkey-statefulset.yaml with StatefulSet resource (1 replica)
- Created valkey-service.yaml with headless service
- Removed Valkey sidecar container from deployment.yaml
- Updated REDIS_HOST from 127.0.0.1 to valkey-0.valkey.towerops.svc.cluster.local
- Added new resources to kustomization.yaml
Benefits:
- Valkey persists across web app deployments
- No connection errors during rolling updates
- Cache data preserved
- Independent scaling and resource management
Prevent WebSocket disconnections during deployments by:
1. Rolling Update Strategy:
- maxSurge: 1 (allow 1 extra pod during rollout)
- maxUnavailable: 0 (keep all pods running)
- minReadySeconds: 10 (wait before continuing rollout)
2. Graceful Shutdown:
- terminationGracePeriodSeconds: 30 (allow Phoenix to drain connections)
3. PodDisruptionBudget:
- minAvailable: 1 (ensure at least 1 pod always available)
- Prevents all pods from being terminated simultaneously
4. Improved Health Checks:
- Faster readiness probe (5s initial, 3s period)
- More aggressive success/failure thresholds
- New pods marked ready faster
This ensures new pods are fully ready and serving traffic before old
pods are terminated, maintaining WebSocket connections and preventing
'something went wrong' errors during deployments.
Secrets are now managed directly in the cluster rather than generated
from .envrc files. This fixes FluxCD reconciliation errors since .envrc
is gitignored and cannot be used in GitOps workflows.
All secrets have been backed up to 1Password for disaster recovery.
