Replaced GenServer-based periodic workers with Oban Cron jobs to improve
pod rollover resilience and simplify architecture.
Worker Changes:
- NeighborCleanupWorker: GenServer → Oban Cron (hourly)
- Cleans stale neighbors, ARP entries, and MAC addresses
- Runs every hour via Oban.Plugins.Cron
- StaleAgentWorker: GenServer → Oban Cron (every minute)
- Detects agents that haven't checked in for 10+ minutes
- Refactored to reduce nesting (extracted helper functions)
- Removed stateful tracking (now stateless, re-evaluates each run)
- AgentLatencyEvaluator: GenServer → Oban Cron (every 5 minutes)
- Latency-based agent reassignment with 20% threshold
- Removed trigger_evaluation/0 (no longer needed)
- JobHealthCheckWorker: NEW Oban Cron worker (every 10 minutes)
- Safety net to recover missing monitor/poller jobs
- Auto-creates jobs for devices with monitoring/SNMP enabled
Infrastructure Changes:
- Removed Monitoring.Supervisor (no longer needed)
- Updated application.ex to remove GenServer workers from supervision tree
- Added Oban.Plugins.Cron to dev.exs and runtime.exs
- All workers now run cluster-wide via PostgreSQL-backed coordination
Test Updates:
- Updated all worker tests to call perform(%Oban.Job{args: %{}})
- Removed GenServer lifecycle tests (start_link, send messages, etc.)
- Removed async sleep calls (no longer needed)
Benefits:
- Better pod rollover resilience (Cron jobs run cluster-wide)
- Simpler architecture (no GenServers for periodic tasks)
- Better observability (all jobs visible in Oban dashboard)
- Safety net for missing jobs (JobHealthCheckWorker)
- Stateless workers (easier to reason about and test)
Documentation:
- Updated CLAUDE.md with Background Job Architecture section
- Documented job types, queues, and resilience features
All tests passing (3,686 tests, 0 failures).
Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
Redis authentication was failing with "NOAUTH Authentication required"
because the password wasn't being read from the environment variable.
Changes:
- Read REDIS_PASSWORD from environment in runtime.exs
- Add password to :redis config keyword list if present
- Phoenix.PubSub.Redis and Exq now read password from Application config
The application.ex and exq_supervisor.ex already had password support
from System.get_env("REDIS_PASSWORD"), but runtime.exs wasn't passing
the password through to the :redis config, so it was never available
to the application code.
This enables connection to the new Proxmox-hosted Valkey instance at
10.0.15.21 with authentication.
- Add exq ~> 0.23 dependency with redix and elixir_uuid
- Configure Exq in application supervisor with 4 queues (default, discovery, polling, maintenance)
- Set Exq as included_application to prevent auto-start
- Only start Exq in dev/prod environments, skip in test
- Use :env config to determine runtime environment
- Configure 10 concurrent workers with exq namespace
- Connect to Redis/Valkey sidecar via existing :redis config
- Add Valkey 8.0 alpine container as sidecar in k8s deployment
- Configure with appropriate security context (non-root, no privilege escalation)
- Add resource limits (256Mi memory, 200m CPU) and requests (128Mi, 50m)
- Add health probes using valkey-cli ping command
- Configure Redis connection via REDIS_HOST and REDIS_PORT environment variables
- Add Redis config to runtime.exs, dev.exs, and test.exs
- Valkey runs on localhost:6379 within the same pod as Phoenix app
Bandit 1.10.1 does not support read_timeout/write_timeout in http_1_options
or stream_idle_timeout in http_2_options. These invalid options caused the
application to crash on startup with 'Unsupported key(s) in http_1_options'.
Moved read_timeout to thousand_island_options where it belongs, and removed
the unsupported write_timeout and stream_idle_timeout options.
Also fixed UserSettingsHTMLTest to include the missing mobile_sessions assign
that is required by the edit.html.heex template.
