Replaced GenServer-based periodic workers with Oban Cron jobs to improve
pod rollover resilience and simplify architecture.
Worker Changes:
- NeighborCleanupWorker: GenServer → Oban Cron (hourly)
- Cleans stale neighbors, ARP entries, and MAC addresses
- Runs every hour via Oban.Plugins.Cron
- StaleAgentWorker: GenServer → Oban Cron (every minute)
- Detects agents that haven't checked in for 10+ minutes
- Refactored to reduce nesting (extracted helper functions)
- Removed stateful tracking (now stateless, re-evaluates each run)
- AgentLatencyEvaluator: GenServer → Oban Cron (every 5 minutes)
- Latency-based agent reassignment with 20% threshold
- Removed trigger_evaluation/0 (no longer needed)
- JobHealthCheckWorker: NEW Oban Cron worker (every 10 minutes)
- Safety net to recover missing monitor/poller jobs
- Auto-creates jobs for devices with monitoring/SNMP enabled
Infrastructure Changes:
- Removed Monitoring.Supervisor (no longer needed)
- Updated application.ex to remove GenServer workers from supervision tree
- Added Oban.Plugins.Cron to dev.exs and runtime.exs
- All workers now run cluster-wide via PostgreSQL-backed coordination
Test Updates:
- Updated all worker tests to call perform(%Oban.Job{args: %{}})
- Removed GenServer lifecycle tests (start_link, send messages, etc.)
- Removed async sleep calls (no longer needed)
Benefits:
- Better pod rollover resilience (Cron jobs run cluster-wide)
- Simpler architecture (no GenServers for periodic tasks)
- Better observability (all jobs visible in Oban dashboard)
- Safety net for missing jobs (JobHealthCheckWorker)
- Stateless workers (easier to reason about and test)
Documentation:
- Updated CLAUDE.md with Background Job Architecture section
- Documented job types, queues, and resilience features
All tests passing (3,686 tests, 0 failures).
Fixed two critical issues preventing agent communication:
1. Fix Mix.env() runtime error in production
- Replace Mix.env() with Application.get_env(:towerops, :env)
- Add :env config to test.exs
- Mix module not available in production releases
2. Add Protocol Buffers support to agent API endpoints
- GET /api/v1/agent/config now accepts application/x-protobuf
- POST /api/v1/agent/heartbeat now accepts application/x-protobuf
- Added conversion functions for config/equipment/sensors/interfaces
- Maintains JSON backward compatibility as fallback
All agent controller tests passing (14 tests, 0 failures)
Document complete setup process for deploying Towerops to a Talos
Kubernetes cluster, including:
- All required infrastructure prerequisites
- GitLab agent for CI/CD
- Newt for Pangolin forwarding
- Secrets management with 1Password
- FluxCD GitOps configuration
- Deployment process and troubleshooting
