towerops/lib/towerops/monitoring
Graham McIntire ee8a3220c4
refactor: convert periodic workers to Oban Cron for better resilience
Replaced GenServer-based periodic workers with Oban Cron jobs to improve
pod rollover resilience and simplify architecture.

Worker Changes:
- NeighborCleanupWorker: GenServer → Oban Cron (hourly)
  - Cleans stale neighbors, ARP entries, and MAC addresses
  - Runs every hour via Oban.Plugins.Cron

- StaleAgentWorker: GenServer → Oban Cron (every minute)
  - Detects agents that haven't checked in for 10+ minutes
  - Refactored to reduce nesting (extracted helper functions)
  - Removed stateful tracking (now stateless, re-evaluates each run)

- AgentLatencyEvaluator: GenServer → Oban Cron (every 5 minutes)
  - Latency-based agent reassignment with 20% threshold
  - Removed trigger_evaluation/0 (no longer needed)

- JobHealthCheckWorker: NEW Oban Cron worker (every 10 minutes)
  - Safety net to recover missing monitor/poller jobs
  - Auto-creates jobs for devices with monitoring/SNMP enabled

Infrastructure Changes:
- Removed Monitoring.Supervisor (no longer needed)
- Updated application.ex to remove GenServer workers from supervision tree
- Added Oban.Plugins.Cron to dev.exs and runtime.exs
- All workers now run cluster-wide via PostgreSQL-backed coordination

Test Updates:
- Updated all worker tests to call perform(%Oban.Job{args: %{}})
- Removed GenServer lifecycle tests (start_link, send messages, etc.)
- Removed async sleep calls (no longer needed)

Benefits:
- Better pod rollover resilience (Cron jobs run cluster-wide)
- Simpler architecture (no GenServers for periodic tasks)
- Better observability (all jobs visible in Oban dashboard)
- Safety net for missing jobs (JobHealthCheckWorker)
- Stateless workers (easier to reason about and test)

Documentation:
- Updated CLAUDE.md with Background Job Architecture section
- Documented job types, queues, and resilience features

All tests passing (3,686 tests, 0 failures).
2026-01-24 16:56:28 -06:00
..
check.ex remote agent/cloud improvements 2026-01-24 09:16:41 -06:00
ping.ex fix: use system ping command instead of raw ICMP sockets 2026-01-21 11:15:59 -06:00
ping_behaviour.ex snmp bits 2026-01-03 14:41:28 -06:00