Changed discover_all/1 to enqueue individual Oban jobs instead of using
Task.async_stream for better error visibility and resilience.
Benefits:
- Errors are visible in Oban dashboard with full stack traces
- Automatic retries on failure via Oban
- Jobs persist across pod restarts
- No more silent failures from Task crashes
- Better concurrency control via Oban queue configuration
Changes:
- discover_all/1 now enqueues DiscoveryWorker jobs and returns immediately
- Return type changed from %{success, failed, errors} to %{enqueued, failed, errors}
- Updated tests to match new async behavior
- Added DiscoveryWorker alias to Discovery module
All tests passing (3,625 tests, 0 failures).
The health check endpoint was calling Towerops.RedisHealthCheck.check_health/2
but the module didn't exist, causing startup probes to fail with 503 errors.
Added RedisHealthCheck module with:
- check_health/2: Single PING command with timeout for health checks
- wait_for_redis/2: Exponential backoff retry logic for startup
This fixes the failing health checks that were blocking pod rollouts.
All tests passing.
Replaced GenServer-based periodic workers with Oban Cron jobs to improve
pod rollover resilience and simplify architecture.
Worker Changes:
- NeighborCleanupWorker: GenServer → Oban Cron (hourly)
- Cleans stale neighbors, ARP entries, and MAC addresses
- Runs every hour via Oban.Plugins.Cron
- StaleAgentWorker: GenServer → Oban Cron (every minute)
- Detects agents that haven't checked in for 10+ minutes
- Refactored to reduce nesting (extracted helper functions)
- Removed stateful tracking (now stateless, re-evaluates each run)
- AgentLatencyEvaluator: GenServer → Oban Cron (every 5 minutes)
- Latency-based agent reassignment with 20% threshold
- Removed trigger_evaluation/0 (no longer needed)
- JobHealthCheckWorker: NEW Oban Cron worker (every 10 minutes)
- Safety net to recover missing monitor/poller jobs
- Auto-creates jobs for devices with monitoring/SNMP enabled
Infrastructure Changes:
- Removed Monitoring.Supervisor (no longer needed)
- Updated application.ex to remove GenServer workers from supervision tree
- Added Oban.Plugins.Cron to dev.exs and runtime.exs
- All workers now run cluster-wide via PostgreSQL-backed coordination
Test Updates:
- Updated all worker tests to call perform(%Oban.Job{args: %{}})
- Removed GenServer lifecycle tests (start_link, send messages, etc.)
- Removed async sleep calls (no longer needed)
Benefits:
- Better pod rollover resilience (Cron jobs run cluster-wide)
- Simpler architecture (no GenServers for periodic tasks)
- Better observability (all jobs visible in Oban dashboard)
- Safety net for missing jobs (JobHealthCheckWorker)
- Stateless workers (easier to reason about and test)
Documentation:
- Updated CLAUDE.md with Background Job Architecture section
- Documented job types, queues, and resilience features
All tests passing (3,686 tests, 0 failures).
Changes:
- Add Oban configuration to config/dev.exs (was only in runtime.exs)
- Replace Exq telemetry metrics with Oban equivalents
- Update publish_exq_stats to publish_oban_stats using Oban.Job queries
- Track queue sizes, executing jobs, and available jobs
This fixes the startup error where Oban config was not available in dev.
Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
Fixes two production issues:
1. NOAUTH Redis authentication error:
- application.ex: Read password from application config
- exq_supervisor.ex: Read password from application config
- Both were bypassing runtime.exs config by reading env directly
2. EtcdCoordinator FunctionClauseError:
- connect_etcd/0 now returns connection name (:towerops_etcd)
- eetcd functions expect the atom name, not the PID
- Fixes crash when trying to acquire locks
The app was crashing when the etcd watcher couldn't be created, even
though 2/3 etcd nodes were healthy and the connection succeeded.
Changes:
- Made watcher creation optional (nil if it fails)
- App now starts successfully even if watcher is unavailable
- Falls back to periodic rebalancing (60s intervals) instead of instant failover
- Changed error level from error to warning when watcher creation fails
- Added helpful context to warning message
This allows the app to run with degraded performance instead of crashing
completely when there are etcd cluster issues.
🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The pods were crashing with a MatchError when the etcd watcher failed
to initialize. The code was pattern matching on {:ok, watcher} but
eetcd was returning {:error, :eetcd_conn_unavailable}.
Error:
** (MatchError) no match of right hand side value:
{:error, :eetcd_conn_unavailable}
This was causing CrashLoopBackOff because the application couldn't
start the Monitoring.Supervisor due to EtcdCoordinator initialization
failure.
Fix:
- Add case statement to handle both {:ok, watcher} and {:error, reason}
- Log the specific error when watcher creation fails
- Return proper error tuple to supervisor for clean shutdown
The :eetcd Erlang library expects charlists, not binary strings.
This was causing a FunctionClauseError in :lists.reverse/1 because
it was receiving a binary string instead of a charlist.
Error:
(FunctionClauseError) no function clause matching in :lists.reverse/1
(stdlib 7.2) lists.erl:291: :lists.reverse("etcd-0.etcd...")
Solution:
Use ~c sigil to convert endpoint strings to charlists for the
Erlang library to consume correctly.
Add REDIS_PASSWORD environment variable support to both Phoenix.PubSub.Redis
and Exq background job processor. Password is optional - if not set, connects
without authentication.
Changes:
- application.ex: Add password to PubSub Redis config if REDIS_PASSWORD is set
- exq_supervisor.ex: Add password to Exq Redis config if REDIS_PASSWORD is set
Required for connecting to Proxmox-hosted Redis with authentication.
Add 8 new vendor modules for wireless sensor discovery:
- PMP: Cambium PMP 450/430/100 and PTP series
- Mimosa: B5/B5c PTP and A5/A5c PTMP radios
- Siklu: EtherHaul mmWave radios
- Radwin: WinLink and JET series
- UniFi: Ubiquiti UniFi Access Points
- LigoOS: LigoWave and Deliberant devices
- cnPilot: Cambium cnPilot E and R series
- SAF: SAF Tehnika Integra/Lumina/CFIP series
Each module implements the Vendor behaviour with:
- profile_names/0: OS profile identifiers
- detect_hardware/1: device model detection
- wireless_oid_defs/0: sensor OID definitions
- discover_wireless_sensors/1: sensor discovery
Includes comprehensive tests for all modules (104 tests).
Mirrors LibreNMS architecture where vendor-specific discovery logic
lives in OS PHP classes. Creates modular vendor system with:
- Vendor behaviour defining callbacks for profile_names, detect_hardware,
discover_wireless_sensors, wireless_oid_defs
- Epmp vendor module for Cambium ePMP devices
- Airos vendor module for Ubiquiti AirOS devices
- Routeros vendor module for MikroTik devices with table-based discovery
- Registry module for dispatching to appropriate vendor by profile name
Removes 250+ lines of inline vendor code from dynamic.ex.
- Change sensor_value to last_value to match Sensor schema
- Make wireless sensor_index unique using OID suffix
- Values now persist and can be graphed
