Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
Fixes two production issues:
1. NOAUTH Redis authentication error:
- application.ex: Read password from application config
- exq_supervisor.ex: Read password from application config
- Both were bypassing runtime.exs config by reading env directly
2. EtcdCoordinator FunctionClauseError:
- connect_etcd/0 now returns connection name (:towerops_etcd)
- eetcd functions expect the atom name, not the PID
- Fixes crash when trying to acquire locks
The app was crashing when the etcd watcher couldn't be created, even
though 2/3 etcd nodes were healthy and the connection succeeded.
Changes:
- Made watcher creation optional (nil if it fails)
- App now starts successfully even if watcher is unavailable
- Falls back to periodic rebalancing (60s intervals) instead of instant failover
- Changed error level from error to warning when watcher creation fails
- Added helpful context to warning message
This allows the app to run with degraded performance instead of crashing
completely when there are etcd cluster issues.
🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The pods were crashing with a MatchError when the etcd watcher failed
to initialize. The code was pattern matching on {:ok, watcher} but
eetcd was returning {:error, :eetcd_conn_unavailable}.
Error:
** (MatchError) no match of right hand side value:
{:error, :eetcd_conn_unavailable}
This was causing CrashLoopBackOff because the application couldn't
start the Monitoring.Supervisor due to EtcdCoordinator initialization
failure.
Fix:
- Add case statement to handle both {:ok, watcher} and {:error, reason}
- Log the specific error when watcher creation fails
- Return proper error tuple to supervisor for clean shutdown
The :eetcd Erlang library expects charlists, not binary strings.
This was causing a FunctionClauseError in :lists.reverse/1 because
it was receiving a binary string instead of a charlist.
Error:
(FunctionClauseError) no function clause matching in :lists.reverse/1
(stdlib 7.2) lists.erl:291: :lists.reverse("etcd-0.etcd...")
Solution:
Use ~c sigil to convert endpoint strings to charlists for the
Erlang library to consume correctly.
Add REDIS_PASSWORD environment variable support to both Phoenix.PubSub.Redis
and Exq background job processor. Password is optional - if not set, connects
without authentication.
Changes:
- application.ex: Add password to PubSub Redis config if REDIS_PASSWORD is set
- exq_supervisor.ex: Add password to Exq Redis config if REDIS_PASSWORD is set
Required for connecting to Proxmox-hosted Redis with authentication.
Add 8 new vendor modules for wireless sensor discovery:
- PMP: Cambium PMP 450/430/100 and PTP series
- Mimosa: B5/B5c PTP and A5/A5c PTMP radios
- Siklu: EtherHaul mmWave radios
- Radwin: WinLink and JET series
- UniFi: Ubiquiti UniFi Access Points
- LigoOS: LigoWave and Deliberant devices
- cnPilot: Cambium cnPilot E and R series
- SAF: SAF Tehnika Integra/Lumina/CFIP series
Each module implements the Vendor behaviour with:
- profile_names/0: OS profile identifiers
- detect_hardware/1: device model detection
- wireless_oid_defs/0: sensor OID definitions
- discover_wireless_sensors/1: sensor discovery
Includes comprehensive tests for all modules (104 tests).
Mirrors LibreNMS architecture where vendor-specific discovery logic
lives in OS PHP classes. Creates modular vendor system with:
- Vendor behaviour defining callbacks for profile_names, detect_hardware,
discover_wireless_sensors, wireless_oid_defs
- Epmp vendor module for Cambium ePMP devices
- Airos vendor module for Ubiquiti AirOS devices
- Routeros vendor module for MikroTik devices with table-based discovery
- Registry module for dispatching to appropriate vendor by profile name
Removes 250+ lines of inline vendor code from dynamic.ex.
- Change sensor_value to last_value to match Sensor schema
- Make wireless sensor_index unique using OID suffix
- Values now persist and can be graphed
