Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
User deletion bug:
- Store client IP in socket assigns during mount
- Access stored IP during delete event instead of connect_info
- Fixes RuntimeError when attempting to delete users
Timezone support:
- Add timezone offset maps for common timezones (no external deps)
- Implement shift_timezone/2 helper using DateTime.add/3
- Add timezone-aware format_datetime/2, format_date/2, format_iso8601/2
- Keep backwards-compatible 1-arity versions defaulting to UTC
- Add comprehensive test coverage for timezone conversions
- All tests passing (23 tests, 0 failures)
Note: Timezone offsets are standard time only (no DST handling)
Common timezones supported: America/New_York, America/Los_Angeles,
Europe/London, Europe/Paris, Asia/Tokyo, Asia/Shanghai, Australia/Sydney
Prevents duplicate polling in multi-node production environment:
- Replace DynamicSupervisor with Horde.DynamicSupervisor (distributed)
- Replace Registry with Horde.Registry (cluster-aware)
- Each device monitor runs on exactly ONE node across the cluster
- Each SNMP poller runs on exactly ONE node across the cluster
- Automatic failover when nodes go down
- Automatic rebalancing when nodes join/leave
Architecture:
- Horde.Registry for process naming (distributed lookup)
- Horde.DynamicSupervisor for DeviceMonitor processes
- Horde.DynamicSupervisor for PollerWorker processes
- Task.Supervisor remains local per-node for parallel operations
- members: :auto enables automatic cluster discovery via libcluster
Production impact:
- 2 replicas = devices split ~50/50 across both pods
- No duplicate SNMP polls to same device
- If pod dies, surviving pod takes over all devices
- When new pod starts, devices rebalance automatically
Dependencies added:
- horde ~> 0.9.0 (distributed supervisor/registry)
- delta_crdt (Horde dependency for CRDT-based state)
- libring (Horde dependency for consistent hashing)
- Configure Phoenix.PubSub to use Redis adapter for pod-resilient messaging
- Add PubSub broadcasts for sensor readings updates in PollerWorker
- Add PubSub broadcasts for interface stats updates in PollerWorker
- Add PubSub broadcasts for neighbor discovery updates in PollerWorker
- Add PubSub broadcasts for monitoring check updates in MonitorWorker
- Broadcast interface and sensor events to device-specific topics
- Add event handlers in DeviceLive.Show for all update types
- Device show page now updates in real-time without polling
- Updates survive pod restarts and work across multiple pods
- Add exq ~> 0.23 dependency with redix and elixir_uuid
- Configure Exq in application supervisor with 4 queues (default, discovery, polling, maintenance)
- Set Exq as included_application to prevent auto-start
- Only start Exq in dev/prod environments, skip in test
- Use :env config to determine runtime environment
- Configure 10 concurrent workers with exq namespace
- Connect to Redis/Valkey sidecar via existing :redis config
- Add MIB files from LibreNMS in priv/mibs/ for reference
- Create MibParser module to validate OIDs against official MIB definitions
- Add MIB validation tests to ensure hardcoded OIDs match MIB specs
- Refactor SNMP tests to be generic/behavior-focused instead of vendor-specific
- Remove vendor-specific test files (cisco_test, net_snmp_test)
- All 104 tests passing with automated OID validation
Security improvements to prevent potential issues:
1. Port Validation (normalize_port):
- Validate port range is 1-65535
- Use Integer.parse instead of String.to_integer to prevent crashes
- Return default port 161 for any invalid input
2. SNMP Test Validation (validate_test_snmp_input):
- Validate IP address format before allowing SNMP tests
- Require non-empty community string
- Prevent network scanning with invalid/missing IPs
- Return clear error messages for validation failures
These changes ensure user input is properly validated and prevent:
- Integer overflow/underflow with ports
- Process crashes from invalid input
- Unauthorized network scanning via SNMP test feature
- SNMP requests with missing credentials
Implement comprehensive admin interface allowing designated superusers to view all users and organizations, impersonate users for debugging, and perform administrative operations. All superuser actions are tracked in audit logs for compliance.
Features:
- Superuser authentication with dedicated admin routes at /admin
- User impersonation with session state preservation
- Admin dashboard with system statistics
- User and organization management interfaces
- Comprehensive audit logging with IP tracking
- Visual impersonation banner with exit capability
- Security controls preventing self-impersonation and superuser-to-superuser impersonation
Database:
- Add is_superuser boolean field to users table
- Create audit_logs table for tracking sensitive operations
- Set graham@mcintire.me as initial superuser
