Add 8 new vendor modules for wireless sensor discovery:
- PMP: Cambium PMP 450/430/100 and PTP series
- Mimosa: B5/B5c PTP and A5/A5c PTMP radios
- Siklu: EtherHaul mmWave radios
- Radwin: WinLink and JET series
- UniFi: Ubiquiti UniFi Access Points
- LigoOS: LigoWave and Deliberant devices
- cnPilot: Cambium cnPilot E and R series
- SAF: SAF Tehnika Integra/Lumina/CFIP series
Each module implements the Vendor behaviour with:
- profile_names/0: OS profile identifiers
- detect_hardware/1: device model detection
- wireless_oid_defs/0: sensor OID definitions
- discover_wireless_sensors/1: sensor discovery
Includes comprehensive tests for all modules (104 tests).
Mirrors LibreNMS architecture where vendor-specific discovery logic
lives in OS PHP classes. Creates modular vendor system with:
- Vendor behaviour defining callbacks for profile_names, detect_hardware,
discover_wireless_sensors, wireless_oid_defs
- Epmp vendor module for Cambium ePMP devices
- Airos vendor module for Ubiquiti AirOS devices
- Routeros vendor module for MikroTik devices with table-based discovery
- Registry module for dispatching to appropriate vendor by profile name
Removes 250+ lines of inline vendor code from dynamic.ex.
- Change sensor_value to last_value to match Sensor schema
- Make wireless sensor_index unique using OID suffix
- Values now persist and can be graphed
- Add discover_vlans/1 to Cisco profile for VTP MIB discovery
- VTP OIDs: vtpVlanName, vtpVlanState, vtpVlanType
- Falls back to Q-BRIDGE-MIB when VTP not available
- Maps VTP states (operational, suspended, mtuTooBig)
- Maps VTP types (ethernet, fddi, tokenRing, etc.)
- Validates VLAN ID range (1-4094)
- Added state_sensors association to Snmp.Device schema
- Updated get_device_with_associations to preload state_sensors
- Added update_state_sensor and list_state_sensors functions to Snmp context
- Added state sensor polling task in PollerWorker parallel polling
- Poll state sensors via ENTITY-STATE-MIB OIDs every polling interval
- Detect and broadcast state changes (ok/warning/critical/unknown)
- State changes emit device events for real-time UI updates
- Rewrote Ping module to use system ping binary instead of raw ICMP
sockets which required CAP_NET_RAW privileges
- Added iputils-ping package to Docker images for production
- Added proper IP address validation before executing ping command
- Updated tests to account for 1 second minimum timeout of system ping
- System ping binary is setuid root and works without elevated privileges
- Create snmp_physical_entities table with migration
- Add PhysicalEntity schema with entity_class validation
- Support parent/child hierarchy via parent_entity_id
- Implement discover_physical_entities/1 in Base profile
- Discover chassis, modules, PSUs, fans from ENTITY-MIB
- Map entity class integers to string names
- Add 10 schema tests and 4 discovery tests
- Fix flaky AlertNotifier test with unique names and mailbox clearing
- Create snmp_ip_addresses table with migration
- Add IpAddress schema with ip_type validation (ipv4/ipv6)
- Add prefix_length validation based on ip_type
- Implement discover_ip_addresses/1 in Base profile
- Extract IP addresses from IP-MIB ipAdEntTable
- Support multiple IPs per interface with subnet masks
- Add 12 schema tests and 4 discovery tests
Reduced nesting and extracted helper functions to address credo
warnings. The logic remains the same:
- Test environment: synchronous update
- Production: async Task with sandbox support
Fixes credo warnings:
- Reduced function body nesting (was 3, now 2)
- Extracted spawn_async_update and helper functions
The async Task.start pattern was causing DBConnection.Ownership errors
when tests finished before the spawned Task could complete its database
update. By making the update synchronous in test environment, we ensure
the database connection is properly managed within the test process.
In production, we still use the async Task.start to avoid blocking
requests when updating last_used_at timestamps.
Fixes all "owner #PID<...> exited" errors in test suite.
Changes:
- ApiTokens.update_last_used/1: Allow Task to access sandbox in tests
- DeviceMonitor.enqueue_check/1: Allow Task to access sandbox in tests
- PollerWorker.enqueue_poll/1: Allow Task to access sandbox in tests
Problem:
When tests spawn async Tasks that access the database, those Tasks
don't have ownership of the database connection in Sandbox mode,
causing DBConnection.ConnectionError warnings.
Solution:
Before accessing the database in a Task, check if we're in test mode
and call Ecto.Adapters.SQL.Sandbox.allow/3 to transfer connection
ownership from the parent process to the Task.
Pattern used:
```elixir
parent = self()
Task.start(fn ->
if Application.get_env(:towerops, :sql_sandbox) do
Ecto.Adapters.SQL.Sandbox.allow(Repo, parent, self())
end
# ... database operations ...
end)
```
All 1041 tests passing. Some harmless disconnection warnings remain
when Tasks complete after tests finish, but these don't affect results.
Changes:
- Updated MonitorWorker to use configured ping_module adapter instead of System.cmd
- Removed direct ping command execution and regex parsing
- Simplified ping_device/1 to use adapter.ping/2 (mockable in tests)
- Added PingMock expectations to all MonitorWorker tests
- Removed @tag :integration from unreachable host test (now mocked)
- All ping calls now use mocked responses in tests
Benefits:
- MonitorWorker tests now run in 0.2s instead of 5+ seconds
- No real ICMP network traffic during test runs
- Tests are deterministic and don't depend on network conditions
- Follows same adapter pattern as SNMP (testable, configurable)
All 1041 tests passing with no real network operations.
- Add 'error' to valid sensor reading status values
- SensorReading changeset was rejecting 'error' status
- PollWorker creates error readings when SNMP fails
- Fixed validation to include: ok, warning, critical, error
- Replace log capture tests with functional assertions
- ExUnit.CaptureLog not reliably capturing worker logs
- Changed to verify actual behavior instead of log messages
- Tests now check created records and state changes
- Add debug assertions to sensor error test
- Verify SNMP device exists and has sensors
- Better error messages for test failures
All 1045 tests now passing (0 failures, 2 skipped)
Configures Redix (Exq's Redis client) with better connection handling:
- sync_connect: true - fail fast if Redis unavailable at startup
- exit_on_disconnection: false - don't crash process on disconnect
- backoff_initial: 500ms, backoff_max: 30s - retry with exponential backoff
This helps Exq survive temporary Redis disconnections without crashing,
reducing the frequency of Protocol.UndefinedError crashes in Exq.Node.Server.
Fixes crashes in Exq.Node.Server when Redis connections fail by adding
health checks before startup and wrapping Exq in a custom supervisor
with better resilience.
Problem:
- Valkey pod restarting frequently (24 times in 28h) due to K8s Flannel CNI issues
- When Redis disconnects, Exq.Node.Server crashes trying to process_signals/2
- Gets nil from Redis instead of expected list, crashes on Enum.each/2
- Rapid crash/restart cycles reduce application stability
Changes:
1. RedisHealthCheck module (lib/towerops/redis_health_check.ex)
- Waits for Redis availability with exponential backoff before starting Exq
- Prevents Exq from starting when Redis is unavailable
- Handles connection errors gracefully without crashing caller processes
- Supports configurable retry attempts, timeouts, and backoff intervals
2. ExqSupervisor module (lib/towerops/exq_supervisor.ex)
- Custom supervisor that wraps Exq with health checks
- Uses one_for_one strategy with limited restarts (3 per 60s)
- Prevents rapid crash loops when Redis is unstable
- Increased Exq retry/backoff settings for better Redis recovery
- Allows application to continue functioning without background jobs if Redis unavailable
3. Updated Application.ex
- Replaced direct Exq start with ExqSupervisor
- Removed inline exq_config/0 function (moved to supervisor)
4. Tests
- Comprehensive tests for RedisHealthCheck (health checks, retries, timeouts)
- Tests for ExqSupervisor (configuration, restart strategy)
- All 1006 tests passing
Benefits:
- Prevents Exq crashes from propagating when Redis fails
- Application continues running even when Redis is temporarily unavailable
- Better logging of Redis connection issues
- More graceful degradation during infrastructure problems
Infrastructure Issue (to be addressed separately):
- K8s Flannel CNI plugin failing: /run/flannel/subnet.env not found
- Causing Valkey pod network issues and restarts
- Needs investigation of Flannel daemonset and node configuration
🤖 Generated with Claude Code
