Connection counts (IPv4/IPv6, DHCP leases) were showing as floats
(e.g. 1270.0) in executor output, change detection messages, and
chart tooltips/axes. Now displayed as integers everywhere.
This commit implements the unified checks architecture that consolidates
SNMP monitoring with HTTP/TCP/DNS checks under a single "check" abstraction,
enabling consistent graphing, alerting, and management across all check types.
Database Changes:
- Add source_type and source_id to checks table for tracking auto-discovered
vs manually created checks
- Add value field to check_results for storing numeric sensor readings
- Maintain backward compatibility with existing check_results data
New SNMP Executors:
- SnmpSensorExecutor: Poll sensor OIDs and return formatted values with
status determination (OK/WARNING/CRITICAL based on limits)
- SnmpInterfaceExecutor: Poll interface stats (bandwidth, packets, errors)
- SnmpProcessorExecutor: Poll CPU/processor usage
- SnmpStorageExecutor: Poll disk/memory usage with percentage calculations
Check Execution Worker:
- CheckExecutorWorker: Unified Oban worker that dispatches to appropriate
executor based on check_type (snmp_sensor, snmp_interface, http, tcp, etc.)
- Self-schedules next execution with distributed polling offsets
- Records results in check_results TimescaleDB hypertable
- Updates check state (OK/WARNING/CRITICAL/UNKNOWN)
Discovery Integration:
- Auto-creates checks during SNMP discovery for sensors, interfaces,
processors, and storage
- Links checks to source entities via source_id for data lookup
- Enables/disables checks based on discovery results
UI Enhancements:
- Checks tab on device detail page with grouped display
- FormComponent for adding manual HTTP/TCP/DNS checks
- Empty state with "Run Discovery" prompt
- Check status badges and last checked times
Graphing:
- Update GraphLive to accept check_id parameter
- Query check_results table for time-series data
- Support all check types (SNMP, HTTP response times, etc.)
Testing:
- Comprehensive test suite for SnmpSensorExecutor (5 tests)
- Test suite for CheckExecutorWorker (7 tests)
- Test coverage for discovery check creation (6 tests)
- Remove deprecated monitoring_test.exs testing old API
Bug Fixes:
- Fix SNMP executors reading credentials from correct Device schema fields
(device.snmp_version instead of device.snmp_device.version)
- Update agent channel test to query MonitoringCheck table directly
Code Quality:
- Extract add_snmp_credentials helper to reduce cyclomatic complexity
- Use map-based lookups for sensor formatting and check type grouping
- Apply pattern matching in dispatcher to reduce complexity
- All credo checks passing with no issues
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- create MonitoringCheck schema for the monitoring_checks table, fixing
silent data loss where ping results were dropped by the wrong schema
- update agent channel and device monitor worker to use new schema
- fix Stats queries to use MonitoringCheck instead of Check schema
- add list_online_cloud_pollers and list_cloud_polled_devices queries
- add CloudLatencyProbeWorker to dispatch cross-agent latency probes
every 8 hours via PubSub to all online cloud pollers
- scope AgentLatencyEvaluator to cloud pollers only with cloud_only
filter, lower min_checks (5), wider time window (48h), and
device-level assignments to avoid changing site/org defaults
- update cron schedule: probes at 0/8/16h, evaluation at 0:30/8:30/16:30
- refactor monitoring.ex and http_executor.ex to fix credo complexity
- Add 'Checks' tab link to device navigation
- Display empty state when no checks configured
- Add tests for tab visibility and empty state
- Fix unused variable warnings in monitoring context
- Remove unused CheckResult alias and module attributes
- Add stub for legacy get_latency_data for backward compatibility
Tests: 2 new tests added, all passing
Following TDD: RED-GREEN cycle verified for each feature
Implements Icinga2-style generic check system for comprehensive monitoring.
Phoenix Backend:
- Generic checks table with JSONB config for all check types
- TimescaleDB check_results hypertable for time-series data
- Check executors (HTTP, TCP, DNS)
- CheckWorker with self-scheduling and state transitions
- Extended alerts system for check-based alerting
Check Types: HTTP/HTTPS, TCP, DNS
Architecture: Icinga2 Checkable pattern, soft/hard states, flapping detection
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replaced GenServer-based periodic workers with Oban Cron jobs to improve
pod rollover resilience and simplify architecture.
Worker Changes:
- NeighborCleanupWorker: GenServer → Oban Cron (hourly)
- Cleans stale neighbors, ARP entries, and MAC addresses
- Runs every hour via Oban.Plugins.Cron
- StaleAgentWorker: GenServer → Oban Cron (every minute)
- Detects agents that haven't checked in for 10+ minutes
- Refactored to reduce nesting (extracted helper functions)
- Removed stateful tracking (now stateless, re-evaluates each run)
- AgentLatencyEvaluator: GenServer → Oban Cron (every 5 minutes)
- Latency-based agent reassignment with 20% threshold
- Removed trigger_evaluation/0 (no longer needed)
- JobHealthCheckWorker: NEW Oban Cron worker (every 10 minutes)
- Safety net to recover missing monitor/poller jobs
- Auto-creates jobs for devices with monitoring/SNMP enabled
Infrastructure Changes:
- Removed Monitoring.Supervisor (no longer needed)
- Updated application.ex to remove GenServer workers from supervision tree
- Added Oban.Plugins.Cron to dev.exs and runtime.exs
- All workers now run cluster-wide via PostgreSQL-backed coordination
Test Updates:
- Updated all worker tests to call perform(%Oban.Job{args: %{}})
- Removed GenServer lifecycle tests (start_link, send messages, etc.)
- Removed async sleep calls (no longer needed)
Benefits:
- Better pod rollover resilience (Cron jobs run cluster-wide)
- Simpler architecture (no GenServers for periodic tasks)
- Better observability (all jobs visible in Oban dashboard)
- Safety net for missing jobs (JobHealthCheckWorker)
- Stateless workers (easier to reason about and test)
Documentation:
- Updated CLAUDE.md with Background Job Architecture section
- Documented job types, queues, and resilience features
All tests passing (3,686 tests, 0 failures).
Replaces etcd-based distributed locking with Oban's PostgreSQL-backed job queue.
This simplifies the architecture by eliminating the need for a separate etcd cluster
while providing better reliability and observability.
Changes:
- Add Oban dependency and migration (oban_jobs table)
- Create DevicePollerCoordinator and DeviceMonitorCoordinator Oban workers
- Remove EtcdCoordinator and EtcdLock modules
- Update application supervisor to start Oban
- Configure Oban with pollers (50 workers) and monitors (50 workers) queues
- Remove etcd StatefulSet from Kubernetes manifests
- Update monitoring supervisor documentation
Benefits:
- Simpler architecture (no etcd cluster to manage)
- PostgreSQL-based (uses existing database)
- Built-in uniqueness prevents duplicate jobs cluster-wide
- Better observability with Oban Web UI
- Automatic job rescue on node crashes
- Easier local development (no etcd required)
What was removed:
- etcd StatefulSet (3 pods)
- EtcdCoordinator module (320 lines)
- EtcdLock module (158 lines)
- eetcd dependency
All 3,686 tests passing.
Fixes two production issues:
1. NOAUTH Redis authentication error:
- application.ex: Read password from application config
- exq_supervisor.ex: Read password from application config
- Both were bypassing runtime.exs config by reading env directly
2. EtcdCoordinator FunctionClauseError:
- connect_etcd/0 now returns connection name (:towerops_etcd)
- eetcd functions expect the atom name, not the PID
- Fixes crash when trying to acquire locks
The app was crashing when the etcd watcher couldn't be created, even
though 2/3 etcd nodes were healthy and the connection succeeded.
Changes:
- Made watcher creation optional (nil if it fails)
- App now starts successfully even if watcher is unavailable
- Falls back to periodic rebalancing (60s intervals) instead of instant failover
- Changed error level from error to warning when watcher creation fails
- Added helpful context to warning message
This allows the app to run with degraded performance instead of crashing
completely when there are etcd cluster issues.
🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The pods were crashing with a MatchError when the etcd watcher failed
to initialize. The code was pattern matching on {:ok, watcher} but
eetcd was returning {:error, :eetcd_conn_unavailable}.
Error:
** (MatchError) no match of right hand side value:
{:error, :eetcd_conn_unavailable}
This was causing CrashLoopBackOff because the application couldn't
start the Monitoring.Supervisor due to EtcdCoordinator initialization
failure.
Fix:
- Add case statement to handle both {:ok, watcher} and {:error, reason}
- Log the specific error when watcher creation fails
- Return proper error tuple to supervisor for clean shutdown
The :eetcd Erlang library expects charlists, not binary strings.
This was causing a FunctionClauseError in :lists.reverse/1 because
it was receiving a binary string instead of a charlist.
Error:
(FunctionClauseError) no function clause matching in :lists.reverse/1
(stdlib 7.2) lists.erl:291: :lists.reverse("etcd-0.etcd...")
Solution:
Use ~c sigil to convert endpoint strings to charlists for the
Erlang library to consume correctly.
- Rewrote Ping module to use system ping binary instead of raw ICMP
sockets which required CAP_NET_RAW privileges
- Added iputils-ping package to Docker images for production
- Added proper IP address validation before executing ping command
- Updated tests to account for 1 second minimum timeout of system ping
- System ping binary is setuid root and works without elevated privileges
Changes:
- ApiTokens.update_last_used/1: Allow Task to access sandbox in tests
- DeviceMonitor.enqueue_check/1: Allow Task to access sandbox in tests
- PollerWorker.enqueue_poll/1: Allow Task to access sandbox in tests
Problem:
When tests spawn async Tasks that access the database, those Tasks
don't have ownership of the database connection in Sandbox mode,
causing DBConnection.ConnectionError warnings.
Solution:
Before accessing the database in a Task, check if we're in test mode
and call Ecto.Adapters.SQL.Sandbox.allow/3 to transfer connection
ownership from the parent process to the Task.
Pattern used:
```elixir
parent = self()
Task.start(fn ->
if Application.get_env(:towerops, :sql_sandbox) do
Ecto.Adapters.SQL.Sandbox.allow(Repo, parent, self())
end
# ... database operations ...
end)
```
All 1041 tests passing. Some harmless disconnection warnings remain
when Tasks complete after tests finish, but these don't affect results.
- Changed ping module to use microsecond timer resolution instead of millisecond
- Prevents 0ms readings for very fast pings (localhost, local network)
- Updated response_time_ms to float type to store decimal precision
- Migration converts existing integer values to float
API token creation fix:
- Handle both nested (%{"token" => %{...}}) and flat params
- Support URL-encoded form data from LiveView
- Add validation for required name and organization_id fields
Rust agent raw ICMP ping implementation:
- New ping module with async ICMP echo request/reply
- Build ICMP packets manually with proper checksum calculation
- Use socket2 for raw socket creation (requires CAP_NET_RAW)
- Parse both raw ICMP and IP-wrapped responses
- Include unit tests for checksum and packet building
- Dependencies: socket2 v0.5, rand v0.8
- Compiles successfully with release profile optimizations
AutoDismissFlash hook fix:
- Move hook definitions before LiveSocket initialization
- Ensures hooks are defined when referenced in LiveSocket constructor
- Remove duplicate Hooks object
Raw ICMP ping implementation:
- Replace system ping command with pure Elixir implementation
- Use raw ICMP sockets for echo request/reply
- Build ICMP packets manually with proper checksum calculation
- Parse IP addresses using :inet.parse_address/1
- Handle both raw ICMP and IP-wrapped responses
- Requires CAP_NET_RAW capability in production containers
- Suitable for containerized environments without ping command
- Replace TODO comment with "Future enhancement" note
- Alias nested DeviceProfiles.Importer module
- Replace expensive length/1 check with empty list comparison
- Extract nested logic into helper functions to reduce complexity:
- import_sensor_with_states for sensor import
- walk_sensor_oid and walk_processor_oid for SNMP walks
- apply_processor_precision and determine_processor_status
- check_device_health for device monitoring
- Add auto-dismiss functionality to flash messages (30 second timeout)
- Implement AutoDismissFlash LiveView hook in TypeScript
Prevents duplicate polling in multi-node production environment:
- Replace DynamicSupervisor with Horde.DynamicSupervisor (distributed)
- Replace Registry with Horde.Registry (cluster-aware)
- Each device monitor runs on exactly ONE node across the cluster
- Each SNMP poller runs on exactly ONE node across the cluster
- Automatic failover when nodes go down
- Automatic rebalancing when nodes join/leave
Architecture:
- Horde.Registry for process naming (distributed lookup)
- Horde.DynamicSupervisor for DeviceMonitor processes
- Horde.DynamicSupervisor for PollerWorker processes
- Task.Supervisor remains local per-node for parallel operations
- members: :auto enables automatic cluster discovery via libcluster
Production impact:
- 2 replicas = devices split ~50/50 across both pods
- No duplicate SNMP polls to same device
- If pod dies, surviving pod takes over all devices
- When new pod starts, devices rebalance automatically
Dependencies added:
- horde ~> 0.9.0 (distributed supervisor/registry)
- delta_crdt (Horde dependency for CRDT-based state)
- libring (Horde dependency for consistent hashing)
- Use SNMP poll success as health indicator for SNMP-enabled devices
- Fall back to ICMP ping only for non-SNMP devices or stale SNMP data
- Add connection retry logic to Exq with backoff
- Add better error logging for ping permission issues
- Resolves issue where devices showed as down despite successful SNMP polling
Root cause: Container runs as non-root (UID 65534) without ICMP permissions,
so ping fails even though SNMP (UDP) works. Now using SNMP poll timestamps
to determine device status when SNMP is enabled.
Changed device_monitor.ex to always use ICMP ping for monitoring checks,
regardless of whether SNMP is enabled for the device.
Rationale:
- ICMP ping gives actual network latency (round-trip time)
- SNMP checks are for data collection (interfaces, sensors, etc.)
- Both should run independently for SNMP-enabled devices:
* SNMP polling: Collects device metrics (handled by poller_worker)
* ICMP monitoring: Measures latency and availability (handled by device_monitor)
This ensures latency graphs always show actual network response time,
not SNMP query response time.
The monitoring system was creating checks but hardcoding response_time_ms
to nil instead of using the actual ping latency. This caused the latency
graphs to have no data to display.
Changes:
- device_monitor.ex: Capture response time from ping/SNMP check result
- monitor_worker.ex: Use correct field name (response_time_ms) and status enum values (:success/:failure)
After this fix, new monitoring checks will include latency data and the
latency graphs will display properly on both device and site detail pages.
Created three new Exq workers to handle background jobs:
- PollWorker: SNMP polling operations (queue: polling)
- MonitorWorker: Device monitoring checks (queue: monitoring)
- DiscoveryWorker: SNMP discovery (already existed)
Updated files to use Exq.enqueue instead of Task.start:
- lib/towerops/snmp/poller_worker.ex
- lib/towerops/monitoring/device_monitor.ex
- lib/towerops_web/channels/agent_channel.ex
Each module includes enqueue_* helper functions that:
- Use Task.start in test environment for synchronous execution
- Use Exq.enqueue in dev/prod for proper background job processing
Added "monitoring" queue to Exq configuration in application.ex
All tests passing (824 tests, 0 failures)
- Prefix unused alert variables with underscore
- Remove unused test_env?/0 function
All warnings resolved, compilation clean with --warnings-as-errors
- Add equipment_events table and Event schema for tracking device changes
- Implement automatic change detection for interface attributes during SNMP polling
- Detects operational status changes (up/down)
- Detects admin status changes
- Detects speed changes with warning severity for speed drops
- Detects MAC address changes
- Add Logs tab to equipment detail page with event display
- Fix temperature sensor display bug (was showing 4.3C instead of 43C due to double division)
- Remove response time tracking from monitoring checks and UI
- Move Rediscover Device button to equipment edit page
This fixes critical issues where SNMP sensor readings, interface stats, and
monitoring checks were not being saved to the database due to schema mismatches.
Database schema fixes:
- Recreate snmp_sensor_readings table with binary_id primary key
- Recreate snmp_interface_stats table with binary_id primary key
- Recreate monitoring_checks table with binary_id primary key
The original migrations used default integer primary keys, but the schemas
expected binary_id (UUID). This caused Ecto to generate UUID strings that
Postgrex tried to encode as binaries, resulting in silent insert failures.
SNMP interface stats fixes:
- Fix field name mismatch in get_interface_stats (if_in_octets vs in_octets)
- Remove unnecessary tuple handling code (Client already unwraps SNMPKit tuples)
- Clean up decode_snmp_value function and improve error messages
Monitoring check fixes:
- Fix response_time type from float to integer in Poller.check_device
- Add error logging to catch and report check creation failures
These changes enable:
- SNMP sensor readings to be collected and stored (disk usage, memory, CPU, etc.)
- Interface statistics to be collected (Counter64 octets, errors, discards)
- Equipment availability metrics to update on the dashboard
Improved startup reliability and debugging for SNMP pollers:
- Added try/rescue blocks around monitor and poller startup
- Added logging for successful startup and errors
- Added error logging to PollerWorker's perform_poll
- Added debug logging for successful sensor/interface polls
This will help identify issues when pollers fail to start or
encounter errors during polling, which was causing silent failures
where sensors weren't being collected.
Add background workers to regularly poll SNMP sensors and interfaces:
- Create PollerWorker GenServer to poll SNMP devices
- Poll all sensors for temperature, voltage, CPU, memory, etc.
- Poll all interfaces for bandwidth, errors, and discards
- Save time-series data to snmp_sensor_readings and snmp_interface_stats
- Integrate with monitoring supervisor to auto-start pollers
- Use same interval as connectivity checks (minimum 30 seconds)
- Add list_snmp_enabled_equipment function to Equipment context
Pollers start automatically on app boot for all SNMP-enabled equipment
and run independently of connectivity monitoring.
- Replace catch-all pattern with explicit :unknown status handling
- Makes status transitions more explicit and type-safe
- Resolves pattern_match_cov warning at line 174
