This commit implements the unified checks architecture that consolidates
SNMP monitoring with HTTP/TCP/DNS checks under a single "check" abstraction,
enabling consistent graphing, alerting, and management across all check types.
Database Changes:
- Add source_type and source_id to checks table for tracking auto-discovered
vs manually created checks
- Add value field to check_results for storing numeric sensor readings
- Maintain backward compatibility with existing check_results data
New SNMP Executors:
- SnmpSensorExecutor: Poll sensor OIDs and return formatted values with
status determination (OK/WARNING/CRITICAL based on limits)
- SnmpInterfaceExecutor: Poll interface stats (bandwidth, packets, errors)
- SnmpProcessorExecutor: Poll CPU/processor usage
- SnmpStorageExecutor: Poll disk/memory usage with percentage calculations
Check Execution Worker:
- CheckExecutorWorker: Unified Oban worker that dispatches to appropriate
executor based on check_type (snmp_sensor, snmp_interface, http, tcp, etc.)
- Self-schedules next execution with distributed polling offsets
- Records results in check_results TimescaleDB hypertable
- Updates check state (OK/WARNING/CRITICAL/UNKNOWN)
Discovery Integration:
- Auto-creates checks during SNMP discovery for sensors, interfaces,
processors, and storage
- Links checks to source entities via source_id for data lookup
- Enables/disables checks based on discovery results
UI Enhancements:
- Checks tab on device detail page with grouped display
- FormComponent for adding manual HTTP/TCP/DNS checks
- Empty state with "Run Discovery" prompt
- Check status badges and last checked times
Graphing:
- Update GraphLive to accept check_id parameter
- Query check_results table for time-series data
- Support all check types (SNMP, HTTP response times, etc.)
Testing:
- Comprehensive test suite for SnmpSensorExecutor (5 tests)
- Test suite for CheckExecutorWorker (7 tests)
- Test coverage for discovery check creation (6 tests)
- Remove deprecated monitoring_test.exs testing old API
Bug Fixes:
- Fix SNMP executors reading credentials from correct Device schema fields
(device.snmp_version instead of device.snmp_device.version)
- Update agent channel test to query MonitoringCheck table directly
Code Quality:
- Extract add_snmp_credentials helper to reduce cyclomatic complexity
- Use map-based lookups for sensor formatting and check type grouping
- Apply pattern matching in dispatcher to reduce complexity
- All credo checks passing with no issues
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add 'Checks' tab link to device navigation
- Display empty state when no checks configured
- Add tests for tab visibility and empty state
- Fix unused variable warnings in monitoring context
- Remove unused CheckResult alias and module attributes
- Add stub for legacy get_latency_data for backward compatibility
Tests: 2 new tests added, all passing
Following TDD: RED-GREEN cycle verified for each feature
Enhanced AgentChannel logging to track device IDs across job list updates.
Added tests verifying server correctly excludes deleted devices from job lists.
Root cause: Bug is in Go agent (appends jobs without clearing), not Phoenix server.
Build a rich network topology from SNMP polling data using evidence-based
confidence scoring. LLDP/CDP neighbors, MAC address tables, and ARP data
are combined to infer device links with weighted confidence merging.
- Add DeviceLink and DeviceLinkEvidence schemas for persistent topology
- Implement evidence collectors: LLDP (0.95), CDP (0.95), MAC (0.7), ARP (0.6)
- Add device role inference from sysObjectID/sysDescr patterns
- Hook topology inference into DevicePollerWorker pipeline
- Add stale link cleanup (24h mark stale, 72h delete) via NeighborCleanupWorker
- Update NetworkMapLive with "Added" vs "All Devices" tabs
- Add connected devices section to device detail page
- Add device role selector to device edit form
- Update Cytoscape.js with role-based node shapes/colors and confidence edges
The plug was attempting to register a before_send callback after
blocking a banned IP and sending a 403 response. This caused
Plug.Conn.AlreadySentError in production.
Fixed by checking conn.halted before attempting to register the
404 tracking callback. If the request was already blocked and sent,
we skip the callback registration.
Also updated the test to verify proper behavior without the
workaround for AlreadySentError.
Always broadcast from update_device_status/2 so the device index
LiveView refreshes even when status hasn't changed. Add a 15-second
tick timer to keep "Last Checked" timestamps fresh between polls.
Scope alert PubSub topics to organization for defense-in-depth.
- use 64-bit HC counters (ifHCInOctets/ifHCOutOctets) instead of 32-bit
counters that wrap every ~34s on gigabit links
- fix negative integer parsing in replay adapter so sensor scale values
like -3 produce correct divisors (1000) instead of falling through to 1
- replace walking entire Cisco enterprise tree with targeted subtrees to
avoid timeouts; add Juniper, HP/H3C, Fortinet, Cambium vendor OIDs
- add missing discovery OIDs: ENTITY-STATE-MIB, HOST-RESOURCES-MIB
tables, and UCD-SNMP-MIB Linux CPU stats fallback
newly discovered devices sat idle with no metric data until the
next scheduling cycle. now the channel sends a poll job (and ping
job if monitoring enabled) right after discovery succeeds, using
send(self(), ...) so discovery DB writes commit first.
also reduces k8s replicas from 3 to 2.
- increase timing threshold in session activity test (flaky at 50ms)
- show org name instead of agent ID in admin agents list
- fix agent live tests wiping auth session with init_test_session
- add resilient error handling in FourOhFourTracker for missing redis
Replace phx-click JS.navigate with real <.link navigate> elements
on table rows so users can Ctrl+Click to open in new tabs, use
right-click context menus, and navigate via keyboard. Adds row_link
attr to <.table> core component and applies it to all agent tables.
Converts device list custom table from phx-click to inner links.
Broadcast org-scoped PubSub events from Devices context mutations
(create, update, delete, status change). Convert DeviceLive.Index
from assigns to streams with flat row items. Subscribe on mount so
the device list updates automatically across browser tabs.
POST /api/v1/webhooks/agent-release triggers all connected agents
to self-update via their existing WebSocket channels. Authenticated
with a shared secret (AGENT_WEBHOOK_SECRET env var).
- Add ReleaseChecker.invalidate_cache/0 for fresh GitHub fetch
- Add Agents.list_updatable_agents/0 (enabled + seen < 10min)
- Add Agents.broadcast_mass_update/0 orchestration function
- Add WebhookAuth plug with timing-safe secret comparison
- Add AgentReleaseWebhookController with :webhook pipeline
- Configure AGENT_WEBHOOK_SECRET in dev/test/runtime configs
Add server-initiated restart and self-update capabilities for remote
agents. Restart broadcasts on the lifecycle PubSub topic causing the
agent to exit cleanly for Docker to restart. Self-update fetches the
latest release from GitHub, finds the matching binary for the agent's
architecture, and sends the download URL with SHA256 checksum.
- Add restart_agent/1 and update_agent/3 to Agents context
- Handle :restart_requested and :update_requested in AgentChannel
- Add superuser-only Restart and Update buttons on agent show page
- Add ReleaseChecker module for GitHub Releases API with 5-min cache
- Add arch field to AgentHeartbeat protobuf for architecture detection
- Store arch in agent metadata from heartbeat
The admin agents page links to /agents/:id but the show and edit
pages were scoped to the current user's organization, causing 404s
for agents belonging to other orgs.
Broadcast agent lifecycle events (created/deleted) on a separate
"admin:agents" topic so the admin page updates dynamically when
agents are added or removed from any organization.
Superuser-only page showing all agents across all organizations
and cloud pollers in a single table with live-ticking timestamps
and real-time status updates via PubSub.
Added integration tests for real-time updates and metrics display.
Also fixed dialyzer warnings in worker event broadcasting by removing
unreachable error handling branches.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add LiveView for real-time job monitoring dashboard:
- Subscribe to job:lifecycle PubSub topic for live updates
- Display health metrics (completed, failed, avg duration, active jobs)
- Show active operations with real-time updates
- Display problems section (stuck/failed job counts)
- Add router entry under /admin/monitoring
Tests verify PubSub subscription and basic rendering.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Remove unused Ecto.Query import from device_poller_worker_test
- Update route paths from /orgs/:slug to /dashboard after navigation refactor
- Fix HostParser tests to match actual hostname resolution behavior
- Update UserAuth redirect assertions to expect /dashboard instead of /devices
- Fix DashboardLiveTest missing organization context
- Fix OrgLive.NewTest to verify organization creation correctly
All 4850 tests now passing with zero failures and zero warnings.
Changes:
- Fix traffic graph to handle nil interface octets gracefully
- Add comprehensive tests for nil octets scenarios
- Fix agent polling query to support devices assigned directly to org (left join on sites)
- Fix test SNMP connection to inherit credentials from site/org
- Refactor agent_channel process_polling_result to reduce nesting (Credo)
- Add alias for JobCleanupTask in application.ex (Credo)
This fixes crashes when interface stats have nil octets and ensures
devices without a site assignment can be polled by agents.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Extract duplicate access control logic from multiple LiveView files into
a reusable AccessControl helper module.
**Changes:**
- NEW: lib/towerops_web/live/helpers/access_control.ex
- NEW: test/towerops_web/live/helpers/access_control_test.exs
- Refactor device_live/index.ex to use AccessControl.verify_site_access/2
and verify_device_access/2
- Refactor device_live/form.ex to use AccessControl.verify_device_access/2
- Refactor alert_live/index.ex to use AccessControl.verify_alert_access/2
- Refactor device_live/show.ex to use AccessControl.verify_device_access/2
- Remove unused Repo aliases from refactored files
**Benefits:**
- Reduces code duplication across 7+ locations
- Centralizes security-critical access checks
- Improves testability (helper module has >90% coverage)
- Consistent error handling across all LiveViews
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fixed production bug where visiting the edit page for a device with no
SNMP discovery data would crash with "expected boolean on left-side of
'and', got: nil" error.
The mikrotik_device?/1 function was using && operator which returns nil
when device.snmp_device is nil, instead of returning false. The template
conditional on line 455 requires a proper boolean value.
Changed from:
device.snmp_device && (...)
To:
not is_nil(device.snmp_device) and (...)
This ensures the function always returns a boolean (true/false) instead
of potentially returning nil.
Added regression test to verify the page renders correctly when a device
has no SNMP discovery data yet.
Fixes: https://app.honeybadger.io/projects/136860/faults/127120431
Critical fixes:
- Renamed controller actions to follow Phoenix conventions (verify→new, create→verify)
- Added sudo mode check in GET action using last_sudo_at timestamp
- Added TOTP enrollment check in GET action with redirect
- Changed default return path from /orgs to /users/settings
- Updated route paths from /users/sudo/verify to /users/sudo-verify (dash separator)
- Added success flash message "Identity verified."
Implementation details:
- new/2 (GET): Checks recently_verified_sudo?/1 helper that examines last_sudo_at
instead of authenticated_at to distinguish sudo verification from regular login
- verify/2 (POST): Adds info flash and defaults to /users/settings
- recently_verified_sudo?/1: Private helper checking last_sudo_at within 10 minutes
- Updated all routes, templates, and tests to use new paths and action names
Tests:
- Added tests for already-in-sudo-mode redirect behavior
- Added test for TOTP enrollment requirement
- Updated all test assertions for new route paths and success messages
- All 15 controller tests passing
- Full test suite passing (4076 tests, 0 failures)
- Add UserSudoController with GET and POST /users/sudo/verify routes
- Create verify.html.heex template for TOTP verification form
- Only accept TOTP codes (6 numeric digits), reject recovery codes
- Update grant_sudo_mode to set authenticated_at virtual field
- Exclude /users/sudo paths from return_to overwriting
- Add comprehensive controller tests (12 test cases)
- Verify redirect behavior, error handling, and session management
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
