Added telemetry metrics for monitoring:
Exq Metrics:
- Queue sizes for all queues (default, discovery, polling, monitoring, maintenance)
- Number of busy worker processes
- Total number of worker processes
Redis/Valkey Metrics:
- Connected clients
- Memory usage
- Total commands processed
Metrics are collected every 10 seconds via telemetry_poller and displayed
in LiveDashboard at /dashboard
Updated all references to use the more technically accurate term 'ICMP Latency'
instead of 'Ping Latency' in:
- Graph titles and labels
- HTML comments
- Test assertions
- Added comprehensive tests for GraphLive.Show to ensure:
* Page renders with correct assigns
* Uses @current_organization (not @organization)
* Latency, processor, memory, and traffic graphs work
* Time range selection works correctly
* All required assigns are present
- Reordered device show page so Overall Traffic appears above Ping Latency
Changed device_monitor.ex to always use ICMP ping for monitoring checks,
regardless of whether SNMP is enabled for the device.
Rationale:
- ICMP ping gives actual network latency (round-trip time)
- SNMP checks are for data collection (interfaces, sensors, etc.)
- Both should run independently for SNMP-enabled devices:
* SNMP polling: Collects device metrics (handled by poller_worker)
* ICMP monitoring: Measures latency and availability (handled by device_monitor)
This ensures latency graphs always show actual network response time,
not SNMP query response time.
The monitoring system was creating checks but hardcoding response_time_ms
to nil instead of using the actual ping latency. This caused the latency
graphs to have no data to display.
Changes:
- device_monitor.ex: Capture response time from ping/SNMP check result
- monitor_worker.ex: Use correct field name (response_time_ms) and status enum values (:success/:failure)
After this fix, new monitoring checks will include latency data and the
latency graphs will display properly on both device and site detail pages.
Shows ping latency for all devices in the site over the last 24 hours.
Each device is displayed as a separate line on the graph with its name
as the label.
Graph only appears if there is latency data available for at least
one device in the site.
Created API token system for programmatic access:
- API tokens table with organization scoping
- ApiTokens context for token management
- ApiAuth plug for Bearer token authentication
- Tokens shown once on creation, then hashed for security
Implemented RESTful API v1 endpoints:
- SitesController: CRUD operations for sites
- DevicesController: CRUD operations for devices
- All operations scoped to authenticated organization
- Proper authorization checks via site ownership
Technical details:
- Tokens prefixed with "towerops_" for easy identification
- SHA-256 hashing for token storage
- Last used timestamp tracking (async update)
- Optional token expiration support
- Standard JSON error responses (40x status codes)
Routes:
- /api/v1/sites (GET, POST, PATCH, DELETE)
- /api/v1/devices (GET, POST, PATCH, DELETE)
Authentication:
- Authorization: Bearer towerops_xxxxx header required
- Returns 401 for invalid/expired tokens
- Returns 403 for unauthorized resource access
Created three new Exq workers to handle background jobs:
- PollWorker: SNMP polling operations (queue: polling)
- MonitorWorker: Device monitoring checks (queue: monitoring)
- DiscoveryWorker: SNMP discovery (already existed)
Updated files to use Exq.enqueue instead of Task.start:
- lib/towerops/snmp/poller_worker.ex
- lib/towerops/monitoring/device_monitor.ex
- lib/towerops_web/channels/agent_channel.ex
Each module includes enqueue_* helper functions that:
- Use Task.start in test environment for synchronous execution
- Use Exq.enqueue in dev/prod for proper background job processing
Added "monitoring" queue to Exq configuration in application.ex
All tests passing (824 tests, 0 failures)
- Create DiscoveryWorker for handling SNMP device discovery
- Replace Task.start with Exq.enqueue for discovery operations
- Enqueue jobs to 'discovery' queue with device_id parameter
- Add enqueue_discovery/1 helper that falls back to Task.start in test
- Discovery jobs now persistent and retryable via Exq
- Triggered on: new device creation, device update, manual rediscover button
- Add exq ~> 0.23 dependency with redix and elixir_uuid
- Configure Exq in application supervisor with 4 queues (default, discovery, polling, maintenance)
- Set Exq as included_application to prevent auto-start
- Only start Exq in dev/prod environments, skip in test
- Use :env config to determine runtime environment
- Configure 10 concurrent workers with exq namespace
- Connect to Redis/Valkey sidecar via existing :redis config
- Make device name field optional when SNMP is enabled (ICMP Only mode still requires name)
- Add database migration to make name column nullable
- Add conditional validation: name required only when snmp_enabled is false
- Update SNMP discovery to populate device name from sysName when empty
- Add helper text explaining SNMP name population behavior in form
- Add comprehensive tests for device name SNMP population
- Update test mocks to account for NetSnmp profile sensor discovery
- After creating a device, stay on /devices/new instead of navigating to device show page
- Display success flash message with creation confirmation
- Reset form to defaults (preserving selected site) for adding next device
- Allows users to quickly add multiple devices in a row without navigation
- Add tests for staying on page and adding multiple devices sequentially
- All 91 device-related tests passing with no Credo warnings
- Add tab selector in Monitoring Configuration section for choosing monitoring mode
- SNMP & ICMP mode: enables ICMP pings + SNMP discovery (default)
- ICMP Only mode: enables only ICMP ping monitoring, hides SNMP fields
- Monitoring mode controls snmp_enabled field automatically on save
- Test SNMP Connection button shows informational message when agent is configured
(agents handle SNMP testing during their polling cycle)
- Update all tests to use tab selector instead of setting snmp_enabled directly
- All 818 tests passing with no Credo warnings
- Add Monitoring.get_latency_data/2 to query successful checks with response times
- Support 'latency' sensor type in GraphLive.Show with configurable time ranges
- Add latency chart to device show page alongside other metric charts
- Add comprehensive tests for get_latency_data/2 (limit, since, failed checks)
- All 813 tests passing with no Credo warnings
Implemented hierarchical SNMP community string inheritance following TDD:
Schema changes:
- Updated Device.changeset to remove required validation for snmp_community
- Community string is now optional when SNMP is enabled
- Allows inheritance from site or organization level
Inheritance hierarchy:
- Device-level community (highest priority)
- Site-level community
- Organization-level community
- nil (no community set at any level)
Form updates:
- extract_snmp_config now resolves inherited community for SNMP testing
- validate_test_snmp_input provides clearer error message about inheritance
- SNMP test uses effective community from hierarchy
Tests:
- Added tests for creating devices with nil/empty community string
- Added comprehensive SNMP configuration inheritance tests
- Tests verify hierarchy: device > site > org > default
- All 28 device tests passing
All 808 tests passing (2 pre-existing alert notifier failures unrelated to this change)
No Credo warnings
Implemented ability to rename agents through a new edit page following TDD:
Backend changes:
- Added update_agent_token/2 in Agents context for updating agent names
- Added update_changeset/2 in AgentToken schema that only allows name updates
- Security: Token field cannot be changed, only name field is allowed
LiveView implementation:
- Created AgentLive.Edit module with mount, validate, and save handlers
- Created edit.html.heex template with form for renaming agents
- Added route /orgs/:org_slug/agents/:id/edit
- Added Edit button to agent show page header
- Validates organization ownership using Repo.get_by!
Tests:
- Added comprehensive context tests for update_agent_token/2
- Added LiveView tests for edit page functionality
- Tests cover: loading, validation, save/redirect, auth, and org ownership
All 801 tests passing, no Credo warnings.
Replace the revoke functionality with delete to properly clean up agent references.
When an agent is deleted:
- All direct device assignments are removed
- Agent is removed from site defaults
- Agent is removed from organization defaults
- Devices automatically fall back to site/org agents or cloud polling
This ensures no orphaned references and provides better clarity about
what happens to device monitoring when an agent is removed.
- Add delete_agent_token/1 function in Agents context
- Update LiveView to use delete_agent event instead of revoke_agent
- Update UI button from "Revoke" to "Delete" with clearer confirmation message
- Add comprehensive tests for delete functionality and fallback behavior
- All 792 tests passing, no Credo warnings
Change the default filter from 'all' to 'active' on the alerts page.
This shows only active alerts by default, which is more useful for
monitoring current issues. Users can still click 'All Alerts' to see
the complete history.
Change (optional) text color from zinc-500/400 to zinc-400/500 for
better contrast and lighter appearance. Applied to:
- Parent Site field label
- Location field label
- Description field label
- Agent Configuration heading
- SNMP Configuration heading
Add greyed out '(optional)' text to Parent Site, Location, and
Description field labels for better UX. The optional indicator uses
a lighter color (text-zinc-500/400) to distinguish it from the
required field label.
- Replace phx:copy event with CopyToClipboard LiveView hook for more
reliable clipboard functionality
- Add visual feedback (checkmark) when text is copied to clipboard
- Apply hook to both agent token and docker compose copy buttons
- Add readonly attribute to docker compose textarea
- Fix org settings page to stay on settings after save instead of
redirecting to dashboard
The clipboard copy now works correctly by using a dedicated LiveView
hook that directly handles the click event and clipboard API, rather
than relying on custom event dispatching which was causing issues.
Change 'explicitly assigned' to 'equipment-level override' in organization
settings to better reflect that this counts equipment with custom agent
assignments, not equipment directly assigned to organizations (which doesn't
exist - equipment always belongs to sites).
- Add Organization Settings link in top-right org menu dropdown
- Position it between Switch Org and Admin Panel
- Add gear icon for visual clarity
- Rename 'Settings' to 'User Settings' with user icon for clarity
- Organization Settings link only shows when current_organization is set
Adds forms to organization, site, and equipment pages to configure SNMP
community strings with hierarchical fallback.
Changes:
- Organization settings: Add SNMP version and community fields
- Site form: Add SNMP config section with inheritance notice
- Equipment form: Update SNMP fields to show inheritance source and effective values
The UI now shows:
- Where the SNMP config is coming from (equipment, site, organization, or default)
- What the effective values are when inheriting
- Visual indicators for overrides vs inheritance
All forms use the Equipment.get_snmp_config/1 function for consistent
hierarchical resolution.
Implements a hierarchical SNMP community configuration system where settings
can be defined at the organization level (global default), overridden at the
site level, or overridden at the equipment level.
Changes:
- Migration: Add snmp_version and snmp_community to organizations and sites
- Organization schema: Add SNMP fields with default version "2c"
- Site schema: Add SNMP fields that override organization defaults
- Equipment context: Add get_snmp_config/1 with hierarchical fallback logic
- Discovery: Use Equipment.get_snmp_config instead of equipment.snmp_community
- PollerWorker: Use Equipment.get_snmp_config instead of equipment.snmp_community
The fallback order is:
1. Equipment-level (highest priority)
2. Site-level
3. Organization-level (default)
All 785 tests passing.
Fixes an issue where storage usage was reporting values over 200,000% due to
incorrect SNMP values from the device. This can occur when:
- Device firmware has bugs in hrStorageUsed/hrStorageSize reporting
- SNMP allocation units differ between OIDs
- Values are read during device reconfiguration
Changes:
- PollerWorker: Cap percentage sensors at 100%, log warning with raw values
- Mikrotik profile: Cap storage usage at 100% during discovery
- NetSnmp profile: Cap memory usage at 100% during discovery
The raw percentage is logged before capping to help diagnose device issues.
All percentage sensors now have consistent validation across polling and discovery.
- Update GitLab CI deploy job to set DEPLOY_TIMESTAMP when deploying
- Use 'kubectl set env' to update all pods with same timestamp
- Change deployment.yaml to use static placeholder value
- Update documentation to explain GitLab CI timestamp approach
This ensures all pods show the same deployment time, regardless of:
- Which pod handles the request (with replicas=2)
- When individual pods were created or restarted
- Pod restarts from crashes or rolling updates
The timestamp now represents when the deployment was initiated by GitLab CI,
not when individual pods were created.
- Read DEPLOY_TIMESTAMP environment variable for actual deploy time
- Fall back to compile time in development when env var not set
- Add Kubernetes Downward API to inject metadata.creationTimestamp
- Update k8s/deployment.yaml to set DEPLOY_TIMESTAMP env var
- Document deployment timestamp mechanism in k8s/README.md
This ensures the footer shows when the pod was deployed to the cluster,
not when the Docker image was built in CI/CD.
- Add compile-time @build_timestamp to Application module
- Create ToweropsWeb.TimeHelpers module for shared time formatting
- Update footer to display 'Last deployed Xm ago · YYYY-MM-DD HH:MM:SS UTC'
- Refactor AgentLive.Helpers to delegate to shared TimeHelpers
- Consolidate duplicate time formatting logic into single module
- Timestamp updates automatically on each build/deploy
- Add @spec to AgentSocket callbacks (connect/3, id/1)
- Add @spec to AgentChannel callbacks (join/3, handle_info/2, handle_in/3)
- Add @spec to AgentChannel private function build_jobs_for_agent/1
- Add @spec to Agents.verify_agent_token/1
- Add @spec to Agents.update_agent_token_heartbeat/3
- Add @spec to Agents.list_agent_polling_targets/1
These specs provide strict type checking for the critical WebSocket
agent communication code implemented in this session.
- Change default agent image to gmcintire/towerops-agent:main
- Add Watchtower service to docker-compose instructions
- Configure 15-minute update check interval for development
- Add DOCKER_API_VERSION=1.44 to prevent API compatibility errors
- Remove deprecated docker-compose version field
