Fixes crashes in Exq.Node.Server when Redis connections fail by adding
health checks before startup and wrapping Exq in a custom supervisor
with better resilience.
Problem:
- Valkey pod restarting frequently (24 times in 28h) due to K8s Flannel CNI issues
- When Redis disconnects, Exq.Node.Server crashes trying to process_signals/2
- Gets nil from Redis instead of expected list, crashes on Enum.each/2
- Rapid crash/restart cycles reduce application stability
Changes:
1. RedisHealthCheck module (lib/towerops/redis_health_check.ex)
- Waits for Redis availability with exponential backoff before starting Exq
- Prevents Exq from starting when Redis is unavailable
- Handles connection errors gracefully without crashing caller processes
- Supports configurable retry attempts, timeouts, and backoff intervals
2. ExqSupervisor module (lib/towerops/exq_supervisor.ex)
- Custom supervisor that wraps Exq with health checks
- Uses one_for_one strategy with limited restarts (3 per 60s)
- Prevents rapid crash loops when Redis is unstable
- Increased Exq retry/backoff settings for better Redis recovery
- Allows application to continue functioning without background jobs if Redis unavailable
3. Updated Application.ex
- Replaced direct Exq start with ExqSupervisor
- Removed inline exq_config/0 function (moved to supervisor)
4. Tests
- Comprehensive tests for RedisHealthCheck (health checks, retries, timeouts)
- Tests for ExqSupervisor (configuration, restart strategy)
- All 1006 tests passing
Benefits:
- Prevents Exq crashes from propagating when Redis fails
- Application continues running even when Redis is temporarily unavailable
- Better logging of Redis connection issues
- More graceful degradation during infrastructure problems
Infrastructure Issue (to be addressed separately):
- K8s Flannel CNI plugin failing: /run/flannel/subnet.env not found
- Causing Valkey pod network issues and restarts
- Needs investigation of Flannel daemonset and node configuration
🤖 Generated with Claude Code
- Created 23 tests covering all public functions and edge cases
- Tests for IPv4 and IPv6 address validation
- Tests for timeout behavior and error handling
- Tests for PingBehaviour implementation
- All tests passing (23 tests, 0 failures)
- Module now has improved test coverage
Improves test coverage toward 90% target by adding tests for:
- Organizations context (bulk SNMP/agent operations)
- Sites context (hierarchical structure and bulk operations)
- Profiles context and schemas (device/sensor OID management)
- EctoTypes.JsonAny custom type
- Snmp.Neighbor schema (LLDP/CDP topology)
All new tests validate changeset validations, business logic,
and database constraints to ensure data integrity.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Make device name field optional when SNMP is enabled (ICMP Only mode still requires name)
- Add database migration to make name column nullable
- Add conditional validation: name required only when snmp_enabled is false
- Update SNMP discovery to populate device name from sysName when empty
- Add helper text explaining SNMP name population behavior in form
- Add comprehensive tests for device name SNMP population
- Update test mocks to account for NetSnmp profile sensor discovery
- Add Monitoring.get_latency_data/2 to query successful checks with response times
- Support 'latency' sensor type in GraphLive.Show with configurable time ranges
- Add latency chart to device show page alongside other metric charts
- Add comprehensive tests for get_latency_data/2 (limit, since, failed checks)
- All 813 tests passing with no Credo warnings
Implemented hierarchical SNMP community string inheritance following TDD:
Schema changes:
- Updated Device.changeset to remove required validation for snmp_community
- Community string is now optional when SNMP is enabled
- Allows inheritance from site or organization level
Inheritance hierarchy:
- Device-level community (highest priority)
- Site-level community
- Organization-level community
- nil (no community set at any level)
Form updates:
- extract_snmp_config now resolves inherited community for SNMP testing
- validate_test_snmp_input provides clearer error message about inheritance
- SNMP test uses effective community from hierarchy
Tests:
- Added tests for creating devices with nil/empty community string
- Added comprehensive SNMP configuration inheritance tests
- Tests verify hierarchy: device > site > org > default
- All 28 device tests passing
All 808 tests passing (2 pre-existing alert notifier failures unrelated to this change)
No Credo warnings
Implemented ability to rename agents through a new edit page following TDD:
Backend changes:
- Added update_agent_token/2 in Agents context for updating agent names
- Added update_changeset/2 in AgentToken schema that only allows name updates
- Security: Token field cannot be changed, only name field is allowed
LiveView implementation:
- Created AgentLive.Edit module with mount, validate, and save handlers
- Created edit.html.heex template with form for renaming agents
- Added route /orgs/:org_slug/agents/:id/edit
- Added Edit button to agent show page header
- Validates organization ownership using Repo.get_by!
Tests:
- Added comprehensive context tests for update_agent_token/2
- Added LiveView tests for edit page functionality
- Tests cover: loading, validation, save/redirect, auth, and org ownership
All 801 tests passing, no Credo warnings.
Replace the revoke functionality with delete to properly clean up agent references.
When an agent is deleted:
- All direct device assignments are removed
- Agent is removed from site defaults
- Agent is removed from organization defaults
- Devices automatically fall back to site/org agents or cloud polling
This ensures no orphaned references and provides better clarity about
what happens to device monitoring when an agent is removed.
- Add delete_agent_token/1 function in Agents context
- Update LiveView to use delete_agent event instead of revoke_agent
- Update UI button from "Revoke" to "Delete" with clearer confirmation message
- Add comprehensive tests for delete functionality and fallback behavior
- All 792 tests passing, no Credo warnings
Set explicit min/max bounds on chart x-axis to ensure all graphs
consistently show 24 hours of data, even when data points are sparse.
Also fix AlertNotifierTest race condition by disabling async execution.
Add three-level agent assignment hierarchy (Equipment > Site > Organization)
allowing flexible agent deployment strategies. Agents now receive only equipment
assigned to them through direct assignment or inheritance from site/organization
defaults.
Key changes:
- Add agent_token_id to Sites table with migration
- Implement get_effective_agent_token/1 for hierarchical resolution
- Add list_agent_polling_targets/1 to return polling targets per agent
- Update API config endpoint to use hierarchical polling targets
- Add agent assignment UI to equipment, site, and organization forms
- Show agent source (direct/site/org/none) in equipment form
- Add equipment count column to agent list
- Update terminology from "poll from server" to "cloud polling"
Tests:
- Add 8 comprehensive tests for list_agent_polling_targets/1
- Add end-to-end test for hierarchical config endpoint
- All 775 tests passing
Login flow improvements:
- Remove "Welcome back!" flash message on successful login
- Auto-login users clicking magic links without confirmation prompt
- Always use "remember me" for magic link logins
- Fix Accounts.login_user_by_magic_link/1 to handle invalid token format
Test fixes:
- Add @tag :integration to Ping tests (require actual system ping)
- Add halt() calls to UserAuth.start_impersonation error paths
Users now go directly to their intended destination after clicking a
magic link, with no interruption for "log in and stay logged in" choice.
- Test valid changesets with required and all fields
- Test validation for name, description, location lengths
- Test parent site relationships and circular reference validation
- Test all constraint validations
- Add comprehensive tests for Towerops.Monitoring.Ping module
- Test successful pings, timeouts, error handling, and OS detection
- Add tests for ToweropsWeb.OrgLive.New LiveView
- Test form validation, organization creation, and authentication
- Test deliver_update_email_instructions/2
- Test deliver_login_instructions/2 for confirmed and unconfirmed users
- Verify email content and recipients
- Coverage remains at 67.05%
- Test start_monitor/1 and stop_monitor/1
- Test start_snmp_poller/1 and stop_snmp_poller/1
- Test start_all_monitors/0 and start_all_snmp_pollers/0
- Verify process lifecycle management
- Coverage: 67.05% -> 67.09%
- Updated Monitoring module to calculate hourly/daily stats on-demand from monitoring_checks table
- Removed TimescaleDB continuous aggregate dependency (disabled due to licensing)
- Enabled all previously skipped TimescaleDB tests with proper test data
- Fixed UUID encoding for SQL queries using Ecto.UUID.dump!
- Fixed Decimal handling in uptime percentage calculation
- Added test for get_uptime_percentage with nil checks
- Monitoring module now at 100% coverage
Added 6 new tests to improve coverage:
- list_snmp_enabled_equipment/0 for SNMP-enabled equipment filtering
- update_snmp_poll_time/1 for updating polling timestamps
- create_event/1 for creating equipment events with proper schema
- list_equipment_events/2 for event retrieval and ordering
- list_equipment_events/2 with limit parameter
This improves Equipment coverage from 87.50% to higher.
Added 5 new tests to improve coverage:
- list_root_sites/1 for filtering root sites without parents
- list_child_sites/1 for getting child sites of a parent
- list_child_sites/1 edge case with no children
- get_site!/1 for verifying preload behavior with parent and child sites
This improves Sites coverage from 83.33% to higher.
Added 9 new tests to expand coverage:
- get_agent_token!/1 success and failure cases
- update_agent_token_heartbeat/3 edge cases (nil and empty metadata)
- get_equipment_assignment/1 for assigned and unassigned equipment
- update_equipment_assignment/2 for creating, updating, and removing assignments
This improves Agents coverage from 89.29% to higher.
Improved coverage from 60.00% to 100% by adding 13 tests covering:
- Required field validations (email, role, organization_id, invited_by_id)
- Email format validation
- Automatic token generation using crypto
- Token preservation when provided
- Automatic expires_at generation (7 days from now)
- Expires_at preservation when provided
- All valid role values (admin, member, viewer)
- Accepted_at field handling
Overall coverage: 66.34%
- Added comprehensive tests for ToweropsWeb.AgentLive.Index
- Tests mount, create_agent, revoke_agent, and close_token_modal events
- Tests agent status display and last_seen formatting
- Coverage improved from 57.50% to 80.00%
- Enhanced ToweropsWeb.UserAuth tests
- Added tests for impersonation functionality
- Added tests for organization loading and membership verification
- Added tests for superuser authorization
- Coverage improved from 51.08% to 61.87%
- Expanded Towerops.Snmp.Profiles.Cisco tests
- Added tests for all sensor type parsing functions
- Added tests for different sensor scales, statuses, and error conditions
- Tests sensor discovery with various data formats
- Coverage improved from 57.30% to 100.00%
Overall test coverage improved from 64.25% to 65.86%
Added extensive test suites to improve overall test coverage from 55.62% to 59.88%:
- Created Towerops.Snmp test suite (38 tests) - coverage: 10.87% → 97.83%
- Created Towerops.Snmp.Profiles.Cisco tests (14 tests) - coverage: 20.22% → 57.30%
- Created Towerops.Snmp.Profiles.NetSnmp tests (13 tests) - coverage: 56.79% → 92.59%
- Created Towerops.Monitoring.Supervisor tests (10 tests) - coverage: 21.21% → 63.64%
- Enhanced Towerops.Monitoring tests with TimescaleDB aggregate tests (skipped in test env)
Tests use Mox for SNMP client mocking with proper mock expectations for get/walk operations.
All tests follow established patterns and pass successfully.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add MIB files from LibreNMS in priv/mibs/ for reference
- Create MibParser module to validate OIDs against official MIB definitions
- Add MIB validation tests to ensure hardcoded OIDs match MIB specs
- Refactor SNMP tests to be generic/behavior-focused instead of vendor-specific
- Remove vendor-specific test files (cisco_test, net_snmp_test)
- All 104 tests passing with automated OID validation
Replace expensive length/1 comparisons with direct list comparisons or Enum.empty?/1 checks (13 instances in source code and tests). Add module aliases for Phoenix.HTML.Form and Towerops.Accounts.Scope to reduce nested module references.
Changes:
- Replace length(list) > 0 with list != [] or refute Enum.empty?(list)
- Add Phoenix.HTML.Form alias in CoreComponents
- Add Towerops.Accounts.Scope alias in ConnCase test helper
This eliminates all credo warnings and software design suggestions.
- Add EventLogger tests for PubSub event system
- Update navigation tests to match current UI patterns
- Fix alert notifier from address expectations
- Update site and equipment tests for relocated delete buttons
- Remove obsolete tests for removed UI elements (Recent Checks, monitoring widgets)
- All 254 tests now passing
- Add active_page attribute to authenticated layout
- Update nav_link component to show active state with border and text styling
- Update all LiveView templates to pass active_page parameter
- Fix datetime truncation issues in tests and schemas for :utc_datetime fields
- Fix dashboard test assertions for equipment status text
