Problem: upsert_mac_address was using Repo.one() which crashes with
Ecto.MultipleResultsError when duplicate MAC address records exist
in the database.
Root Cause: Race condition or historical data corruption created
duplicate entries for the same (device_id, mac_address, vlan_id)
tuple, violating the unique constraint.
Fix:
- Changed Repo.one() to Repo.all() to handle duplicates gracefully
- When duplicates found: keep first entry, delete extras, log warning
- Includes device_id, mac_address, vlan_id, and duplicate IDs in log
- Auto-cleans corrupted data while maintaining service availability
Example Log Output:
Found 1 duplicate MAC address entries, cleaning up
device_id=f3ff19d5-b519-4327-9f7d-dc416e494cfa
mac_address=78:9a:18:52:b1:c6
vlan_id=nil
duplicate_ids=[...]
Impact:
- Prevents polling crashes when duplicate MAC entries exist
- Self-healing: automatically cleans up database corruption
- Maintains data integrity while preserving service availability
Related: Issue 2.9 from bugs analysis (ARP/MAC parsing issues)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit addresses Phase 1 (CRITICAL - Data Loss Prevention) issues
from the comprehensive bugs and inconsistencies analysis.
CRITICAL FIXES:
1. Protobuf Schema Inconsistencies
- Added missing job_id field (field 5) to SnmpResult message
- Fixed AgentError message structure (job_id, message, timestamp)
- Removed obsolete transport field from SnmpDevice struct
- Regenerated protobuf code with protoc-gen-elixir
- Prevents crashes when agent sends SNMP results or errors
2. Interface/Sensor Discovery Timeout Data Loss
- Changed timeout behavior to return error instead of empty list
- Prevents deletion of ALL existing interfaces/sensors on slow networks
- Discovery fails gracefully instead of destroying historical data
- Addresses commit 7a57f7c sensor discovery pipeline data loss issue
3. Polling Offset Schedule Mismatch
- Fixed schedule_next_poll() to use offset only (not interval + offset)
- Maintains consistent polling intervals across all executions
- Prevents load bunching that offset was designed to prevent
- Example: 300s interval with offset=94s now consistently polls every 5m
4. Always Reschedule Race Condition
- Added should_continue_polling?() and should_continue_monitoring?()
- Only reschedule if device is still enabled and should be polled by Phoenix
- Prevents zombie jobs from continuing to poll disabled/deleted/reassigned devices
- Stops race condition where stop_polling() is bypassed by in-flight jobs
Files Changed:
- priv/proto/agent.proto
- lib/towerops/proto/agent.pb.ex (regenerated)
- lib/towerops_web/channels/agent_channel.ex
- lib/towerops/snmp/discovery.ex
- lib/towerops/workers/device_poller_worker.ex
- lib/towerops/workers/device_monitor_worker.ex
- test/towerops/workers/device_poller_worker_test.exs
Test Results: All passing
- device_poller_worker_test.exs: 8 tests, 0 failures
- device_monitor_worker_test.exs: 3 tests, 0 failures
- discovery_test.exs: 13 tests, 0 failures
Remaining Phases:
- Phase 2 (HIGH): 8 data integrity issues
- Phase 3 (MEDIUM): 10 reliability issues
- Phase 4 (LOW): 3 cleanup issues
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add logger filter to drop repetitive 'Cannot find module' errors
from net-snmp library. These errors occur when snmptranslate can't
find vendor-specific MIB modules but don't affect functionality
since the system falls back to numeric OIDs.
The errors were flooding production logs (hundreds of identical
messages) during polling of Morningstar devices that reference
TRISTAR MIB modules.
Changes:
- Add drop_snmp_mib_errors/2 filter to LoggerFilters module
- Register filter in production logger configuration
- Filter matches exact error format from snmptranslate stderr
The root cause (missing NIF binary) will be addressed separately
by compiling the towerops_nif.so during Docker build.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
When a device is deleted, now broadcasts an assignment change event
to the agent (if assigned), causing the agent to receive an updated
job list without the deleted device. This ensures agents stop
processing any in-flight jobs for deleted devices.
Changes:
- Get agent assignment before deleting device
- Broadcast :device_deleted event to agent after deletion
- Agent channel receives broadcast and sends updated job list
- Agents naturally stop processing jobs for devices not in new list
Previously only cancelled Oban jobs but didn't notify agents,
leaving potential for agents to continue processing jobs for
deleted devices.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Previously redirected to /dashboard or /orgs based on whether the
superuser had organizations. Now always goes to /devices for
consistency.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The protobuf Elixir module was missing the PING (value 4) enum
entry, causing encoding errors when building jobs for agents.
Manually added the field since protoc-gen-elixir has asdf version
issues. The generated module now matches the updated agent.proto
definition.
Fixes: ** (Protobuf.EncodeError) no function clause matching in
Towerops.Agent.JobType
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Enables Phoenix to send PING jobs to agents and receive monitoring
check results, fixing the "unknown" status for devices assigned to
local (non-cloud) agents.
Changes:
- Rename JobType::MONITOR to JobType::PING in protobuf for clarity
- Add validate_monitoring_check_message() to Validator module
- Add "monitoring_check" channel handler to receive ping results
- Implement store_monitoring_check() to save results to database
- Add build_ping_job() to generate PING jobs for agents
- Update build_jobs_for_device() to include ping jobs when
monitoring_enabled=true
Phoenix now sends PING jobs alongside SNMP/MikroTik jobs, and agents
send back MonitoringCheck results that are stored in the
monitoring_checks table with proper agent_token_id tracking.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
When an organization has sites disabled (use_sites=false), the API
now automatically removes any site_id from device creation requests,
ensuring devices are assigned directly to the organization instead.
This prevents Terraform (and other API clients) from creating devices
in sites when sites functionality is disabled for the organization.
Fixes issue where Terraform could bypass site restrictions via API.
Show the specific cloud poller name (e.g., dfw, lon) alongside "Cloud"
when a device is assigned to a cloud poller. Displays as "Cloud (dfw)"
instead of just "Cloud" for better visibility of which poller is being used.
When no agent is assigned, continues to show just "Cloud".
Display latency values with single decimal precision (e.g., 114.1ms)
for improved readability in graphs. This ensures consistency between
the raw chart data and tooltip formatting.
The previous commit (90e8964) introduced an incorrect LoggerBackends.add/2
call that prevented Oban from starting, causing all discovery jobs to remain
stuck in "available" state without being processed.
Root cause: LoggerBackends.add expects a tuple {module, id} as the first
argument, not the module alone with configuration options.
Fix: Use the correct API pattern:
1. Add backend with tuple identifier: {LoggerFileBackend, :file_log}
2. Configure separately with Logger.configure_backend/2
This resolves device discovery being completely broken - all background
jobs (discovery, polling, monitoring) now process correctly.
Critical fixes:
- Add [:safe] option to binary_to_term to prevent RCE attacks
- Implement whitelist validation for String.to_atom conversions
- Add input validation before String.to_existing_atom usage
Changes:
- MIB compiler and cache: Use safe binary deserialization
- SNMP contexts: Whitelist protocol, device type, and source atoms
- API controllers: Validate error message keys before atom conversion
- Reduce function nesting to comply with Credo standards
All 6,145 tests passing with zero Credo issues.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Move last_discovery_at update to after successful processing so failed
discoveries are retried on the next poll cycle instead of being blocked
for 24 hours
- Add disabled assignment check to agent polling target query so devices
with enabled=false assignments don't leak through site/org cascade
- Wrap spawned polling data processing in try/rescue so crashes are logged
instead of silently disappearing
Added integration tests for real-time updates and metrics display.
Also fixed dialyzer warnings in worker event broadcasting by removing
unreachable error handling branches.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add Job Monitoring card to admin dashboard with:
- Hero icon for visual identification
- Brief description of monitoring capabilities
- Link to /admin/monitoring dashboard
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add LiveView for real-time job monitoring dashboard:
- Subscribe to job:lifecycle PubSub topic for live updates
- Display health metrics (completed, failed, avg duration, active jobs)
- Show active operations with real-time updates
- Display problems section (stuck/failed job counts)
- Add router entry under /admin/monitoring
Tests verify PubSub subscription and basic rendering.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add Events.broadcast_job_event calls to track job lifecycle:
- :started when job begins executing
- :completed when job finishes (including :discard status)
- :failed when job crashes with error
- Track duration in seconds for all job executions
Wrap main logic in try/rescue to ensure events broadcast even on crash.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add Events.broadcast_job_event calls to track job lifecycle:
- :started when job begins executing
- :completed/:failed when job finishes
- Track duration in seconds for all job executions
Broadcast events enable real-time monitoring of polling operations.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Create JobMonitoring.Metrics module with calculate_all/0 to provide
comprehensive metrics including execution counts, success rates,
queue depths, and average execution times.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add list_failed_jobs/0 to return retryable and cancelled jobs
and list_recent_completions/1 to return completed, cancelled,
or discarded jobs with configurable limit.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The API was not returning SNMPv3 fields (security_level, username,
auth_protocol, auth_password, priv_protocol, priv_password) in device
responses, causing Terraform provider to see inconsistent state between
plan and apply.
Updated both format_device/1 and format_device_details/1 to include
all SNMPv3 fields in JSON responses.
Log SNMPv3 credentials being sent to agents (with passwords redacted) to help diagnose authentication issues. Shows security level, username, auth/priv protocols, and whether passwords are present.
Previously, if a device was deleted while a discovery job was in progress, the wait_loop would crash with Ecto.NoResultsError when trying to check if discovery had completed.
Now uses get_device_with_details which returns nil for missing devices, and gracefully returns {:error, :device_deleted} instead of crashing.
Changed from one large WALK query with all OIDs to separate WALK queries
per table type. This prevents a failure in one unsupported table (like ARP
on a device that doesn't support it) from killing the entire polling
operation.
Before: Single WALK with neighbor + ARP + MAC + IP + HOST-RESOURCES OIDs
After: Separate WALKs for each table type
This makes polling more resilient - if a device doesn't support ARP tables
or HOST-RESOURCES-MIB, those WALKs will fail individually while neighbors,
MAC tables, and IP addresses can still succeed.
Fixes agent errors where a single SNMP receive error was killing the
entire poller thread and failing all subsequent operations with
'Poller thread died'.
Fixes KeyError when processing processors/storage during agent polling.
The sync_processors and sync_storage functions expect device.id to be
the snmp_device id, not the main device id.
Changed from:
%{device_id: device.id}
To:
%{id: device.snmp_device.id}
This matches the expected structure for these sync functions which
query where snmp_device_id == device.id.
Fixes ArgumentError when upserting MAC addresses with nil vlan_id.
Ecto doesn't allow direct comparison with nil in queries - must use
is_nil/1 instead.
Changed upsert_mac_address to build dynamic query that uses is_nil(m.vlan_id)
when vlan_id is nil, or direct comparison when it has a value.
Error was:
Comparison with nil is forbidden as it is unsafe.
Instead write a query with is_nil/1
Fixes KeyError when processing neighbor discovery during agent polling.
NeighborDiscovery.discover_neighbors/2 requires interfaces to have a
:device_id field for building neighbor records, but interfaces from the
database only have :snmp_device_id.
Solution matches the approach in DevicePollerWorker which adds device_id
to interfaces before calling discover_neighbors.
Replaces agent-supplied timestamps with server-side DateTime.utc_now()
to prevent clock drift issues between agents from causing weird graphs.
All metrics now use consistent server time regardless of polling method:
- Agent polling (via WebSocket): server time on receipt
- Phoenix polling (direct SNMP): server time on poll (already correct)
This ensures time-series data aligns correctly when graphing metrics
from devices polled by different agents with potentially different
system clocks.
Extended agent polling to collect all available SNMP data during regular
polls, not just sensors and interfaces. This ensures data stays current
and reduces reliance on periodic discovery runs.
Changes:
- Added polling queries for: neighbors (LLDP/CDP), ARP tables, MAC tables,
IP addresses, HOST-RESOURCES processors and storage
- Process additional data using Replay adapter pattern to reuse existing
parsing logic from discovery modules
- Made discovery helper functions public for reuse in polling context:
save_neighbors/2, save_arp_entries/3, sync_ip_addresses/2,
sync_processors/2, sync_storage/2
- Optimized by fetching interfaces once and reusing across operations
- Fixed compilation warnings by using proper guard syntax (list != [])
Technical details:
- Detection threshold: polls with more OIDs than sensors+interfaces
are processed for additional data
- Background processing: spawns async process to avoid blocking metric
recording
- Replay adapter: allows reusing discovery parsing logic without
re-querying SNMP device
Fixes missing IP addresses tab on device page after agent-based discovery.
Issue:
Discovery was not collecting IP address data because the agent's
discovery job didn't include IP address table OID walks.
Root cause:
build_discovery_queries() walked interface tables, sensors, vendor MIBs,
and neighbors, but was missing the IP-MIB address tables.
Changes:
- Add new SnmpQuery to walk IP address tables during discovery:
- 1.3.6.1.2.1.4.20 (ipAddrTable - IPv4, deprecated but widely supported)
- 1.3.6.1.2.1.4.34 (ipAddressTable - RFC 4293, unified IPv4/IPv6)
This matches the OIDs used by Phoenix's direct discovery in
lib/towerops/snmp/profiles/base.ex discover_all_ip_addresses/1.
Result:
Agent now collects IP address data during discovery, which is then
synced to the database and displayed in the IP Addresses tab.
Tests: 5578 tests passing (1 pre-existing failure in unrelated Dahua test)
Shows IP address discovery and sync results at INFO level in dev
environment only to help diagnose discovery issues.
Changes:
- Add log_ip_discovery_results/2 helper to log discovered IPs
- Enhanced sync_ip_addresses/2 to log IPv4/IPv6 counts and removals
- Only logs at INFO level when env is :dev, uses DEBUG in prod
Example output in dev:
Discovered IP addresses: 5 IPv4, 2 IPv6 (total: 7)
Synced IP addresses: 5 IPv4, 2 IPv6, removed 1
This helps diagnose why IP addresses tab isn't showing on device page.
