- Move last_discovery_at update to after successful processing so failed
discoveries are retried on the next poll cycle instead of being blocked
for 24 hours
- Add disabled assignment check to agent polling target query so devices
with enabled=false assignments don't leak through site/org cascade
- Wrap spawned polling data processing in try/rescue so crashes are logged
instead of silently disappearing
Log SNMPv3 credentials being sent to agents (with passwords redacted) to help diagnose authentication issues. Shows security level, username, auth/priv protocols, and whether passwords are present.
Changed from one large WALK query with all OIDs to separate WALK queries
per table type. This prevents a failure in one unsupported table (like ARP
on a device that doesn't support it) from killing the entire polling
operation.
Before: Single WALK with neighbor + ARP + MAC + IP + HOST-RESOURCES OIDs
After: Separate WALKs for each table type
This makes polling more resilient - if a device doesn't support ARP tables
or HOST-RESOURCES-MIB, those WALKs will fail individually while neighbors,
MAC tables, and IP addresses can still succeed.
Fixes agent errors where a single SNMP receive error was killing the
entire poller thread and failing all subsequent operations with
'Poller thread died'.
Fixes KeyError when processing processors/storage during agent polling.
The sync_processors and sync_storage functions expect device.id to be
the snmp_device id, not the main device id.
Changed from:
%{device_id: device.id}
To:
%{id: device.snmp_device.id}
This matches the expected structure for these sync functions which
query where snmp_device_id == device.id.
Fixes KeyError when processing neighbor discovery during agent polling.
NeighborDiscovery.discover_neighbors/2 requires interfaces to have a
:device_id field for building neighbor records, but interfaces from the
database only have :snmp_device_id.
Solution matches the approach in DevicePollerWorker which adds device_id
to interfaces before calling discover_neighbors.
Replaces agent-supplied timestamps with server-side DateTime.utc_now()
to prevent clock drift issues between agents from causing weird graphs.
All metrics now use consistent server time regardless of polling method:
- Agent polling (via WebSocket): server time on receipt
- Phoenix polling (direct SNMP): server time on poll (already correct)
This ensures time-series data aligns correctly when graphing metrics
from devices polled by different agents with potentially different
system clocks.
Extended agent polling to collect all available SNMP data during regular
polls, not just sensors and interfaces. This ensures data stays current
and reduces reliance on periodic discovery runs.
Changes:
- Added polling queries for: neighbors (LLDP/CDP), ARP tables, MAC tables,
IP addresses, HOST-RESOURCES processors and storage
- Process additional data using Replay adapter pattern to reuse existing
parsing logic from discovery modules
- Made discovery helper functions public for reuse in polling context:
save_neighbors/2, save_arp_entries/3, sync_ip_addresses/2,
sync_processors/2, sync_storage/2
- Optimized by fetching interfaces once and reusing across operations
- Fixed compilation warnings by using proper guard syntax (list != [])
Technical details:
- Detection threshold: polls with more OIDs than sensors+interfaces
are processed for additional data
- Background processing: spawns async process to avoid blocking metric
recording
- Replay adapter: allows reusing discovery parsing logic without
re-querying SNMP device
Fixes missing IP addresses tab on device page after agent-based discovery.
Issue:
Discovery was not collecting IP address data because the agent's
discovery job didn't include IP address table OID walks.
Root cause:
build_discovery_queries() walked interface tables, sensors, vendor MIBs,
and neighbors, but was missing the IP-MIB address tables.
Changes:
- Add new SnmpQuery to walk IP address tables during discovery:
- 1.3.6.1.2.1.4.20 (ipAddrTable - IPv4, deprecated but widely supported)
- 1.3.6.1.2.1.4.34 (ipAddressTable - RFC 4293, unified IPv4/IPv6)
This matches the OIDs used by Phoenix's direct discovery in
lib/towerops/snmp/profiles/base.ex discover_all_ip_addresses/1.
Result:
Agent now collects IP address data during discovery, which is then
synced to the database and displayed in the IP Addresses tab.
Tests: 5578 tests passing (1 pre-existing failure in unrelated Dahua test)
Add detailed error logging to show specific validation error type and
message when SNMP results fail validation. Also handle base64 decode
failures separately.
This will help diagnose validation issues introduced by recent strict
protobuf validation changes (commit 8044a6d1).
Adds @spec annotations to all public functions in agent-related modules for
better static analysis and documentation.
Changes:
- Added 17 typespecs to Towerops.Agents context module
- All CRUD operations (create, read, update, delete, revoke)
- Assignment management functions
- Agent token resolution and lookup functions
- PubSub broadcast function
- Added 9 typespecs to Towerops.Agents.Stats module
- Agent health and metrics statistics
- Device assignment breakdowns
- Latency analysis and reassignment candidate detection
- Added typespec to Towerops.Workers.StaleAgentWorker
- find_stale_agents/0 function
- Preserved existing typespecs in AgentChannel (socket types)
Benefits:
- Improved code documentation with clear function signatures
- Better static analysis via dialyzer
- Clearer API contracts for all agent management functions
- Type-safe UUID handling throughout agent system
Implements comprehensive validation layer on top of protobuf schema to
prevent malformed data, DoS attacks, and business logic violations.
Changes:
- Created Towerops.Agent.Validator module with strict validation for all
agent message types (AgentHeartbeat, MetricBatch, SnmpResult, AgentError,
CredentialTestResult, MikrotikResult)
- Added validation limits to prevent memory exhaustion and DoS attacks
(max string lengths, collection sizes, numeric ranges)
- Added format validation for UUIDs, IP addresses, versions, hostnames
- Added enum validation for status and protocol fields
- Modified AgentChannel handlers to validate all incoming messages before
processing
- Added comprehensive typespecs to AgentChannel for better static analysis
- Created safe_base64_decode/1 helper with error handling
- Added 32 comprehensive tests covering valid cases, invalid inputs, and
edge cases
Security improvements:
- Prevents memory exhaustion via oversized strings and collections
- Validates all numeric fields are within expected ranges
- Checks UUID format for all ID fields
- Validates IP addresses and hostnames against RFC standards
- Rejects future timestamps and excessive values
Backward compatibility:
- Empty version and hostname strings allowed for older agents
Changes:
- Fix traffic graph to handle nil interface octets gracefully
- Add comprehensive tests for nil octets scenarios
- Fix agent polling query to support devices assigned directly to org (left join on sites)
- Fix test SNMP connection to inherit credentials from site/org
- Refactor agent_channel process_polling_result to reduce nesting (Credo)
- Add alias for JobCleanupTask in application.ex (Credo)
This fixes crashes when interface stats have nil octets and ensures
devices without a site assignment can be polled by agents.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Discovery is now working successfully with SNMPv3 after increasing
agent timeout to 60s. Removing the temporary debug logging that was
added to verify credentials were being properly decrypted and sent
to the agent.
Added logging to show what SNMPv3 parameters are being sent from Phoenix
to the agent, including password lengths to verify they're decrypted
properly.
Fixes three critical issues affecting agent-based SNMP polling:
1. MikroTik gauge sensors incorrectly showing 10x values (223°C instead of 22.3°C)
- Root cause: get_int_value() rejected zero values, causing mtxrGaugeUnit
to be treated as nil and bypassing divisor logic
- Fixed: Accept zero values and apply correct divisor=10 for temp/voltage/current/power
2. Empty SNMP community strings sent to agents causing authentication failures
- Root cause: Devices with source="device" but null community weren't falling
back to organization/site inheritance
- Fixed: Enhanced resolve_snmp_community() to fall back to inheritance chain
even when source="device" but community is empty
3. Continuous MikroTik API commands sent every 60s instead of once per 24h
- Root cause: MikroTik jobs built on every poll cycle, not just discovery
- Fixed: Only build MikroTik jobs during discovery phase
Migrations update existing data to fix incorrect divisor values and community
source tracking. Added comprehensive debug logging to diagnose future issues.
Cloud pollers (is_cloud_poller = true, organization_id = nil) were
being blocked from polling devices because verify_device_organization
was checking if the device's organization matched the agent's
organization.
Since cloud pollers don't belong to any organization (organization_id
is nil), they should be allowed to poll devices from any organization.
Changes:
- verify_device_organization now allows access if organization_id is nil
- This fixes "Device not in agent's organization" errors for cloud pollers
Error before:
Device b7e8dd99-0b19-496d-97ae-9f2601d16464 not in agent's organization
Fixed dialyzer warnings across multiple files by properly handling return
values from functions that are called for side effects.
Changes:
- lib/towerops/agents.ex: Explicitly match broadcast and if-expression returns
- lib/towerops/devices.ex: Match worker start function returns in if-expressions
- lib/towerops/workers/agent_latency_evaluator.ex: Match broadcast return
- lib/towerops/workers/device_monitor_worker.ex: Match perform_check and handle_status_change returns, handle schedule_next_check errors
- lib/towerops/workers/device_poller_worker.ex: Match all PubSub broadcast returns, handle schedule_next_poll errors, remove unreachable pattern
- lib/towerops/workers/stale_agent_worker.ex: Match process_stale_agents and broadcast returns
- lib/towerops_web/channels/agent_channel.ex: Match PubSub.subscribe return, improve get_addr pattern matching, remove unreachable format_ip clause
- lib/towerops_web/live/device_live/form.ex: Match enqueue_discovery return, replace compile-time @dev_env with runtime check
Pattern applied:
- PubSub broadcasts: `_ = Phoenix.PubSub.broadcast(...)`
- If-expressions with side effects: `_ = if condition do ... end`
- Critical operations: Added error logging with case statements
All 3625 tests passing. No dialyzer warnings remaining in lib/towerops or lib/towerops_web.
🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Created three new Exq workers to handle background jobs:
- PollWorker: SNMP polling operations (queue: polling)
- MonitorWorker: Device monitoring checks (queue: monitoring)
- DiscoveryWorker: SNMP discovery (already existed)
Updated files to use Exq.enqueue instead of Task.start:
- lib/towerops/snmp/poller_worker.ex
- lib/towerops/monitoring/device_monitor.ex
- lib/towerops_web/channels/agent_channel.ex
Each module includes enqueue_* helper functions that:
- Use Task.start in test environment for synchronous execution
- Use Exq.enqueue in dev/prod for proper background job processing
Added "monitoring" queue to Exq configuration in application.ex
All tests passing (824 tests, 0 failures)
- Add @spec to AgentSocket callbacks (connect/3, id/1)
- Add @spec to AgentChannel callbacks (join/3, handle_info/2, handle_in/3)
- Add @spec to AgentChannel private function build_jobs_for_agent/1
- Add @spec to Agents.verify_agent_token/1
- Add @spec to Agents.update_agent_token_heartbeat/3
- Add @spec to Agents.list_agent_polling_targets/1
These specs provide strict type checking for the critical WebSocket
agent communication code implemented in this session.
