- Add Towerops.TaskSupervisor to supervision tree, replace bare spawn/1
in agent channel with supervised tasks for polling data processing
- Change DiscoveryWorker from max_attempts: 1 to 3 with retryable error
classification (transient errors retry, permanent errors discard)
- Name all 9 parallel polling tasks and log timeouts/crashes instead of
silently dropping results via Stream.run()
- Add tests for TaskSupervisor presence and error classification
- Convert agent_tokens from assign to stream to prevent full page re-renders
- Update handle_info callbacks to only update the specific agent that changed
- Only update stats when agent status changes, not device counts
- Prevents IP address from temporarily disappearing during updates
Problem: upsert_mac_address was using Repo.one() which crashes with
Ecto.MultipleResultsError when duplicate MAC address records exist
in the database.
Root Cause: Race condition or historical data corruption created
duplicate entries for the same (device_id, mac_address, vlan_id)
tuple, violating the unique constraint.
Fix:
- Changed Repo.one() to Repo.all() to handle duplicates gracefully
- When duplicates found: keep first entry, delete extras, log warning
- Includes device_id, mac_address, vlan_id, and duplicate IDs in log
- Auto-cleans corrupted data while maintaining service availability
Example Log Output:
Found 1 duplicate MAC address entries, cleaning up
device_id=f3ff19d5-b519-4327-9f7d-dc416e494cfa
mac_address=78:9a:18:52:b1:c6
vlan_id=nil
duplicate_ids=[...]
Impact:
- Prevents polling crashes when duplicate MAC entries exist
- Self-healing: automatically cleans up database corruption
- Maintains data integrity while preserving service availability
Related: Issue 2.9 from bugs analysis (ARP/MAC parsing issues)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit addresses Phase 1 (CRITICAL - Data Loss Prevention) issues
from the comprehensive bugs and inconsistencies analysis.
CRITICAL FIXES:
1. Protobuf Schema Inconsistencies
- Added missing job_id field (field 5) to SnmpResult message
- Fixed AgentError message structure (job_id, message, timestamp)
- Removed obsolete transport field from SnmpDevice struct
- Regenerated protobuf code with protoc-gen-elixir
- Prevents crashes when agent sends SNMP results or errors
2. Interface/Sensor Discovery Timeout Data Loss
- Changed timeout behavior to return error instead of empty list
- Prevents deletion of ALL existing interfaces/sensors on slow networks
- Discovery fails gracefully instead of destroying historical data
- Addresses commit 7a57f7c sensor discovery pipeline data loss issue
3. Polling Offset Schedule Mismatch
- Fixed schedule_next_poll() to use offset only (not interval + offset)
- Maintains consistent polling intervals across all executions
- Prevents load bunching that offset was designed to prevent
- Example: 300s interval with offset=94s now consistently polls every 5m
4. Always Reschedule Race Condition
- Added should_continue_polling?() and should_continue_monitoring?()
- Only reschedule if device is still enabled and should be polled by Phoenix
- Prevents zombie jobs from continuing to poll disabled/deleted/reassigned devices
- Stops race condition where stop_polling() is bypassed by in-flight jobs
Files Changed:
- priv/proto/agent.proto
- lib/towerops/proto/agent.pb.ex (regenerated)
- lib/towerops_web/channels/agent_channel.ex
- lib/towerops/snmp/discovery.ex
- lib/towerops/workers/device_poller_worker.ex
- lib/towerops/workers/device_monitor_worker.ex
- test/towerops/workers/device_poller_worker_test.exs
Test Results: All passing
- device_poller_worker_test.exs: 8 tests, 0 failures
- device_monitor_worker_test.exs: 3 tests, 0 failures
- discovery_test.exs: 13 tests, 0 failures
Remaining Phases:
- Phase 2 (HIGH): 8 data integrity issues
- Phase 3 (MEDIUM): 10 reliability issues
- Phase 4 (LOW): 3 cleanup issues
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add logger filter to drop repetitive 'Cannot find module' errors
from net-snmp library. These errors occur when snmptranslate can't
find vendor-specific MIB modules but don't affect functionality
since the system falls back to numeric OIDs.
The errors were flooding production logs (hundreds of identical
messages) during polling of Morningstar devices that reference
TRISTAR MIB modules.
Changes:
- Add drop_snmp_mib_errors/2 filter to LoggerFilters module
- Register filter in production logger configuration
- Filter matches exact error format from snmptranslate stderr
The root cause (missing NIF binary) will be addressed separately
by compiling the towerops_nif.so during Docker build.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
When a device is deleted, now broadcasts an assignment change event
to the agent (if assigned), causing the agent to receive an updated
job list without the deleted device. This ensures agents stop
processing any in-flight jobs for deleted devices.
Changes:
- Get agent assignment before deleting device
- Broadcast :device_deleted event to agent after deletion
- Agent channel receives broadcast and sends updated job list
- Agents naturally stop processing jobs for devices not in new list
Previously only cancelled Oban jobs but didn't notify agents,
leaving potential for agents to continue processing jobs for
deleted devices.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Previously redirected to /dashboard or /orgs based on whether the
superuser had organizations. Now always goes to /devices for
consistency.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The protobuf Elixir module was missing the PING (value 4) enum
entry, causing encoding errors when building jobs for agents.
Manually added the field since protoc-gen-elixir has asdf version
issues. The generated module now matches the updated agent.proto
definition.
Fixes: ** (Protobuf.EncodeError) no function clause matching in
Towerops.Agent.JobType
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Enables Phoenix to send PING jobs to agents and receive monitoring
check results, fixing the "unknown" status for devices assigned to
local (non-cloud) agents.
Changes:
- Rename JobType::MONITOR to JobType::PING in protobuf for clarity
- Add validate_monitoring_check_message() to Validator module
- Add "monitoring_check" channel handler to receive ping results
- Implement store_monitoring_check() to save results to database
- Add build_ping_job() to generate PING jobs for agents
- Update build_jobs_for_device() to include ping jobs when
monitoring_enabled=true
Phoenix now sends PING jobs alongside SNMP/MikroTik jobs, and agents
send back MonitoringCheck results that are stored in the
monitoring_checks table with proper agent_token_id tracking.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
When an organization has sites disabled (use_sites=false), the API
now automatically removes any site_id from device creation requests,
ensuring devices are assigned directly to the organization instead.
This prevents Terraform (and other API clients) from creating devices
in sites when sites functionality is disabled for the organization.
Fixes issue where Terraform could bypass site restrictions via API.
Show the specific cloud poller name (e.g., dfw, lon) alongside "Cloud"
when a device is assigned to a cloud poller. Displays as "Cloud (dfw)"
instead of just "Cloud" for better visibility of which poller is being used.
When no agent is assigned, continues to show just "Cloud".
Display latency values with single decimal precision (e.g., 114.1ms)
for improved readability in graphs. This ensures consistency between
the raw chart data and tooltip formatting.
The previous commit (90e8964) introduced an incorrect LoggerBackends.add/2
call that prevented Oban from starting, causing all discovery jobs to remain
stuck in "available" state without being processed.
Root cause: LoggerBackends.add expects a tuple {module, id} as the first
argument, not the module alone with configuration options.
Fix: Use the correct API pattern:
1. Add backend with tuple identifier: {LoggerFileBackend, :file_log}
2. Configure separately with Logger.configure_backend/2
This resolves device discovery being completely broken - all background
jobs (discovery, polling, monitoring) now process correctly.
Critical fixes:
- Add [:safe] option to binary_to_term to prevent RCE attacks
- Implement whitelist validation for String.to_atom conversions
- Add input validation before String.to_existing_atom usage
Changes:
- MIB compiler and cache: Use safe binary deserialization
- SNMP contexts: Whitelist protocol, device type, and source atoms
- API controllers: Validate error message keys before atom conversion
- Reduce function nesting to comply with Credo standards
All 6,145 tests passing with zero Credo issues.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Move last_discovery_at update to after successful processing so failed
discoveries are retried on the next poll cycle instead of being blocked
for 24 hours
- Add disabled assignment check to agent polling target query so devices
with enabled=false assignments don't leak through site/org cascade
- Wrap spawned polling data processing in try/rescue so crashes are logged
instead of silently disappearing
Added integration tests for real-time updates and metrics display.
Also fixed dialyzer warnings in worker event broadcasting by removing
unreachable error handling branches.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add Job Monitoring card to admin dashboard with:
- Hero icon for visual identification
- Brief description of monitoring capabilities
- Link to /admin/monitoring dashboard
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
