send check_jobs protobuf to agents during job dispatch cycle, handle
check_result messages back with state transitions and alert
creation/resolution. auto-assign checks to device's effective agent
token on creation, broadcast check changes via PubSub so connected
agents get immediate updates.
add edit/delete support for service checks on device show page with
form pre-population from existing check config. auto-fill TCP host
from device IP address.
includes validate_check_result/1 for protobuf validation and
list_checks_for_agent/2 for querying agent-assigned service checks.
also fixes pre-existing broken device_type_icon reference in device
index template.
Reviewed-on: graham/towerops-web#50
- add MobileSocket for token-authenticated mobile WebSocket connections
- add MobileChannel with org-level and device-level real-time events
- add /mobile/socket endpoint for iOS app connections
- add "Link Mobile App" to user dropdown menu for discoverability
- improve QR code page with numbered steps and clearer instructions
- add socket/channel tests (17 total)
The server only sent polling/ping jobs to agents once on WebSocket join
and never re-dispatched them. After the first batch completed (~5-10s),
devices were never polled again until the agent reconnected.
:send_jobs now schedules a follow-up dispatch every 60s (configurable).
Unifies the timer management with :assignments_changed to prevent
accumulation.
Previous fix only resolved alerts when device status changed from down to up.
This didn't help devices already marked as 'up' with stuck unresolved alerts.
Now:
- Always check for stuck device_down alerts after successful SNMP poll
- Also check for stuck device_up alerts (should always be auto-resolved)
- Logs when stuck alerts are found and resolved
- Works for existing stuck alerts without waiting for status to change
This fixes the 8 stuck alerts in production - they'll auto-resolve on next poll.
When agents poll devices via SNMP and get successful results, the code was
only processing sensor/interface data without updating device status or
resolving alerts. This caused device_down alerts to persist even after
devices came back online.
This fix:
- Updates device status to 'up' when SNMP polling succeeds
- Auto-resolves any active device_down alerts
- Creates device_up alert (matching DeviceMonitorWorker behavior)
- Broadcasts device_status_changed via PubSub
Also fixes device_up alert creation to include resolved_at timestamp
(matching DeviceMonitorWorker behavior).
Test coverage: Added test verifying auto-resolution when device recovers.
CRITICAL fixes:
- Mask SNMP community string in agent channel logs (CRITICAL-1)
- Remove snmpv3 passwords from REST API responses, return _set booleans (CRITICAL-2)
- Replace snmp_community with snmp_community_set in GraphQL type (CRITICAL-3)
HIGH fixes:
- Fix cookie same_site from invalid 'Towerops' to 'Lax' (HIGH-4)
- Remove unsafe-eval from CSP script-src (HIGH-6)
- Block GraphQL introspection queries in production (HIGH-7)
- Sanitize LIKE wildcards in SNMP device name search (HIGH-8)
- Reject webhooks when no secret configured instead of accepting (HIGH-9)
MEDIUM fixes:
- Hash mobile session tokens (SHA-256) before DB storage (MEDIUM-10)
- Apply security headers in all environments, not just prod (MEDIUM-14)
- Add GraphQL query complexity limit (500) in production (MEDIUM-16)
- Fix X-Frame-Options to DENY to match frame-ancestors 'none' (MEDIUM-13)
- Fix broken route paths in dashboard (/discovery, /subscribers/trace, /config-changes)
- Fix insights empty state settings link (org-scoped route)
- Add underscores to all 86400 literals across 6 files
- Alphabetize aliases in search.ex and agent_channel.ex
- Extract changelog parser helper to reduce nesting
- Extract dashboard impact calculation to reduce nesting
- Refactor agent_channel config change detection (pattern match, extract function)
- Combine double Enum.reject into single call in trace.ex
- Fix line length in trace.ex search query
- Replace length/1 > 0 with != [] in trace_live
- Run mix format on all files
- create MonitoringCheck schema for the monitoring_checks table, fixing
silent data loss where ping results were dropped by the wrong schema
- update agent channel and device monitor worker to use new schema
- fix Stats queries to use MonitoringCheck instead of Check schema
- add list_online_cloud_pollers and list_cloud_polled_devices queries
- add CloudLatencyProbeWorker to dispatch cross-agent latency probes
every 8 hours via PubSub to all online cloud pollers
- scope AgentLatencyEvaluator to cloud pollers only with cloud_only
filter, lower min_checks (5), wider time window (48h), and
device-level assignments to avoid changing site/org defaults
- update cron schedule: probes at 0/8/16h, evaluation at 0:30/8:30/16:30
- refactor monitoring.ex and http_executor.ex to fix credo complexity
Enhanced AgentChannel logging to track device IDs across job list updates.
Added tests verifying server correctly excludes deleted devices from job lists.
Root cause: Bug is in Go agent (appends jobs without clearing), not Phoenix server.
Always broadcast from update_device_status/2 so the device index
LiveView refreshes even when status hasn't changed. Add a 15-second
tick timer to keep "Last Checked" timestamps fresh between polls.
Scope alert PubSub topics to organization for defense-in-depth.
Replace individual Repo.insert() with Repo.insert_all() for sensor readings,
interface stats, processor readings, and storage readings in agent channel and
device poller worker. Add partial/composite database indexes for common query
patterns. Dashboard now uses GROUP BY aggregation instead of loading all devices.
DeviceLive.Show only reloads data for the active tab on PubSub events.
DeviceLive.Index debounces status changes and skips quota queries on updates.
Agent polling preloads filtered to monitored sensors/interfaces only.
- use 64-bit HC counters (ifHCInOctets/ifHCOutOctets) instead of 32-bit
counters that wrap every ~34s on gigabit links
- fix negative integer parsing in replay adapter so sensor scale values
like -3 produce correct divisors (1000) instead of falling through to 1
- replace walking entire Cisco enterprise tree with targeted subtrees to
avoid timeouts; add Juniper, HP/H3C, Fortinet, Cambium vendor OIDs
- add missing discovery OIDs: ENTITY-STATE-MIB, HOST-RESOURCES-MIB
tables, and UCD-SNMP-MIB Linux CPU stats fallback
strip leading dots from OID keys returned by agent so they match
sensor OIDs stored without leading dots. fixes ubiquiti airos
wireless sensors (clients, ccq, frequency, etc) not being polled.
also fix flaky sensor_change_detector tests using global pubsub topic.
extract sensor change detection from DevicePollerWorker into shared
SensorChangeDetector module. add sensor_value_changed event for non-%
sensors when value differs from last_value. agent channel now detects
changes and updates last_value, matching phoenix poller behavior.
division by sensor_divisor produced full float precision values
like 34.09999999. apply Float.round/2 in both the phoenix poller
and agent channel paths.
store_monitoring_check saved ping results from agents but never
updated device status. DeviceMonitorWorker skips agent-assigned
devices, so nothing ever moved them from :unknown to :up/:down.
added update_device_status_from_check to derive status from check
result, update the device, create alerts on transitions, and
broadcast via PubSub — matching DeviceMonitorWorker behavior.
newly discovered devices sat idle with no metric data until the
next scheduling cycle. now the channel sends a poll job (and ping
job if monitoring enabled) right after discovery succeeds, using
send(self(), ...) so discovery DB writes commit first.
also reduces k8s replicas from 3 to 2.
Older agents send PING job results as SnmpResult messages instead of
MonitoringCheck messages, causing FunctionClauseError when
process_job_result/3 tried to pattern match on job_type: :PING.
Added handler that:
- Catches PING results in SnmpResult format
- Logs warning that agent should be upgraded
- Creates monitoring check with success status
- Stores result via store_monitoring_check
This prevents crashes while maintaining backward compatibility until
all production agents are upgraded to use MonitoringCheck format.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add server-initiated restart and self-update capabilities for remote
agents. Restart broadcasts on the lifecycle PubSub topic causing the
agent to exit cleanly for Docker to restart. Self-update fetches the
latest release from GitHub, finds the matching binary for the agent's
architecture, and sends the download URL with SHA256 checksum.
- Add restart_agent/1 and update_agent/3 to Agents context
- Handle :restart_requested and :update_requested in AgentChannel
- Add superuser-only Restart and Update buttons on agent show page
- Add ReleaseChecker module for GitHub Releases API with 5-min cache
- Add arch field to AgentHeartbeat protobuf for architecture detection
- Store arch in agent metadata from heartbeat
- Add Towerops.TaskSupervisor to supervision tree, replace bare spawn/1
in agent channel with supervised tasks for polling data processing
- Change DiscoveryWorker from max_attempts: 1 to 3 with retryable error
classification (transient errors retry, permanent errors discard)
- Name all 9 parallel polling tasks and log timeouts/crashes instead of
silently dropping results via Stream.run()
- Add tests for TaskSupervisor presence and error classification
This commit addresses Phase 1 (CRITICAL - Data Loss Prevention) issues
from the comprehensive bugs and inconsistencies analysis.
CRITICAL FIXES:
1. Protobuf Schema Inconsistencies
- Added missing job_id field (field 5) to SnmpResult message
- Fixed AgentError message structure (job_id, message, timestamp)
- Removed obsolete transport field from SnmpDevice struct
- Regenerated protobuf code with protoc-gen-elixir
- Prevents crashes when agent sends SNMP results or errors
2. Interface/Sensor Discovery Timeout Data Loss
- Changed timeout behavior to return error instead of empty list
- Prevents deletion of ALL existing interfaces/sensors on slow networks
- Discovery fails gracefully instead of destroying historical data
- Addresses commit 7a57f7c sensor discovery pipeline data loss issue
3. Polling Offset Schedule Mismatch
- Fixed schedule_next_poll() to use offset only (not interval + offset)
- Maintains consistent polling intervals across all executions
- Prevents load bunching that offset was designed to prevent
- Example: 300s interval with offset=94s now consistently polls every 5m
4. Always Reschedule Race Condition
- Added should_continue_polling?() and should_continue_monitoring?()
- Only reschedule if device is still enabled and should be polled by Phoenix
- Prevents zombie jobs from continuing to poll disabled/deleted/reassigned devices
- Stops race condition where stop_polling() is bypassed by in-flight jobs
Files Changed:
- priv/proto/agent.proto
- lib/towerops/proto/agent.pb.ex (regenerated)
- lib/towerops_web/channels/agent_channel.ex
- lib/towerops/snmp/discovery.ex
- lib/towerops/workers/device_poller_worker.ex
- lib/towerops/workers/device_monitor_worker.ex
- test/towerops/workers/device_poller_worker_test.exs
Test Results: All passing
- device_poller_worker_test.exs: 8 tests, 0 failures
- device_monitor_worker_test.exs: 3 tests, 0 failures
- discovery_test.exs: 13 tests, 0 failures
Remaining Phases:
- Phase 2 (HIGH): 8 data integrity issues
- Phase 3 (MEDIUM): 10 reliability issues
- Phase 4 (LOW): 3 cleanup issues
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Enables Phoenix to send PING jobs to agents and receive monitoring
check results, fixing the "unknown" status for devices assigned to
local (non-cloud) agents.
Changes:
- Rename JobType::MONITOR to JobType::PING in protobuf for clarity
- Add validate_monitoring_check_message() to Validator module
- Add "monitoring_check" channel handler to receive ping results
- Implement store_monitoring_check() to save results to database
- Add build_ping_job() to generate PING jobs for agents
- Update build_jobs_for_device() to include ping jobs when
monitoring_enabled=true
Phoenix now sends PING jobs alongside SNMP/MikroTik jobs, and agents
send back MonitoringCheck results that are stored in the
monitoring_checks table with proper agent_token_id tracking.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Move last_discovery_at update to after successful processing so failed
discoveries are retried on the next poll cycle instead of being blocked
for 24 hours
- Add disabled assignment check to agent polling target query so devices
with enabled=false assignments don't leak through site/org cascade
- Wrap spawned polling data processing in try/rescue so crashes are logged
instead of silently disappearing
Log SNMPv3 credentials being sent to agents (with passwords redacted) to help diagnose authentication issues. Shows security level, username, auth/priv protocols, and whether passwords are present.
Changed from one large WALK query with all OIDs to separate WALK queries
per table type. This prevents a failure in one unsupported table (like ARP
on a device that doesn't support it) from killing the entire polling
operation.
Before: Single WALK with neighbor + ARP + MAC + IP + HOST-RESOURCES OIDs
After: Separate WALKs for each table type
This makes polling more resilient - if a device doesn't support ARP tables
or HOST-RESOURCES-MIB, those WALKs will fail individually while neighbors,
MAC tables, and IP addresses can still succeed.
Fixes agent errors where a single SNMP receive error was killing the
entire poller thread and failing all subsequent operations with
'Poller thread died'.
Fixes KeyError when processing processors/storage during agent polling.
The sync_processors and sync_storage functions expect device.id to be
the snmp_device id, not the main device id.
Changed from:
%{device_id: device.id}
To:
%{id: device.snmp_device.id}
This matches the expected structure for these sync functions which
query where snmp_device_id == device.id.
Fixes KeyError when processing neighbor discovery during agent polling.
NeighborDiscovery.discover_neighbors/2 requires interfaces to have a
:device_id field for building neighbor records, but interfaces from the
database only have :snmp_device_id.
Solution matches the approach in DevicePollerWorker which adds device_id
to interfaces before calling discover_neighbors.
Replaces agent-supplied timestamps with server-side DateTime.utc_now()
to prevent clock drift issues between agents from causing weird graphs.
All metrics now use consistent server time regardless of polling method:
- Agent polling (via WebSocket): server time on receipt
- Phoenix polling (direct SNMP): server time on poll (already correct)
This ensures time-series data aligns correctly when graphing metrics
from devices polled by different agents with potentially different
system clocks.
Extended agent polling to collect all available SNMP data during regular
polls, not just sensors and interfaces. This ensures data stays current
and reduces reliance on periodic discovery runs.
Changes:
- Added polling queries for: neighbors (LLDP/CDP), ARP tables, MAC tables,
IP addresses, HOST-RESOURCES processors and storage
- Process additional data using Replay adapter pattern to reuse existing
parsing logic from discovery modules
- Made discovery helper functions public for reuse in polling context:
save_neighbors/2, save_arp_entries/3, sync_ip_addresses/2,
sync_processors/2, sync_storage/2
- Optimized by fetching interfaces once and reusing across operations
- Fixed compilation warnings by using proper guard syntax (list != [])
Technical details:
- Detection threshold: polls with more OIDs than sensors+interfaces
are processed for additional data
- Background processing: spawns async process to avoid blocking metric
recording
- Replay adapter: allows reusing discovery parsing logic without
re-querying SNMP device
Fixes missing IP addresses tab on device page after agent-based discovery.
Issue:
Discovery was not collecting IP address data because the agent's
discovery job didn't include IP address table OID walks.
Root cause:
build_discovery_queries() walked interface tables, sensors, vendor MIBs,
and neighbors, but was missing the IP-MIB address tables.
Changes:
- Add new SnmpQuery to walk IP address tables during discovery:
- 1.3.6.1.2.1.4.20 (ipAddrTable - IPv4, deprecated but widely supported)
- 1.3.6.1.2.1.4.34 (ipAddressTable - RFC 4293, unified IPv4/IPv6)
This matches the OIDs used by Phoenix's direct discovery in
lib/towerops/snmp/profiles/base.ex discover_all_ip_addresses/1.
Result:
Agent now collects IP address data during discovery, which is then
synced to the database and displayed in the IP Addresses tab.
Tests: 5578 tests passing (1 pre-existing failure in unrelated Dahua test)
