- Make inotify-tools Linux-only (not available on Darwin)
- Replace nixfmt-rfc-style with nixfmt (deprecated warning)
- Add flake.lock for reproducible builds
Changes:
- nix/shell.nix: Conditionally include inotify-tools only on Linux
- flake.nix: Use pkgs.nixfmt instead of pkgs.nixfmt-rfc-style
- flake.lock: Lock dependencies for reproducibility
- Move last_discovery_at update to after successful processing so failed
discoveries are retried on the next poll cycle instead of being blocked
for 24 hours
- Add disabled assignment check to agent polling target query so devices
with enabled=false assignments don't leak through site/org cascade
- Wrap spawned polling data processing in try/rescue so crashes are logged
instead of silently disappearing
Comprehensive documentation covering architecture, usage, technical
details, and future enhancements for the job monitoring dashboard.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Added integration tests for real-time updates and metrics display.
Also fixed dialyzer warnings in worker event broadcasting by removing
unreachable error handling branches.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add Job Monitoring card to admin dashboard with:
- Hero icon for visual identification
- Brief description of monitoring capabilities
- Link to /admin/monitoring dashboard
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add LiveView for real-time job monitoring dashboard:
- Subscribe to job:lifecycle PubSub topic for live updates
- Display health metrics (completed, failed, avg duration, active jobs)
- Show active operations with real-time updates
- Display problems section (stuck/failed job counts)
- Add router entry under /admin/monitoring
Tests verify PubSub subscription and basic rendering.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add Events.broadcast_job_event calls to track job lifecycle:
- :started when job begins executing
- :completed when job finishes (including :discard status)
- :failed when job crashes with error
- Track duration in seconds for all job executions
Wrap main logic in try/rescue to ensure events broadcast even on crash.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add Events.broadcast_job_event calls to track job lifecycle:
- :started when job begins executing
- :completed/:failed when job finishes
- Track duration in seconds for all job executions
Broadcast events enable real-time monitoring of polling operations.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Create JobMonitoring.Metrics module with calculate_all/0 to provide
comprehensive metrics including execution counts, success rates,
queue depths, and average execution times.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add list_failed_jobs/0 to return retryable and cancelled jobs
and list_recent_completions/1 to return completed, cancelled,
or discarded jobs with configurable limit.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The API was not returning SNMPv3 fields (security_level, username,
auth_protocol, auth_password, priv_protocol, priv_password) in device
responses, causing Terraform provider to see inconsistent state between
plan and apply.
Updated both format_device/1 and format_device_details/1 to include
all SNMPv3 fields in JSON responses.
Log SNMPv3 credentials being sent to agents (with passwords redacted) to help diagnose authentication issues. Shows security level, username, auth/priv protocols, and whether passwords are present.
Previously, if a device was deleted while a discovery job was in progress, the wait_loop would crash with Ecto.NoResultsError when trying to check if discovery had completed.
Now uses get_device_with_details which returns nil for missing devices, and gracefully returns {:error, :device_deleted} instead of crashing.
Changed from one large WALK query with all OIDs to separate WALK queries
per table type. This prevents a failure in one unsupported table (like ARP
on a device that doesn't support it) from killing the entire polling
operation.
Before: Single WALK with neighbor + ARP + MAC + IP + HOST-RESOURCES OIDs
After: Separate WALKs for each table type
This makes polling more resilient - if a device doesn't support ARP tables
or HOST-RESOURCES-MIB, those WALKs will fail individually while neighbors,
MAC tables, and IP addresses can still succeed.
Fixes agent errors where a single SNMP receive error was killing the
entire poller thread and failing all subsequent operations with
'Poller thread died'.
Fixes KeyError when processing processors/storage during agent polling.
The sync_processors and sync_storage functions expect device.id to be
the snmp_device id, not the main device id.
Changed from:
%{device_id: device.id}
To:
%{id: device.snmp_device.id}
This matches the expected structure for these sync functions which
query where snmp_device_id == device.id.
Fixes ArgumentError when upserting MAC addresses with nil vlan_id.
Ecto doesn't allow direct comparison with nil in queries - must use
is_nil/1 instead.
Changed upsert_mac_address to build dynamic query that uses is_nil(m.vlan_id)
when vlan_id is nil, or direct comparison when it has a value.
Error was:
Comparison with nil is forbidden as it is unsafe.
Instead write a query with is_nil/1
Fixes KeyError when processing neighbor discovery during agent polling.
NeighborDiscovery.discover_neighbors/2 requires interfaces to have a
:device_id field for building neighbor records, but interfaces from the
database only have :snmp_device_id.
Solution matches the approach in DevicePollerWorker which adds device_id
to interfaces before calling discover_neighbors.
Replaces agent-supplied timestamps with server-side DateTime.utc_now()
to prevent clock drift issues between agents from causing weird graphs.
All metrics now use consistent server time regardless of polling method:
- Agent polling (via WebSocket): server time on receipt
- Phoenix polling (direct SNMP): server time on poll (already correct)
This ensures time-series data aligns correctly when graphing metrics
from devices polled by different agents with potentially different
system clocks.
Extended agent polling to collect all available SNMP data during regular
polls, not just sensors and interfaces. This ensures data stays current
and reduces reliance on periodic discovery runs.
Changes:
- Added polling queries for: neighbors (LLDP/CDP), ARP tables, MAC tables,
IP addresses, HOST-RESOURCES processors and storage
- Process additional data using Replay adapter pattern to reuse existing
parsing logic from discovery modules
- Made discovery helper functions public for reuse in polling context:
save_neighbors/2, save_arp_entries/3, sync_ip_addresses/2,
sync_processors/2, sync_storage/2
- Optimized by fetching interfaces once and reusing across operations
- Fixed compilation warnings by using proper guard syntax (list != [])
Technical details:
- Detection threshold: polls with more OIDs than sensors+interfaces
are processed for additional data
- Background processing: spawns async process to avoid blocking metric
recording
- Replay adapter: allows reusing discovery parsing logic without
re-querying SNMP device
Fixes missing IP addresses tab on device page after agent-based discovery.
Issue:
Discovery was not collecting IP address data because the agent's
discovery job didn't include IP address table OID walks.
Root cause:
build_discovery_queries() walked interface tables, sensors, vendor MIBs,
and neighbors, but was missing the IP-MIB address tables.
Changes:
- Add new SnmpQuery to walk IP address tables during discovery:
- 1.3.6.1.2.1.4.20 (ipAddrTable - IPv4, deprecated but widely supported)
- 1.3.6.1.2.1.4.34 (ipAddressTable - RFC 4293, unified IPv4/IPv6)
This matches the OIDs used by Phoenix's direct discovery in
lib/towerops/snmp/profiles/base.ex discover_all_ip_addresses/1.
Result:
Agent now collects IP address data during discovery, which is then
synced to the database and displayed in the IP Addresses tab.
Tests: 5578 tests passing (1 pre-existing failure in unrelated Dahua test)
Shows IP address discovery and sync results at INFO level in dev
environment only to help diagnose discovery issues.
Changes:
- Add log_ip_discovery_results/2 helper to log discovered IPs
- Enhanced sync_ip_addresses/2 to log IPv4/IPv6 counts and removals
- Only logs at INFO level when env is :dev, uses DEBUG in prod
Example output in dev:
Discovered IP addresses: 5 IPv4, 2 IPv6 (total: 7)
Synced IP addresses: 5 IPv4, 2 IPv6, removed 1
This helps diagnose why IP addresses tab isn't showing on device page.
Fixes discovery failures on enterprise network devices with many
interfaces and routing table entries.
Issue:
Discovery was failing with "too_many_oids - OID values map exceeds
1000 entries" on SNMPv3 devices. Enterprise routers/switches during
discovery can easily return 10k-50k OIDs from:
- 100+ interfaces with 20-30 OIDs each
- System info, routing tables, ARP tables, neighbor tables
- Various MIB objects
The original limit of 1,000 was too restrictive for real-world devices
while still being intended as DoS protection.
Changes:
- Increase @max_oid_values from 1,000 to 50,000
- Update test to generate 50,100 OIDs (exceeds new limit)
- Add comment explaining why limit is set to 50k
This maintains DoS protection (prevents multi-million OID attacks)
while accommodating realistic device discovery scenarios.
All tests passing (5578 tests, 0 failures).
Add detailed error logging to show specific validation error type and
message when SNMP results fail validation. Also handle base64 decode
failures separately.
This will help diagnose validation issues introduced by recent strict
protobuf validation changes (commit 8044a6d1).
Fixes 3 Dialyzer warnings about impossible comparisons when checking for NaN
and infinity values in sensor validation.
Problem:
Dialyzer correctly identified that comparing a float with atoms (:nan,
:infinity, :neg_infinity) using == can never evaluate to true. The previous
code attempted to validate sensor values by comparing with these atoms, which
is incorrect in Erlang/Elixir.
Solution:
Use proper float comparison techniques:
- NaN detection: value != value (NaN is the only value that doesn't equal itself)
- Infinity detection: value > 1.0e308 or value < -1.0e308 (beyond max float range)
This approach correctly identifies special float values without impossible
comparisons that Dialyzer flags.
Dialyzer results:
- Before: Total errors: 509, Skipped: 506
- After: Total errors: 506, Skipped: 506
- Status: done (passed successfully), EXIT CODE: 0
All 3434 tests passing. Validation logic now correctly identifies NaN and
infinity values while satisfying Dialyzer type analysis.
Adds @spec annotations to all Oban worker perform/1 functions to improve
static analysis and documentation. Return types are specific to each worker's
actual behavior:
Workers returning 🆗
- BackupSummaryWorker
- BackupTimeoutWorker
- DeviceMonitorWorker
- DevicePollerWorker
- JobHealthCheckWorker
- NeighborCleanupWorker
Workers with complex returns:
- DiscoveryWorker: :ok | :discard
- FirmwareVersionFetcherWorker: :ok | {:error, term()}
- LoginHistoryCleanupWorker: {:ok, %{deleted: non_neg_integer(), anonymized_deleted: non_neg_integer()}}
- MikrotikBackupWorker: :ok | {:error, String.t()}
- SessionCleanupWorker: {:ok, %{sessions_deleted: non_neg_integer()}}
Analysis found that none of these workers have complex job parameters that
would benefit from embedded schemas - all use either no parameters or simple
device_id strings passed in job args.
StaleAgentWorker already had a typespec on find_stale_agents/0 from previous work.
All 98 worker tests passing after changes.
