Graham McIntire
aa1bac0d46
Replace individual Repo.insert() with Repo.insert_all() for sensor readings, interface stats, processor readings, and storage readings in agent channel and device poller worker. Add partial/composite database indexes for common query patterns. Dashboard now uses GROUP BY aggregation instead of loading all devices. DeviceLive.Show only reloads data for the active tab on PubSub events. DeviceLive.Index debounces status changes and skips quota queries on updates. Agent polling preloads filtered to monitored sensors/interfaces only.
583 lines
38 KiB
Text
583 lines
38 KiB
Text
CHANGELOG - towerops-web
|
|
========================
|
|
|
|
2026-02-12 - perf: performance improvements for scale (batch inserts, indexes, selective reloading)
|
|
- Files: lib/towerops/snmp.ex, lib/towerops/devices.ex, lib/towerops_web/channels/agent_channel.ex,
|
|
lib/towerops/workers/device_poller_worker.ex, lib/towerops_web/live/dashboard_live.ex,
|
|
lib/towerops_web/live/device_live/show.ex, lib/towerops_web/live/device_live/index.ex,
|
|
lib/towerops/agents.ex, priv/repo/migrations/20260212155105_add_performance_indexes.exs
|
|
|
|
Phase 1: Batch data ingestion - replaced individual Repo.insert() calls with Repo.insert_all()
|
|
for sensor readings, interface stats, processor readings, and storage readings in both
|
|
agent_channel.ex and device_poller_worker.ex. Added create_*_batch/1 functions to snmp.ex.
|
|
|
|
Phase 2: Database indexes - added partial indexes for monitored sensors/interfaces,
|
|
composite index for devices org+status, partial index for active alerts.
|
|
|
|
Phase 3: Dashboard aggregation - replaced loading all devices into memory with
|
|
Devices.get_device_status_counts/1 GROUP BY query.
|
|
|
|
Phase 4: DeviceLive.Show selective updates - split monolithic load_equipment_data into
|
|
tab-specific loaders (base, tab_nav, overview, ports, logs, backups). PubSub handlers
|
|
now only reload data relevant to the active tab.
|
|
|
|
Phase 5: DeviceLive.Index incremental updates - separated structural changes (create/delete)
|
|
from status changes. Added debouncing for rapid status changes. Skip reload when viewing
|
|
discovered tab.
|
|
|
|
Phase 6: Agent polling memory optimization - filtered preloads in list_agent_polling_targets
|
|
to only load monitored sensors and interfaces instead of all.
|
|
|
|
2026-02-12 - feat: complete WISP vendor coverage (ePMP, Raisecom, Netonix)
|
|
- Files: priv/profiles/os_discovery/epmp.yaml (enhanced), lib/towerops/snmp/profiles/vendors/raisecom.ex (new)
|
|
Completed LibreNMS parity for major WISP equipment vendors. Three improvements:
|
|
|
|
(1) Consolidated ePMP YAML profiles - merged cambium-epmp.yaml enhancements into epmp.yaml
|
|
(added GPS satellite count sensors, proper state_name fields, better field naming), deleted
|
|
duplicate file. ePMP now discovers 11 sensors via YAML + 4 wireless sensors via vendor module.
|
|
|
|
(2) Implemented Raisecom vendor module - fills major 0% coverage gap for Asia-Pacific WISP market.
|
|
Hardware detection via RAISECOM-SYSTEM-MIB with sysDescr fallback. Sensor discovery handled by
|
|
existing YAML (raisecom.yaml, raisecom-ros.yaml) covering 15+ sensors: CPU, memory pools, fan
|
|
speed/state, voltage with thresholds, power supply state, config load state. 8 comprehensive
|
|
tests all passing.
|
|
|
|
(3) Verified Netonix coverage - YAML profile matches LibreNMS 100% (PoE status, voltage,
|
|
temperature, fan, power, current). Vendor module provides additional scalar sensors beyond
|
|
LibreNMS baseline. Netonix coverage exceeds LibreNMS.
|
|
|
|
Tarana investigated: No SNMP support in LibreNMS (newer vendor, likely cloud-managed), would
|
|
require vendor MIBs and documentation to implement.
|
|
|
|
2026-02-12 - test: fix Cambium ePMP test Mox expectations
|
|
- File: test/towerops/snmp/profiles/vendors/cambium_test.exs
|
|
Changed expect() to stub() in all discover_wireless_sensors/1 tests to handle multiple SNMP
|
|
get calls. The implementation makes several calls: static sensor fetch (CPU), ePMP mode
|
|
detection (wirelessInterfaceMode OID), and individual sensor value fetches (RSSI, SNR,
|
|
Frequency, Clients). Using stub() allows unlimited calls vs expect() which enforces exactly
|
|
one call, fixing Mox.UnexpectedCallError in all 7 ePMP-related test cases. All 13 Cambium
|
|
tests now pass correctly.
|
|
|
|
2026-02-12 - fix: sync modal state to URL for browser back/forward navigation
|
|
- Files: lib/towerops_web/live/user_settings_live.ex, lib/towerops_web/live/agent_live/index.ex
|
|
Fixed browser back/forward button navigation by syncing modal state to URL parameters.
|
|
Implemented idiomatic Phoenix LiveView pattern using push_patch/2 and handle_params/3.
|
|
|
|
**UserSettingsLive (6 modals):**
|
|
- add_token, token_created, revoke_all, add_device, device_qr, recovery_codes
|
|
- Updated handle_params/3 to read ?modal= param and set all modal states
|
|
- Added build_settings_path/2 helper to preserve tab and page params
|
|
- Updated all show/close handlers to use push_patch instead of assign
|
|
|
|
**AgentLive.Index (1 modal):**
|
|
- setup modal (displays agent token after creation)
|
|
- Updated handle_params/3 to read ?modal=setup param
|
|
- Updated show_setup and close_token_modal handlers to use push_patch
|
|
|
|
**Result:** Browser back button now closes modals without navigating away from page.
|
|
URLs are bookmarkable and restore exact modal state. Tab and pagination state preserved.
|
|
|
|
**Documentation:** Updated CLAUDE.md with comprehensive browser navigation patterns.
|
|
Created design document: docs/plans/2026-02-12-browser-navigation-fix-design.md
|
|
|
|
2026-02-12 - feat: add comprehensive MikroTik wireless sensor discovery (11 sensor types across 4 MIB tables)
|
|
- Files: lib/towerops/snmp/profiles/vendors/mikrotik.ex (enhanced)
|
|
Implemented TIER 4 LARGE priority wireless sensor discovery for MikroTik RouterOS devices.
|
|
Achieves 100% LibreNMS parity for wireless monitoring (0% → 100% coverage).
|
|
Discovers 11 sensor types across 4 MIB tables:
|
|
|
|
**AP Sensors (mtxrWlApTable .1.3.6.1.4.1.14988.1.1.1.3):**
|
|
- Clients: Station count per AP interface
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.3.1.6 (mtxrWlApClientCount)
|
|
- CCQ: Connection quality percentage (skips when clients>0 && CCQ=0 for NV2)
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.3.1.10 (mtxrWlApOverallTxCCQ)
|
|
- Frequency: Operating frequency in MHz (skips if 0)
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.3.1.7 (mtxrWlApFreq)
|
|
- Noise Floor: Background noise level in dBm
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.3.1.9 (mtxrWlApNoiseFloor)
|
|
- TX Rate: Transmit data rate in bps
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.3.1.2 (mtxrWlApTxRate)
|
|
- RX Rate: Receive data rate in bps
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.3.1.3 (mtxrWlApRxRate)
|
|
|
|
**Station Sensors (mtxrWlStatTable .1.3.6.1.4.1.14988.1.1.1.1):**
|
|
- TX CCQ: Transmit connection quality percentage
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.1.1.9 (mtxrWlStatTxCCQ)
|
|
- RX CCQ: Receive connection quality percentage
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.1.1.10 (mtxrWlStatRxCCQ)
|
|
- Frequency: Operating frequency in MHz
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.1.1.7 (mtxrWlStatFreq)
|
|
- TX Rate: Transmit data rate in bps
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.1.1.2 (mtxrWlStatTxRate)
|
|
- RX Rate: Receive data rate in bps
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.1.1.3 (mtxrWlStatRxRate)
|
|
|
|
**60GHz Sensors (mtxrWl60GTable .1.3.6.1.4.1.14988.1.1.1.8):**
|
|
- Frequency: Operating frequency in MHz
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.8.1.6 (mtxrWl60GFreq)
|
|
- RSSI: Received signal strength in dBm
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.8.1.12 (mtxrWl60GRssi)
|
|
- Quality: Signal quality percentage
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.8.1.8 (mtxrWl60GSignal)
|
|
- PHY Rate: Physical layer rate in bps (divisor: 1,000,000)
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.8.1.13 (mtxrWl60GPhyRate)
|
|
|
|
**60GHz Station Sensors (mtxrWl60GStaTable .1.3.6.1.4.1.14988.1.1.1.9):**
|
|
- Distance: Link distance in km (divisor: 100,000)
|
|
OID: .1.3.6.1.4.1.14988.1.1.1.9.1.10 (mtxrWl60GStaDistance)
|
|
|
|
**LTE Modem Sensors (mtxrLTEModemTable .1.3.6.1.4.1.14988.1.1.16.1):**
|
|
- RSRQ: Reference Signal Received Quality in dB
|
|
OID: .1.3.6.1.4.1.14988.1.1.16.1.1.3 (mtxrLTEModemSignalRSRQ)
|
|
- RSRP: Reference Signal Received Power in dBm
|
|
OID: .1.3.6.1.4.1.14988.1.1.16.1.1.4 (mtxrLTEModemSignalRSRP)
|
|
- SINR: Signal-to-Interference-plus-Noise Ratio in dB
|
|
OID: .1.3.6.1.4.1.14988.1.1.16.1.1.7 (mtxrLTEModemSignalSINR)
|
|
|
|
- Implementation Details:
|
|
- discover_wireless_sensors/1: Calls 5 table-specific discovery functions
|
|
- discover_ap_sensors/1: Walks mtxrWlApTable, builds client/CCQ/freq/noise/rate sensors
|
|
- discover_stat_sensors/1: Walks mtxrWlStatTable, builds TX/RX CCQ and rate sensors
|
|
- discover_60g_sensors/1: Walks mtxrWl60GTable, builds 60GHz sensors
|
|
- discover_60g_sta_sensors/1: Walks mtxrWl60GStaTable, builds distance sensors
|
|
- discover_lte_sensors/1: Walks mtxrLTEModemTable, builds LTE signal sensors
|
|
- group_by_index/1: Groups SNMP walk results by interface index (last OID component)
|
|
- freq_to_label/2: Extracts frequency band label (2437 → "2G", 5180 → "5G")
|
|
- to_integer/1: Safe conversion handling both string and integer SNMP values
|
|
|
|
- Smart Skip Logic (matching LibreNMS):
|
|
- CCQ skipped when clients > 0 but CCQ = 0 (NV2 mode reports clients but no CCQ)
|
|
- Frequency skipped if value is 0
|
|
- TX/RX rate skipped if both are 0
|
|
- Empty CCQ sensors skipped (both TX and RX = 0)
|
|
|
|
- Files: test/towerops/snmp/profiles/vendors/mikrotik_test.exs (enhanced)
|
|
Added comprehensive test coverage with 13 test cases:
|
|
- Tests AP sensor discovery (7 sensors: clients, CCQ, freq, noise, TX/RX rate)
|
|
- Tests station sensor discovery (6 sensors: TX/RX CCQ, freq, TX/RX rate)
|
|
- Tests 60GHz sensor discovery (4 sensors: freq, RSSI, quality, PHY rate)
|
|
- Tests 60GHz station sensor discovery (1 sensor: distance)
|
|
- Tests LTE modem sensor discovery (3 sensors: RSRQ, RSRP, SINR)
|
|
- Tests skip logic for sensors with no data
|
|
- Tests graceful handling when SNMP tables are empty
|
|
All 13 tests passing with Mox-based SNMP adapter mocking.
|
|
|
|
- Impact: Complete wireless monitoring for MikroTik RouterOS devices including
|
|
standard Wi-Fi (2.4/5GHz), 60GHz point-to-point links, and LTE modems.
|
|
Enables detection of connection quality issues, frequency changes, link
|
|
distance problems, and cellular signal degradation.
|
|
|
|
- Source: LibreNMS LibreNMS/OS/Routeros.php (11 wireless discovery methods)
|
|
Matches exact OIDs, descriptions, divisors, and skip logic from LibreNMS.
|
|
|
|
- Priority: TIER 4 LARGE (12-16 hour effort, vendor module enhancement)
|
|
- Gap Status: MikroTik wireless coverage 0% → 100% (11 sensor types)
|
|
- Result: TIER 4 COMPLETE - MikroTik wireless monitoring at 100% LibreNMS parity
|
|
|
|
2026-02-12 - feat: add UniFi channel-to-frequency dynamic conversion for accurate frequency monitoring
|
|
- Files: lib/towerops/snmp/profiles/vendors/unifi.ex (enhanced)
|
|
Added TIER 3 MEDIUM priority dynamic frequency sensor discovery for Ubiquiti UniFi APs.
|
|
Implements channel-to-frequency conversion matching LibreNMS behavior exactly.
|
|
- Channel OID: .1.3.6.1.4.1.41112.1.6.1.2.1.4 (unifiVapChannel)
|
|
- Radio Name OID: .1.3.6.1.4.1.41112.1.6.1.2.1.1 (unifiVapRadio)
|
|
- Conversion: 2.4GHz channels 1-14 → 2412-2484 MHz
|
|
- Conversion: 5GHz channels 34-165 → 5170-5825 MHz
|
|
- Unknown channels (not in lookup table) are filtered out
|
|
- Implementation:
|
|
- discover_frequency_sensors/1: Walks channel OID, converts to frequency dynamically
|
|
- channel_to_frequency/1: Lookup table with 29 Wi-Fi channel mappings
|
|
- Frequency sensors added alongside static sensors (clients, CCQ, power, utilization)
|
|
- Per-radio frequency sensors with descriptive names (e.g., "Frequency (ng)")
|
|
- Files: test/towerops/snmp/profiles/vendors/unifi_test.exs (enhanced)
|
|
Added comprehensive test coverage for frequency conversion:
|
|
- Tests 2.4GHz channel conversion (channels 1, 6, 11)
|
|
- Tests 5GHz channel conversion (channels 36, 149, 165)
|
|
- Tests unknown channel filtering (channel 999 → filtered out)
|
|
- Tests integration with static sensors (12 total sensors: 10 static + 2 frequency)
|
|
- Tests graceful handling of channel walk failures
|
|
All 16 tests passing with Mox-based SNMP adapter mocking.
|
|
- Impact: Dynamic frequency monitoring enables accurate channel tracking when APs
|
|
change channels (auto-DFS, manual changes). Static OID approach was inaccurate.
|
|
- Source: LibreNMS LibreNMS/OS/Unifi.php discoverWirelessFrequency/pollWirelessFrequency
|
|
LibreNMS LibreNMS/Modules/Wireless.php channelToFrequency lookup table
|
|
- Priority: TIER 3 MEDIUM (3-5 hour effort, vendor module enhancement)
|
|
- Gap Status: UniFi parity 80% → 90%
|
|
- Result: TIER 3 COMPLETE - All Ubiquiti platform gaps resolved
|
|
|
|
2026-02-12 - feat: add AirFiber AF-LTU optical power sensors (per-chain RX power, TX EIRP)
|
|
- Files: priv/profiles/os_discovery/airos-af-ltu.yaml (enhanced)
|
|
Added TIER 3 MEDIUM priority optical power sensors for Ubiquiti AirFiber AF-LTU:
|
|
- RX Power Chain 0: Received optical power chain 0 in dBm
|
|
OID: .1.3.6.1.4.1.41112.1.10.1.4.1.5
|
|
- RX Power Chain 1: Received optical power chain 1 in dBm
|
|
OID: .1.3.6.1.4.1.41112.1.10.1.4.1.6
|
|
- RX Ideal Power Chain 0: Ideal received power chain 0 in dBm
|
|
OID: .1.3.6.1.4.1.41112.1.10.1.4.1.7
|
|
- RX Ideal Power Chain 1: Ideal received power chain 1 in dBm
|
|
OID: .1.3.6.1.4.1.41112.1.10.1.4.1.8
|
|
- TX EIRP: Transmit effective isotropic radiated power in dBm
|
|
OID: .1.3.6.1.4.1.41112.1.10.1.2.6
|
|
- Impact: Per-chain optical power monitoring enables detection of imbalanced
|
|
signal levels, fiber issues, and link budget problems. Ideal power comparison
|
|
helps identify suboptimal alignment or hardware degradation.
|
|
- Source: LibreNMS LibreNMS/OS/AirosAfLtu.php
|
|
- Priority: TIER 3 MEDIUM (enhanced troubleshooting and diagnostics)
|
|
- Gap Status: AF-LTU parity 80% → 90%
|
|
- Remaining: UniFi channel-to-frequency conversion (requires vendor module work)
|
|
|
|
2026-02-12 - feat: add AirFiber wireless metrics (frequency, distance, capacity, RSSI, SNR)
|
|
- Files: priv/profiles/os_discovery/airos-af60.yaml (enhanced)
|
|
Added TIER 2 HIGH priority wireless metrics for Ubiquiti AirFiber AF60:
|
|
- Frequency: Operating frequency in MHz
|
|
OID: .1.3.6.1.4.1.41112.1.11.1.1.2
|
|
- Distance: Link distance in km (divisor 1000)
|
|
OID: .1.3.6.1.4.1.41112.1.11.1.3.1.15
|
|
- TX Capacity: Transmit capacity in Mbps (divisor 1000)
|
|
OID: .1.3.6.1.4.1.41112.1.11.1.3.1.7
|
|
- RX Capacity: Receive capacity in Mbps (divisor 1000)
|
|
OID: .1.3.6.1.4.1.41112.1.11.1.3.1.8
|
|
- Local RSSI: Local received signal strength in dBm
|
|
OID: .1.3.6.1.4.1.41112.1.11.1.3.1.3
|
|
- Remote RSSI: Remote received signal strength in dBm
|
|
OID: .1.3.6.1.4.1.41112.1.11.1.3.1.18
|
|
- Local SNR: Local signal-to-noise ratio in dB
|
|
OID: .1.3.6.1.4.1.41112.1.11.1.3.1.4
|
|
- Remote SNR: Remote signal-to-noise ratio in dB
|
|
OID: .1.3.6.1.4.1.41112.1.11.1.3.1.19
|
|
- Files: priv/profiles/os_discovery/airos-af-ltu.yaml (enhanced)
|
|
Added TIER 2 HIGH priority wireless metrics for Ubiquiti AirFiber AF-LTU:
|
|
- Frequency: Operating frequency in MHz
|
|
OID: .1.3.6.1.4.1.41112.1.10.1.2.2
|
|
- Distance: Link distance in km (divisor 1000)
|
|
OID: .1.3.6.1.4.1.41112.1.10.1.4.1.23
|
|
- TX Capacity: Transmit capacity in Mbps (divisor 1000)
|
|
OID: .1.3.6.1.4.1.41112.1.10.1.4.1.3
|
|
- RX Capacity: Receive capacity in Mbps (divisor 1000)
|
|
OID: .1.3.6.1.4.1.41112.1.10.1.4.1.4
|
|
- Impact: Essential wireless metrics for capacity planning and troubleshooting.
|
|
Distance enables link budget analysis. RSSI/SNR enable interference detection.
|
|
Capacity metrics enable throughput monitoring and bandwidth planning.
|
|
- Source: LibreNMS LibreNMS/OS/AirosAf60.php, LibreNMS/OS/AirosAfLtu.php
|
|
- Priority: TIER 2 HIGH (enhanced monitoring for capacity planning)
|
|
- Gap Status: AF60 parity 50% → 80%, AF-LTU parity 60% → 80%
|
|
- Next: TIER 3 - UniFi channel-to-frequency conversion, AF-LTU optical power
|
|
|
|
2026-02-12 - feat: add critical AirFiber MCS/modulation rate state sensors
|
|
- Files: priv/profiles/os_discovery/airos-af60.yaml (enhanced)
|
|
Added CRITICAL link quality state sensors for Ubiquiti AirFiber AF60:
|
|
- TX MCS Rate: 9 states (1X-9X) showing transmit modulation level
|
|
OID: .1.3.6.1.4.1.41112.1.11.1.3.1.5 (states: 1-2 warning, 3-4 degraded, 5-9 ok)
|
|
- RX MCS Rate: 9 states (1X-9X) showing receive modulation level
|
|
OID: .1.3.6.1.4.1.41112.1.11.1.3.1.6 (states: 1-2 warning, 3-4 degraded, 5-9 ok)
|
|
- Active Link: 2 states (Main/Backup) showing active link selection
|
|
OID: .1.3.6.1.4.1.41112.1.11.1.3.1.2 (states: 1=Main ok, 2=Backup degraded)
|
|
- Files: priv/profiles/os_discovery/airos-af-ltu.yaml (enhanced)
|
|
Added CRITICAL link quality state sensors for Ubiquiti AirFiber AF-LTU:
|
|
- TX Modulation Rate: 7 QAM states (1X QPSK+SFBC through 12X 4096QAM)
|
|
OID: .1.3.6.1.4.1.41112.1.10.1.4.1.1 (states: 1-2 warning, 4-6 degraded, 8-12 ok)
|
|
- RX Modulation Rate: 7 QAM states (1X QPSK+SFBC through 12X 4096QAM)
|
|
OID: .1.3.6.1.4.1.41112.1.10.1.4.1.2 (states: 1-2 warning, 4-6 degraded, 8-12 ok)
|
|
- Impact: MCS/modulation rate is THE key indicator of point-to-point link health.
|
|
Without these sensors, operators cannot diagnose why throughput changed or when
|
|
links drop from high QAM to low QAM due to interference or alignment issues.
|
|
- Pattern: State sensors with discrete value mappings match LibreNMS exactly.
|
|
Uses generic values for alerting: 0=ok, 1=degraded/warning, 2=critical.
|
|
- Source: LibreNMS includes/discovery/sensors/state/airos-af*.inc.php
|
|
- Priority: TIER 1 CRITICAL (production monitoring gaps for PtP wireless)
|
|
- Gap Status: AF60 parity 35% → 50%, AF-LTU parity 40% → 60%
|
|
- Next: TIER 2 - Add RSSI, SNR, frequency, distance, capacity sensors
|
|
|
|
2026-02-12 - feat: add Dell PowerVault storage array text-based sensor parsing
|
|
- Files: lib/towerops/snmp/profiles/vendors/powervault.ex (created)
|
|
Created vendor post-processing module for Dell PowerVault storage arrays.
|
|
PowerVault devices report sensors as text messages in FCMGMT-MIB::connUnitSensorMessage
|
|
instead of structured SNMP tables, requiring regex-based parsing.
|
|
- Message format: "Sensor Name: Value Unit" (e.g., "Enclosure Temp: 25 C 77.0F")
|
|
- Temperature: Parses Celsius from "N C M.MF" format (e.g., "25 C 77.0F" → 25°C)
|
|
- Voltage: Parses "N.NV" format (e.g., "12.1V" → 12.1V)
|
|
- Current: Parses "N.NA" format (e.g., "0.5A" → 0.5A)
|
|
- Charge: Parses "N%" format (e.g., "95%" → 95% with battery thresholds)
|
|
Sensor index format: powervault_{type}.{oid_index} for unique identification.
|
|
- Files: lib/towerops/snmp/profiles/dynamic.ex (enhanced)
|
|
Integrated PowerVault vendor module into post-processing pipeline.
|
|
Added "dell-powervault" case to apply_vendor_post_processing/3 function.
|
|
- Files: test/towerops/snmp/profiles/vendors/powervault_test.exs (created)
|
|
Comprehensive test suite with 21 tests covering:
|
|
- All sensor types (temperature, voltage, current, charge/percent)
|
|
- Multi-sensor parsing (mixed message types in single walk)
|
|
- Error handling (invalid formats, non-string values, missing colons)
|
|
- Edge cases (empty results, SNMP timeouts, existing sensor preservation)
|
|
All tests passing with Mox-based SNMP adapter mocking.
|
|
- Pattern: Follows Arista vendor module architecture with post_process_sensors/2
|
|
callback. Client.walk returns map format, converted to list for message parsing.
|
|
- Result: Dell PowerVault storage arrays now support temperature/voltage/current/
|
|
battery monitoring via text message parsing. Gap: CRITICAL → RESOLVED.
|
|
Parity: 0% → 85% (limited by FCMGMT-MIB available sensors).
|
|
|
|
2026-02-11 - feat: add Dell UPS power sensor and HP BladeSystem/Moonshot complete monitoring
|
|
- Files: priv/profiles/os_discovery/dell-ups.yaml (enhanced)
|
|
Added Dell UPS power consumption sensor via DELL-SNMP-UPS-MIB:
|
|
- Power: physicalOutputPresentConsumption (OID .1.3.6.1.4.1.674.10902.2.120.2.6.0)
|
|
Enables capacity planning and load tracking for Dell UPS units.
|
|
Gap: MEDIUM (missing power sensor) → RESOLVED. Parity: 80% → 100%.
|
|
- Files: priv/profiles/os_discovery/hpblmos.yaml (enhanced)
|
|
Created comprehensive HP BladeSystem/Moonshot sensor profile:
|
|
- Temperature: Blade chassis component temperatures (OID .1.3.6.1.4.1.232.22.2.3.1.2.1.6)
|
|
- Power: PSU output wattage with max limit (OID .1.3.6.1.4.1.232.22.2.5.1.1.1.10)
|
|
- State: Fan operational status (OID .1.3.6.1.4.1.232.22.2.3.1.3.1.11)
|
|
- State: PSU condition status (OID .1.3.6.1.4.1.232.22.2.5.1.1.1.17)
|
|
Skip logic: Only monitors present fans/PSUs (presence check to avoid non-existent sensors)
|
|
Gap: CRITICAL (no sensor profile) → RESOLVED. Parity: 0% → 80%.
|
|
- Analysis Discovery: Dell Enterprise Servers (iDRAC) already have excellent coverage
|
|
The existing drac.yaml profile includes comprehensive sensor monitoring:
|
|
- Temperature: Multiple probes with thresholds (4 types)
|
|
- Voltage: PSU voltage monitoring (2 types)
|
|
- Current: Amperage probes (3 types, filtered for wattage type)
|
|
- Power: Chassis power (current, peak, potential, idle, max - 5 types)
|
|
- Fanspeed: Cooling device monitoring with thresholds
|
|
- State: Extensive state sensors (global system, IDSDM card, intrusion, IOM, etc.)
|
|
Previous gap analysis incorrectly reported 0% parity - actual parity is 95%+.
|
|
- Result: Tier 4 complete. Dell UPS now has full monitoring capability, HP BladeSystem
|
|
blade chassis fully supported with temperature/power/fan/PSU monitoring. Dell servers
|
|
confirmed to have excellent existing coverage via iDRAC/OpenManage MIBs.
|
|
|
|
2026-02-11 - feat: add Force10 FTOS and optical transceiver monitoring (ProCurve, Comware)
|
|
- Files: priv/profiles/os_discovery/ftos.yaml (enhanced)
|
|
Added Dell Force10 FTOS temperature monitoring for all 3 series:
|
|
- S-Series: chStackUnitTemp (OID .1.3.6.1.4.1.6027.3.10.1.2.2.1.14)
|
|
- C-Series: chSysCardTemp (OID .1.3.6.1.4.1.6027.3.8.1.2.1.1.5)
|
|
- E-Series: chSysCardUpperTemp + chSysCardLowerTemp (OIDs .1.3.6.1.4.1.6027.3.1.1.2.3.1.8-9)
|
|
Completes Tier 1 critical network switch monitoring. Parity: 0% → 90%.
|
|
- Files: priv/profiles/os_discovery/procurve.yaml (enhanced)
|
|
Added HP ProCurve transceiver optical monitoring via HP-ICF-TRANSCEIVER-MIB::hpicfXcvrInfoTable:
|
|
- Temperature: hpicfXcvrTemp (OID .1.3.6.1.4.1.11.2.14.11.5.1.82.1.1.1.1.11)
|
|
- Bias Current: hpicfXcvrBias (OID .1.3.6.1.4.1.11.2.14.11.5.1.82.1.1.1.1.13)
|
|
- Supply Voltage: hpicfXcvrVoltage (OID .1.3.6.1.4.1.11.2.14.11.5.1.82.1.1.1.1.12)
|
|
- RX Power (dBm): hpicfXcvrRxPower (OID .1.3.6.1.4.1.11.2.14.11.5.1.82.1.1.1.1.14)
|
|
- TX Power (dBm): hpicfXcvrTxPower (OID .1.3.6.1.4.1.11.2.14.11.5.1.82.1.1.1.1.15)
|
|
Enables optical SFP/SFP+ monitoring for fiber uplinks. Parity: 60% → 100%.
|
|
- Files: priv/profiles/os_discovery/comware.yaml (enhanced)
|
|
Added HP Comware transceiver optical monitoring via HH3C-TRANSCEIVER-INFO-MIB::hh3cTransceiverInfoTable:
|
|
- Temperature: hh3cTransceiverTemperature (OID .1.3.6.1.4.1.25506.2.70.1.1.1.15)
|
|
- Bias Current: hh3cTransceiverBiasCurrent (OID .1.3.6.1.4.1.25506.2.70.1.1.1.17)
|
|
- Supply Voltage: hh3cTransceiverVoltage (OID .1.3.6.1.4.1.25506.2.70.1.1.1.16)
|
|
- RX Power (dBm): hh3cTransceiverCurRXPower (OID .1.3.6.1.4.1.25506.2.70.1.1.1.9)
|
|
- TX Power (dBm): hh3cTransceiverCurTXPower (OID .1.3.6.1.4.1.25506.2.70.1.1.1.12)
|
|
Completes Comware sensor coverage. Parity: 60% → 95%.
|
|
- Result: Tier 1 + Tier 2 complete. All critical network switches have temperature monitoring,
|
|
and both ProCurve/Comware have complete optical transceiver diagnostics (temp, current, voltage,
|
|
RX/TX power). Force10 FTOS data center switches fully supported. Fiber link monitoring enabled.
|
|
|
|
2026-02-11 - feat: add critical network switch sensor support (HP Comware, Dell PowerConnect, Dell SONiC)
|
|
- Files: priv/profiles/os_discovery/comware.yaml (enhanced)
|
|
Added HP Comware chassis temperature monitoring via HH3C-ENTITY-EXT-MIB::hh3cEntityExtTemperature
|
|
(OID 1.3.6.1.4.1.25506.2.6.1.1.1.1.12). This restores fundamental temperature monitoring capability
|
|
for HP Comware switches, enabling overheating alerts. Gap identified as CRITICAL in Phase 3 analysis.
|
|
- Files: priv/profiles/os_discovery/powerconnect.yaml (enhanced)
|
|
Added Dell PowerConnect/DNOS CPU temperature monitoring via FASTPATH-BOXSERVICES-PRIVATE-MIB
|
|
(OID 1.3.6.1.4.1.674.10895.5000.2.6132.1.1.43.1.8.1.5). Enables temperature monitoring for
|
|
common enterprise access switches.
|
|
- Files: priv/profiles/os_discovery/dell-sonic.yaml (new)
|
|
Created comprehensive sensor profile for Dell SONiC switches using NETGEAR-BOXSERVICES-PRIVATE-MIB
|
|
(Quanta-based platform). Sensors: temperature (boxServicesTempSensorState), fan speed
|
|
(boxServicesFanSpeed), PSU state (boxServicesPowSupplyItemState with 8 states). Enables hardware
|
|
monitoring for modern data center switches.
|
|
- Result: Top 3 critical network switch monitoring gaps resolved. HP Comware gains fundamental
|
|
temperature capability (40% → 60% parity), Dell PowerConnect gains first sensor support
|
|
(0% → 80% parity), Dell SONiC gains complete hardware monitoring (0% → 95% parity).
|
|
|
|
2026-02-11 - audit: Phase 3 - HP/Aruba and Dell sensor analysis complete
|
|
- Files: docs/librenms-audit/sensors/hp-aruba-comparison.md (new)
|
|
Analyzed 6 HP/Aruba platforms: ProCurve (60%), Comware (40%), hpblmos (0%), ArubaOS (100%),
|
|
ArubaOS-CX (100%), Instant On (100%). Identified critical gaps in Comware temperature and
|
|
transceiver support.
|
|
- Files: docs/librenms-audit/sensors/dell-comparison.md (new)
|
|
Analyzed 11 Dell platforms: OS10 (100%), rPDU (100%), UPS (80%), OME-M (100%), Compellent (100%),
|
|
PowerVault (20%), PowerConnect (0%), Force10 (0%), SONiC (0%), Quanta (40%), Servers (0%).
|
|
Found network switches severely underserved compared to storage/infrastructure.
|
|
- Files: docs/librenms-audit/PHASE3-VENDOR-EXPANSION.md (new)
|
|
Executive summary with implementation priorities. Tier 1 critical: Network switch sensor gaps
|
|
(8-12 hours). Tier 2 high: Optical transceiver monitoring (4-6 hours). Tier 3: Storage/compute
|
|
(7-10 hours). Total effort: 26-38 hours for full parity.
|
|
- Key Finding: Modern platforms excellent (Aruba CX, Dell OS10), but legacy network switches
|
|
missing fundamental monitoring. HP Comware temperature gap BLOCKING (breaks alerts).
|
|
Dell PowerConnect/FTOS/SONiC missing all sensors (CRITICAL for enterprise deployments).
|
|
|
|
2026-02-11 - feat: implement Arista EOS sensor enhancements (DOM power, thresholds, grouping)
|
|
- Files: lib/towerops/snmp/profiles/vendors/arista.ex (enhanced)
|
|
Implemented all critical and nice-to-have Arista sensor improvements identified in audit:
|
|
1. DOM power conversion (watts→dBm): Detects optical power sensors via regex, converts
|
|
using formula dBm = 10*log10(watts*1000), preserves original value in metadata
|
|
2. Arista threshold discovery: Walks ARISTA-ENTITY-SENSOR-MIB::aristaEntSensorThresholdTable,
|
|
applies 4 threshold types (low_critical, low_warn, high_warn, high_critical), converts
|
|
thresholds to dBm for optical sensors
|
|
3. Smart grouping: Organizes sensors by SFPs, PSUs, Platform (chipsets), Power Connectors, System
|
|
4. Description cleanup: Removes redundant "sensor" text, simplifies PSU naming, cleans whitespace
|
|
- Files: lib/towerops/snmp/profiles/dynamic.ex (integration)
|
|
Added apply_vendor_post_processing/3 to call Arista enhancements after sensor discovery.
|
|
Applies to both arista_eos and arista-mos profiles.
|
|
- Files: test/towerops/snmp/profiles/vendors/arista_test.exs (comprehensive tests)
|
|
Added 23 new tests covering DOM conversion (6 tests), threshold discovery (4 tests),
|
|
smart grouping (6 tests), description cleanup (5 tests), and end-to-end post-processing (1 test).
|
|
All tests passing (30/30 in arista_test.exs, 6367/6367 total).
|
|
- Result: Arista EOS support upgraded from 85% to 100% LibreNMS parity. All critical gaps closed:
|
|
optical power now displayed correctly in dBm, alerting enabled via thresholds, sensors
|
|
organized for better UX, descriptions cleaned up.
|
|
|
|
2026-02-11 - audit: Phase 2 - complete top vendor sensor analysis (Cisco, Juniper, Arista)
|
|
- Files: docs/librenms-audit/sensors/*.md, PHASE2-VENDOR-ANALYSIS.md (new)
|
|
Completed comprehensive sensor discovery analysis for top 3 enterprise vendors:
|
|
- cisco-ios-comparison.md: Cisco IOS/XE/XR/NX-OS analysis (95% parity, production ready)
|
|
- juniper-junos-comparison.md: JunOS analysis (100% IDENTICAL - perfect parity)
|
|
- arista-eos-comparison.md: Arista EOS analysis (85% parity, gaps identified)
|
|
- PHASE2-VENDOR-ANALYSIS.md: Comprehensive summary and recommendations
|
|
- Key findings:
|
|
- Cisco: All critical sensors covered, YAML-based discovery, minor cellular gaps (rare devices)
|
|
- Juniper: junos.yaml byte-for-byte identical (30+ sensor types, 38+ OIDs, 10 MIBs)
|
|
- Arista: Missing DOM power conversion (watts→dBm) and ARISTA-ENTITY-SENSOR-MIB thresholds
|
|
- Identified actionable fixes for Arista (2 critical, 2 minor):
|
|
1. DOM Rx/Tx power conversion (2-3 hours, enables proper optical monitoring)
|
|
2. Arista threshold discovery (4-6 hours, enables alerting)
|
|
3. Smart grouping (2-3 hours, UX improvement)
|
|
4. Description cleanup (1-2 hours, cosmetic)
|
|
- Towerops advantages: CPU/Memory/Uptime sensors (Arista), PSU/Fan state (Arista),
|
|
cleaner YAML architecture, index templates, pre-resolved MIB cache
|
|
|
|
2026-02-11 - audit: complete LibreNMS parity audit and add missing OS detection profiles
|
|
- Files: docs/librenms-audit/*.md (new)
|
|
Completed comprehensive audit comparing Towerops device detection and sensor discovery
|
|
against LibreNMS. Created detailed documentation:
|
|
- detection-algorithm.md: LibreNMS 2-pass detection system analysis
|
|
- towerops-detection.md: Towerops 4-phase detection system analysis
|
|
- vendor-detection-comparison.csv: Priority vendor comparison (MikroTik, Ubiquiti)
|
|
- librenms-sensors.md: LibreNMS sensor discovery architecture (360+ modules, 24 types)
|
|
- EXECUTIVE-SUMMARY.md: Overall findings (99.5% detection parity, 95%+ sensor parity)
|
|
- IMPLEMENTATION-STATUS.md: Phase 1 completion status
|
|
- Files: priv/profiles/os_detection/conteg-pdu.yaml, microsens-g6.yaml (new)
|
|
Added 2 missing OS detection profiles from LibreNMS to achieve 100% profile parity
|
|
(786 profiles total). Conteg PDU for power monitoring devices, MICROSENS G6 for
|
|
network switches.
|
|
- Key findings: Towerops has full detection parity with LibreNMS plus architectural
|
|
enhancements (4-phase matching, Rust NIF MIB resolution, ETS caching, index templates).
|
|
MikroTik coverage verified: all 22 sensor tables discovered identically. Ubiquiti
|
|
coverage: 8 of 9 profiles identical (airos-af has additional fallback in Towerops).
|
|
|
|
2026-02-11 - test: add 100% coverage for ExpiredBanCleanupWorker
|
|
- File: test/towerops/workers/expired_ban_cleanup_worker_test.exs (new)
|
|
Added 7 tests covering perform/1: expired ban deletion, active ban preservation,
|
|
permanent ban preservation, mixed scenarios, empty database, PubSub broadcast
|
|
on deletion, and no broadcast when nothing deleted.
|
|
|
|
2026-02-11 - fix: handle "null" string values in SNMP parsing
|
|
- File: lib/towerops/snmp/profiles/base.ex (parse_integer/1, parse_float/1)
|
|
Added explicit handling for "null" string values before attempting integer/float
|
|
conversion. Some devices (e.g., AirOS 8.7.14) return the literal string "null"
|
|
for unavailable metrics like UCD CPU stats, which caused ArgumentError when
|
|
parse_integer/parse_float tried to convert them. Now returns nil, which gets
|
|
coerced to 0 by the || 0 fallback in the calling code.
|
|
|
|
2026-02-11 - fix: deduplicate sensors with leading-dot OID differences
|
|
- File: lib/towerops/snmp/profiles/dynamic.ex (deduplicate_by_oid/1)
|
|
Normalized OIDs by stripping leading dots when grouping for deduplication.
|
|
gosnmp (agent) returns OIDs with leading dots (.1.3.6.1...) but vendor
|
|
modules and YAML profiles define them without. Without normalization,
|
|
the same sensor appeared twice (e.g. "DHCP Leases" and "DHCP Lease Count").
|
|
- File: lib/towerops/snmp/discovery.ex (deduplicate_sensors_by_oid/1)
|
|
Same leading-dot normalization applied to the discovery-level deduplication.
|
|
- File: lib/towerops/snmp/profiles/vendors/routeros.ex (wireless_oid_defs/0)
|
|
Removed DHCP Leases entry (mtxrDHCPLeaseCount) since the routeros.yaml
|
|
YAML profile already defines it in count_sensor_oids. Eliminates the
|
|
source of duplication for MikroTik devices.
|
|
- Tests updated in dynamic_test.exs and routeros_test.exs
|
|
|
|
2026-02-11 - fix: agent discovery and polling improvements
|
|
- File: lib/towerops_web/channels/agent_channel.ex (build_polling_queries/1, process_interface_stats/3)
|
|
Changed interface traffic OIDs from 32-bit counters (ifInOctets/ifOutOctets) to 64-bit
|
|
HC counters (ifHCInOctets/ifHCOutOctets). 32-bit counters wrap at ~4.3GB which causes
|
|
wildly inaccurate readings on gigabit+ links.
|
|
- File: lib/towerops/snmp/adapters/replay.ex (parse_value/1)
|
|
Fixed negative integer parsing regex from ^\d+$ to ^-?\d+$ so values like
|
|
entPhySensorScale=-3 are parsed as integers. Previously fell through to string,
|
|
causing sensor divisor calculation to silently return 1 instead of 1000.
|
|
- File: lib/towerops_web/channels/agent_channel.ex (build_discovery_queries/0)
|
|
Replaced walking entire Cisco enterprise tree (1.3.6.1.4.1.9) with targeted subtrees
|
|
for CPU, ENVMON, CDP, and WAP to avoid 50k+ OID responses and timeouts. Added Juniper,
|
|
HP/H3C, Fortinet, and Cambium vendor OIDs for broader device support.
|
|
- File: lib/towerops_web/channels/agent_channel.ex (build_discovery_queries/0)
|
|
Added missing discovery OIDs: ENTITY-STATE-MIB for state sensors, HOST-RESOURCES-MIB
|
|
tables for processors/storage/devices, and UCD-SNMP-MIB for Linux CPU stats fallback.
|
|
- Tests added in replay_test.exs and agent_channel_test.exs
|
|
|
|
2026-02-11 - fix: handle RouterOS version returned as list of integers
|
|
- File: lib/towerops/snmp/profiles/vendors/routeros.ex (detect_version/1)
|
|
Added pattern match to handle case where SNMP version OID returns list
|
|
of integers [7, 20, 6] instead of binary string "7.20.6". Converts list
|
|
to dotted string format using Enum.join/2.
|
|
- File: test/towerops/snmp/profiles/vendors/routeros_test.exs
|
|
Added two test cases for list-formatted versions with and without license level.
|
|
- Bug: Warning logged "Failed to detect version: {:ok, [7, 20, 6]}" even though
|
|
version was successfully retrieved, just in wrong format.
|
|
|
|
2026-02-11 - fix: normalize OIDs so agent-polled sensors match regardless of leading dot
|
|
- File: lib/towerops_web/channels/agent_channel.ex
|
|
Agent (gosnmp) returns OIDs with leading dots (e.g. ".1.3.6.1.4.1.41112...")
|
|
but sensor OIDs stored in DB may omit the dot. process_sensor_reading used
|
|
direct Map.get which silently missed wireless/vendor sensors on Ubiquiti airOS.
|
|
- Fix: Added normalize_oid_keys/1 to strip leading dots from all oid_values keys,
|
|
and normalize sensor.sensor_oid too. Both sides now match without leading dot.
|
|
- File: test/towerops/snmp/sensor_change_detector_test.exs
|
|
Fixed flaky refute_receive tests that subscribed to global "device:events" topic
|
|
and caught unrelated device_discovered events from concurrent async tests.
|
|
Changed to device-specific topic "device:#{device.id}" instead.
|
|
|
|
2026-02-11 - feat: add sensor value change events for all sensor types
|
|
- New file: lib/towerops/snmp/sensor_change_detector.ex
|
|
Extracted shared sensor change detection logic from DevicePollerWorker.
|
|
Detects threshold violations, spike/drop for % sensors, and new
|
|
sensor_value_changed events for all non-% sensors when value changes.
|
|
- File: lib/towerops/devices/event.ex
|
|
Added "sensor_value_changed" to allowed event types.
|
|
- File: lib/towerops/workers/device_poller_worker.ex
|
|
Replaced inline detect_sensor_changes with SensorChangeDetector module.
|
|
Removed ~240 lines of private functions now in shared module.
|
|
- File: lib/towerops_web/channels/agent_channel.ex
|
|
Added change detection and last_value/last_checked_at update to
|
|
process_sensor_reading/3 so agent-polled devices get same events
|
|
as Phoenix-polled devices.
|
|
- New file: test/towerops/snmp/sensor_change_detector_test.exs
|
|
11 tests covering value changes, spike/drop, thresholds, nil handling.
|
|
|
|
2026-02-11 - fix: live polling uses effective agent from site/org cascade
|
|
- File: lib/towerops_web/live/graph_live/show.ex
|
|
- Bug: Live poll mode used Agents.get_device_assignment/1 which only checks
|
|
direct device-level assignments. Devices with agent assigned at site or
|
|
org level fell through to direct Phoenix SNMP instead of routing to agent.
|
|
- Fix: Replaced with Agents.get_effective_agent_token/1 which walks the
|
|
device → site → org cascade, matching how regular polling resolves agents.
|
|
Also simplified request_agent_live_poll_and_wait to accept token_id directly.
|
|
|
|
2026-02-11 - fix: round sensor values to 1 decimal place
|
|
- Files: lib/towerops/workers/device_poller_worker.ex (poll_simple_sensor/2),
|
|
lib/towerops_web/channels/agent_channel.ex (process_sensor_reading/3)
|
|
- Bug: Division by sensor_divisor produced full float precision (e.g. 34.09999999)
|
|
- Fix: Wrap division result with Float.round(..., 1) in both Phoenix and agent paths
|
|
|
|
2026-02-11 - fix: update device status from agent monitoring checks
|
|
- File: lib/towerops_web/channels/agent_channel.ex (store_monitoring_check/2)
|
|
- Bug: Devices polled by remote agents stayed in "unknown" state because
|
|
store_monitoring_check saved the ping result but never called
|
|
Devices.update_device_status/2. The DeviceMonitorWorker (Phoenix-side)
|
|
skips agent-assigned devices entirely, so nothing ever updated status.
|
|
- Fix: Added update_device_status_from_check/2 and handle_agent_status_change/3
|
|
to derive :up/:down from check status, update device, create alerts on
|
|
transitions, and broadcast via PubSub — matching DeviceMonitorWorker behavior
|
|
- Also added alias Towerops.Alerts to the channel module
|
|
|
|
2026-02-11 - fix: update last_snmp_poll_at for agent-polled devices
|
|
- File: lib/towerops_web/channels/agent_channel.ex (process_polling_result/3)
|
|
- Bug: Devices polled by remote agents showed "Last Polled: Never" because
|
|
process_polling_result never called Devices.update_snmp_poll_time/1
|
|
- The Phoenix-side DevicePollerWorker already had this call (line 172),
|
|
but the agent channel path was missing it
|
|
- Fix: Added Devices.update_snmp_poll_time(device) at the top of
|
|
process_polling_result/3, matching the DevicePollerWorker pattern
|