towerops/CHANGELOG.txt

CHANGELOG - towerops-web
========================

2026-02-12 - feat: add Dell PowerVault storage array text-based sensor parsing
  - Files: lib/towerops/snmp/profiles/vendors/powervault.ex (created)
    Created vendor post-processing module for Dell PowerVault storage arrays.
    PowerVault devices report sensors as text messages in FCMGMT-MIB::connUnitSensorMessage
    instead of structured SNMP tables, requiring regex-based parsing.
    - Message format: "Sensor Name: Value Unit" (e.g., "Enclosure Temp: 25 C 77.0F")
    - Temperature: Parses Celsius from "N C M.MF" format (e.g., "25 C 77.0F" → 25°C)
    - Voltage: Parses "N.NV" format (e.g., "12.1V" → 12.1V)
    - Current: Parses "N.NA" format (e.g., "0.5A" → 0.5A)
    - Charge: Parses "N%" format (e.g., "95%" → 95% with battery thresholds)
    Sensor index format: powervault_{type}.{oid_index} for unique identification.
  - Files: lib/towerops/snmp/profiles/dynamic.ex (enhanced)
    Integrated PowerVault vendor module into post-processing pipeline.
    Added "dell-powervault" case to apply_vendor_post_processing/3 function.
  - Files: test/towerops/snmp/profiles/vendors/powervault_test.exs (created)
    Comprehensive test suite with 21 tests covering:
    - All sensor types (temperature, voltage, current, charge/percent)
    - Multi-sensor parsing (mixed message types in single walk)
    - Error handling (invalid formats, non-string values, missing colons)
    - Edge cases (empty results, SNMP timeouts, existing sensor preservation)
    All tests passing with Mox-based SNMP adapter mocking.
  - Pattern: Follows Arista vendor module architecture with post_process_sensors/2
    callback. Client.walk returns map format, converted to list for message parsing.
  - Result: Dell PowerVault storage arrays now support temperature/voltage/current/
    battery monitoring via text message parsing. Gap: CRITICAL → RESOLVED.
    Parity: 0% → 85% (limited by FCMGMT-MIB available sensors).

2026-02-11 - feat: add Dell UPS power sensor and HP BladeSystem/Moonshot complete monitoring
  - Files: priv/profiles/os_discovery/dell-ups.yaml (enhanced)
    Added Dell UPS power consumption sensor via DELL-SNMP-UPS-MIB:
    - Power: physicalOutputPresentConsumption (OID .1.3.6.1.4.1.674.10902.2.120.2.6.0)
    Enables capacity planning and load tracking for Dell UPS units.
    Gap: MEDIUM (missing power sensor) → RESOLVED. Parity: 80% → 100%.
  - Files: priv/profiles/os_discovery/hpblmos.yaml (enhanced)
    Created comprehensive HP BladeSystem/Moonshot sensor profile:
    - Temperature: Blade chassis component temperatures (OID .1.3.6.1.4.1.232.22.2.3.1.2.1.6)
    - Power: PSU output wattage with max limit (OID .1.3.6.1.4.1.232.22.2.5.1.1.1.10)
    - State: Fan operational status (OID .1.3.6.1.4.1.232.22.2.3.1.3.1.11)
    - State: PSU condition status (OID .1.3.6.1.4.1.232.22.2.5.1.1.1.17)
    Skip logic: Only monitors present fans/PSUs (presence check to avoid non-existent sensors)
    Gap: CRITICAL (no sensor profile) → RESOLVED. Parity: 0% → 80%.
  - Analysis Discovery: Dell Enterprise Servers (iDRAC) already have excellent coverage
    The existing drac.yaml profile includes comprehensive sensor monitoring:
    - Temperature: Multiple probes with thresholds (4 types)
    - Voltage: PSU voltage monitoring (2 types)
    - Current: Amperage probes (3 types, filtered for wattage type)
    - Power: Chassis power (current, peak, potential, idle, max - 5 types)
    - Fanspeed: Cooling device monitoring with thresholds
    - State: Extensive state sensors (global system, IDSDM card, intrusion, IOM, etc.)
    Previous gap analysis incorrectly reported 0% parity - actual parity is 95%+.
  - Result: Tier 4 complete. Dell UPS now has full monitoring capability, HP BladeSystem
    blade chassis fully supported with temperature/power/fan/PSU monitoring. Dell servers
    confirmed to have excellent existing coverage via iDRAC/OpenManage MIBs.

2026-02-11 - feat: add Force10 FTOS and optical transceiver monitoring (ProCurve, Comware)
  - Files: priv/profiles/os_discovery/ftos.yaml (enhanced)
    Added Dell Force10 FTOS temperature monitoring for all 3 series:
    - S-Series: chStackUnitTemp (OID .1.3.6.1.4.1.6027.3.10.1.2.2.1.14)
    - C-Series: chSysCardTemp (OID .1.3.6.1.4.1.6027.3.8.1.2.1.1.5)
    - E-Series: chSysCardUpperTemp + chSysCardLowerTemp (OIDs .1.3.6.1.4.1.6027.3.1.1.2.3.1.8-9)
    Completes Tier 1 critical network switch monitoring. Parity: 0% → 90%.
  - Files: priv/profiles/os_discovery/procurve.yaml (enhanced)
    Added HP ProCurve transceiver optical monitoring via HP-ICF-TRANSCEIVER-MIB::hpicfXcvrInfoTable:
    - Temperature: hpicfXcvrTemp (OID .1.3.6.1.4.1.11.2.14.11.5.1.82.1.1.1.1.11)
    - Bias Current: hpicfXcvrBias (OID .1.3.6.1.4.1.11.2.14.11.5.1.82.1.1.1.1.13)
    - Supply Voltage: hpicfXcvrVoltage (OID .1.3.6.1.4.1.11.2.14.11.5.1.82.1.1.1.1.12)
    - RX Power (dBm): hpicfXcvrRxPower (OID .1.3.6.1.4.1.11.2.14.11.5.1.82.1.1.1.1.14)
    - TX Power (dBm): hpicfXcvrTxPower (OID .1.3.6.1.4.1.11.2.14.11.5.1.82.1.1.1.1.15)
    Enables optical SFP/SFP+ monitoring for fiber uplinks. Parity: 60% → 100%.
  - Files: priv/profiles/os_discovery/comware.yaml (enhanced)
    Added HP Comware transceiver optical monitoring via HH3C-TRANSCEIVER-INFO-MIB::hh3cTransceiverInfoTable:
    - Temperature: hh3cTransceiverTemperature (OID .1.3.6.1.4.1.25506.2.70.1.1.1.15)
    - Bias Current: hh3cTransceiverBiasCurrent (OID .1.3.6.1.4.1.25506.2.70.1.1.1.17)
    - Supply Voltage: hh3cTransceiverVoltage (OID .1.3.6.1.4.1.25506.2.70.1.1.1.16)
    - RX Power (dBm): hh3cTransceiverCurRXPower (OID .1.3.6.1.4.1.25506.2.70.1.1.1.9)
    - TX Power (dBm): hh3cTransceiverCurTXPower (OID .1.3.6.1.4.1.25506.2.70.1.1.1.12)
    Completes Comware sensor coverage. Parity: 60% → 95%.
  - Result: Tier 1 + Tier 2 complete. All critical network switches have temperature monitoring,
    and both ProCurve/Comware have complete optical transceiver diagnostics (temp, current, voltage,
    RX/TX power). Force10 FTOS data center switches fully supported. Fiber link monitoring enabled.

2026-02-11 - feat: add critical network switch sensor support (HP Comware, Dell PowerConnect, Dell SONiC)
  - Files: priv/profiles/os_discovery/comware.yaml (enhanced)
    Added HP Comware chassis temperature monitoring via HH3C-ENTITY-EXT-MIB::hh3cEntityExtTemperature
    (OID 1.3.6.1.4.1.25506.2.6.1.1.1.1.12). This restores fundamental temperature monitoring capability
    for HP Comware switches, enabling overheating alerts. Gap identified as CRITICAL in Phase 3 analysis.
  - Files: priv/profiles/os_discovery/powerconnect.yaml (enhanced)
    Added Dell PowerConnect/DNOS CPU temperature monitoring via FASTPATH-BOXSERVICES-PRIVATE-MIB
    (OID 1.3.6.1.4.1.674.10895.5000.2.6132.1.1.43.1.8.1.5). Enables temperature monitoring for
    common enterprise access switches.
  - Files: priv/profiles/os_discovery/dell-sonic.yaml (new)
    Created comprehensive sensor profile for Dell SONiC switches using NETGEAR-BOXSERVICES-PRIVATE-MIB
    (Quanta-based platform). Sensors: temperature (boxServicesTempSensorState), fan speed
    (boxServicesFanSpeed), PSU state (boxServicesPowSupplyItemState with 8 states). Enables hardware
    monitoring for modern data center switches.
  - Result: Top 3 critical network switch monitoring gaps resolved. HP Comware gains fundamental
    temperature capability (40% → 60% parity), Dell PowerConnect gains first sensor support
    (0% → 80% parity), Dell SONiC gains complete hardware monitoring (0% → 95% parity).

2026-02-11 - audit: Phase 3 - HP/Aruba and Dell sensor analysis complete
  - Files: docs/librenms-audit/sensors/hp-aruba-comparison.md (new)
    Analyzed 6 HP/Aruba platforms: ProCurve (60%), Comware (40%), hpblmos (0%), ArubaOS (100%),
    ArubaOS-CX (100%), Instant On (100%). Identified critical gaps in Comware temperature and
    transceiver support.
  - Files: docs/librenms-audit/sensors/dell-comparison.md (new)
    Analyzed 11 Dell platforms: OS10 (100%), rPDU (100%), UPS (80%), OME-M (100%), Compellent (100%),
    PowerVault (20%), PowerConnect (0%), Force10 (0%), SONiC (0%), Quanta (40%), Servers (0%).
    Found network switches severely underserved compared to storage/infrastructure.
  - Files: docs/librenms-audit/PHASE3-VENDOR-EXPANSION.md (new)
    Executive summary with implementation priorities. Tier 1 critical: Network switch sensor gaps
    (8-12 hours). Tier 2 high: Optical transceiver monitoring (4-6 hours). Tier 3: Storage/compute
    (7-10 hours). Total effort: 26-38 hours for full parity.
  - Key Finding: Modern platforms excellent (Aruba CX, Dell OS10), but legacy network switches
    missing fundamental monitoring. HP Comware temperature gap BLOCKING (breaks alerts).
    Dell PowerConnect/FTOS/SONiC missing all sensors (CRITICAL for enterprise deployments).

2026-02-11 - feat: implement Arista EOS sensor enhancements (DOM power, thresholds, grouping)
  - Files: lib/towerops/snmp/profiles/vendors/arista.ex (enhanced)
    Implemented all critical and nice-to-have Arista sensor improvements identified in audit:
    1. DOM power conversion (watts→dBm): Detects optical power sensors via regex, converts
       using formula dBm = 10*log10(watts*1000), preserves original value in metadata
    2. Arista threshold discovery: Walks ARISTA-ENTITY-SENSOR-MIB::aristaEntSensorThresholdTable,
       applies 4 threshold types (low_critical, low_warn, high_warn, high_critical), converts
       thresholds to dBm for optical sensors
    3. Smart grouping: Organizes sensors by SFPs, PSUs, Platform (chipsets), Power Connectors, System
    4. Description cleanup: Removes redundant "sensor" text, simplifies PSU naming, cleans whitespace
  - Files: lib/towerops/snmp/profiles/dynamic.ex (integration)
    Added apply_vendor_post_processing/3 to call Arista enhancements after sensor discovery.
    Applies to both arista_eos and arista-mos profiles.
  - Files: test/towerops/snmp/profiles/vendors/arista_test.exs (comprehensive tests)
    Added 23 new tests covering DOM conversion (6 tests), threshold discovery (4 tests),
    smart grouping (6 tests), description cleanup (5 tests), and end-to-end post-processing (1 test).
    All tests passing (30/30 in arista_test.exs, 6367/6367 total).
  - Result: Arista EOS support upgraded from 85% to 100% LibreNMS parity. All critical gaps closed:
    optical power now displayed correctly in dBm, alerting enabled via thresholds, sensors
    organized for better UX, descriptions cleaned up.

2026-02-11 - audit: Phase 2 - complete top vendor sensor analysis (Cisco, Juniper, Arista)
  - Files: docs/librenms-audit/sensors/*.md, PHASE2-VENDOR-ANALYSIS.md (new)
    Completed comprehensive sensor discovery analysis for top 3 enterprise vendors:
    - cisco-ios-comparison.md: Cisco IOS/XE/XR/NX-OS analysis (95% parity, production ready)
    - juniper-junos-comparison.md: JunOS analysis (100% IDENTICAL - perfect parity)
    - arista-eos-comparison.md: Arista EOS analysis (85% parity, gaps identified)
    - PHASE2-VENDOR-ANALYSIS.md: Comprehensive summary and recommendations
  - Key findings:
    - Cisco: All critical sensors covered, YAML-based discovery, minor cellular gaps (rare devices)
    - Juniper: junos.yaml byte-for-byte identical (30+ sensor types, 38+ OIDs, 10 MIBs)
    - Arista: Missing DOM power conversion (watts→dBm) and ARISTA-ENTITY-SENSOR-MIB thresholds
  - Identified actionable fixes for Arista (2 critical, 2 minor):
    1. DOM Rx/Tx power conversion (2-3 hours, enables proper optical monitoring)
    2. Arista threshold discovery (4-6 hours, enables alerting)
    3. Smart grouping (2-3 hours, UX improvement)
    4. Description cleanup (1-2 hours, cosmetic)
  - Towerops advantages: CPU/Memory/Uptime sensors (Arista), PSU/Fan state (Arista),
    cleaner YAML architecture, index templates, pre-resolved MIB cache

2026-02-11 - audit: complete LibreNMS parity audit and add missing OS detection profiles
  - Files: docs/librenms-audit/*.md (new)
    Completed comprehensive audit comparing Towerops device detection and sensor discovery
    against LibreNMS. Created detailed documentation:
    - detection-algorithm.md: LibreNMS 2-pass detection system analysis
    - towerops-detection.md: Towerops 4-phase detection system analysis
    - vendor-detection-comparison.csv: Priority vendor comparison (MikroTik, Ubiquiti)
    - librenms-sensors.md: LibreNMS sensor discovery architecture (360+ modules, 24 types)
    - EXECUTIVE-SUMMARY.md: Overall findings (99.5% detection parity, 95%+ sensor parity)
    - IMPLEMENTATION-STATUS.md: Phase 1 completion status
  - Files: priv/profiles/os_detection/conteg-pdu.yaml, microsens-g6.yaml (new)
    Added 2 missing OS detection profiles from LibreNMS to achieve 100% profile parity
    (786 profiles total). Conteg PDU for power monitoring devices, MICROSENS G6 for
    network switches.
  - Key findings: Towerops has full detection parity with LibreNMS plus architectural
    enhancements (4-phase matching, Rust NIF MIB resolution, ETS caching, index templates).
    MikroTik coverage verified: all 22 sensor tables discovered identically. Ubiquiti
    coverage: 8 of 9 profiles identical (airos-af has additional fallback in Towerops).

2026-02-11 - test: add 100% coverage for ExpiredBanCleanupWorker
  - File: test/towerops/workers/expired_ban_cleanup_worker_test.exs (new)
    Added 7 tests covering perform/1: expired ban deletion, active ban preservation,
    permanent ban preservation, mixed scenarios, empty database, PubSub broadcast
    on deletion, and no broadcast when nothing deleted.

2026-02-11 - fix: handle "null" string values in SNMP parsing
  - File: lib/towerops/snmp/profiles/base.ex (parse_integer/1, parse_float/1)
    Added explicit handling for "null" string values before attempting integer/float
    conversion. Some devices (e.g., AirOS 8.7.14) return the literal string "null"
    for unavailable metrics like UCD CPU stats, which caused ArgumentError when
    parse_integer/parse_float tried to convert them. Now returns nil, which gets
    coerced to 0 by the || 0 fallback in the calling code.

2026-02-11 - fix: deduplicate sensors with leading-dot OID differences
  - File: lib/towerops/snmp/profiles/dynamic.ex (deduplicate_by_oid/1)
    Normalized OIDs by stripping leading dots when grouping for deduplication.
    gosnmp (agent) returns OIDs with leading dots (.1.3.6.1...) but vendor
    modules and YAML profiles define them without. Without normalization,
    the same sensor appeared twice (e.g. "DHCP Leases" and "DHCP Lease Count").
  - File: lib/towerops/snmp/discovery.ex (deduplicate_sensors_by_oid/1)
    Same leading-dot normalization applied to the discovery-level deduplication.
  - File: lib/towerops/snmp/profiles/vendors/routeros.ex (wireless_oid_defs/0)
    Removed DHCP Leases entry (mtxrDHCPLeaseCount) since the routeros.yaml
    YAML profile already defines it in count_sensor_oids. Eliminates the
    source of duplication for MikroTik devices.
  - Tests updated in dynamic_test.exs and routeros_test.exs

2026-02-11 - fix: agent discovery and polling improvements
  - File: lib/towerops_web/channels/agent_channel.ex (build_polling_queries/1, process_interface_stats/3)
    Changed interface traffic OIDs from 32-bit counters (ifInOctets/ifOutOctets) to 64-bit
    HC counters (ifHCInOctets/ifHCOutOctets). 32-bit counters wrap at ~4.3GB which causes
    wildly inaccurate readings on gigabit+ links.
  - File: lib/towerops/snmp/adapters/replay.ex (parse_value/1)
    Fixed negative integer parsing regex from ^\d+$ to ^-?\d+$ so values like
    entPhySensorScale=-3 are parsed as integers. Previously fell through to string,
    causing sensor divisor calculation to silently return 1 instead of 1000.
  - File: lib/towerops_web/channels/agent_channel.ex (build_discovery_queries/0)
    Replaced walking entire Cisco enterprise tree (1.3.6.1.4.1.9) with targeted subtrees
    for CPU, ENVMON, CDP, and WAP to avoid 50k+ OID responses and timeouts. Added Juniper,
    HP/H3C, Fortinet, and Cambium vendor OIDs for broader device support.
  - File: lib/towerops_web/channels/agent_channel.ex (build_discovery_queries/0)
    Added missing discovery OIDs: ENTITY-STATE-MIB for state sensors, HOST-RESOURCES-MIB
    tables for processors/storage/devices, and UCD-SNMP-MIB for Linux CPU stats fallback.
  - Tests added in replay_test.exs and agent_channel_test.exs

2026-02-11 - fix: handle RouterOS version returned as list of integers
  - File: lib/towerops/snmp/profiles/vendors/routeros.ex (detect_version/1)
    Added pattern match to handle case where SNMP version OID returns list
    of integers [7, 20, 6] instead of binary string "7.20.6". Converts list
    to dotted string format using Enum.join/2.
  - File: test/towerops/snmp/profiles/vendors/routeros_test.exs
    Added two test cases for list-formatted versions with and without license level.
  - Bug: Warning logged "Failed to detect version: {:ok, [7, 20, 6]}" even though
    version was successfully retrieved, just in wrong format.

2026-02-11 - fix: normalize OIDs so agent-polled sensors match regardless of leading dot
  - File: lib/towerops_web/channels/agent_channel.ex
    Agent (gosnmp) returns OIDs with leading dots (e.g. ".1.3.6.1.4.1.41112...")
    but sensor OIDs stored in DB may omit the dot. process_sensor_reading used
    direct Map.get which silently missed wireless/vendor sensors on Ubiquiti airOS.
  - Fix: Added normalize_oid_keys/1 to strip leading dots from all oid_values keys,
    and normalize sensor.sensor_oid too. Both sides now match without leading dot.
  - File: test/towerops/snmp/sensor_change_detector_test.exs
    Fixed flaky refute_receive tests that subscribed to global "device:events" topic
    and caught unrelated device_discovered events from concurrent async tests.
    Changed to device-specific topic "device:#{device.id}" instead.

2026-02-11 - feat: add sensor value change events for all sensor types
  - New file: lib/towerops/snmp/sensor_change_detector.ex
    Extracted shared sensor change detection logic from DevicePollerWorker.
    Detects threshold violations, spike/drop for % sensors, and new
    sensor_value_changed events for all non-% sensors when value changes.
  - File: lib/towerops/devices/event.ex
    Added "sensor_value_changed" to allowed event types.
  - File: lib/towerops/workers/device_poller_worker.ex
    Replaced inline detect_sensor_changes with SensorChangeDetector module.
    Removed ~240 lines of private functions now in shared module.
  - File: lib/towerops_web/channels/agent_channel.ex
    Added change detection and last_value/last_checked_at update to
    process_sensor_reading/3 so agent-polled devices get same events
    as Phoenix-polled devices.
  - New file: test/towerops/snmp/sensor_change_detector_test.exs
    11 tests covering value changes, spike/drop, thresholds, nil handling.

2026-02-11 - fix: live polling uses effective agent from site/org cascade
  - File: lib/towerops_web/live/graph_live/show.ex
  - Bug: Live poll mode used Agents.get_device_assignment/1 which only checks
    direct device-level assignments. Devices with agent assigned at site or
    org level fell through to direct Phoenix SNMP instead of routing to agent.
  - Fix: Replaced with Agents.get_effective_agent_token/1 which walks the
    device → site → org cascade, matching how regular polling resolves agents.
    Also simplified request_agent_live_poll_and_wait to accept token_id directly.

2026-02-11 - fix: round sensor values to 1 decimal place
  - Files: lib/towerops/workers/device_poller_worker.ex (poll_simple_sensor/2),
           lib/towerops_web/channels/agent_channel.ex (process_sensor_reading/3)
  - Bug: Division by sensor_divisor produced full float precision (e.g. 34.09999999)
  - Fix: Wrap division result with Float.round(..., 1) in both Phoenix and agent paths

2026-02-11 - fix: update device status from agent monitoring checks
  - File: lib/towerops_web/channels/agent_channel.ex (store_monitoring_check/2)
  - Bug: Devices polled by remote agents stayed in "unknown" state because
    store_monitoring_check saved the ping result but never called
    Devices.update_device_status/2. The DeviceMonitorWorker (Phoenix-side)
    skips agent-assigned devices entirely, so nothing ever updated status.
  - Fix: Added update_device_status_from_check/2 and handle_agent_status_change/3
    to derive :up/:down from check status, update device, create alerts on
    transitions, and broadcast via PubSub — matching DeviceMonitorWorker behavior
  - Also added alias Towerops.Alerts to the channel module

2026-02-11 - fix: update last_snmp_poll_at for agent-polled devices
  - File: lib/towerops_web/channels/agent_channel.ex (process_polling_result/3)
  - Bug: Devices polled by remote agents showed "Last Polled: Never" because
    process_polling_result never called Devices.update_snmp_poll_time/1
  - The Phoenix-side DevicePollerWorker already had this call (line 172),
    but the agent channel path was missing it
  - Fix: Added Devices.update_snmp_poll_time(device) at the top of
    process_polling_result/3, matching the DevicePollerWorker pattern