1906 lines
56 KiB
Markdown
1906 lines
56 KiB
Markdown
# LibreNMS vs TowerOps SNMP Review
|
|
|
|
## Scope
|
|
|
|
This review compares the local LibreNMS checkout in `~/dev/librenms/` against the current TowerOps codebase in `/Users/graham/dev/towerops/towerops-web`.
|
|
|
|
I focused on:
|
|
|
|
- SNMP transport and polling behavior
|
|
- Device discovery and device lifecycle handling
|
|
- Metric/entity gathering
|
|
- Data normalization and threshold semantics
|
|
- Persistence and historical storage
|
|
- Operational robustness
|
|
- Areas where TowerOps is already better positioned
|
|
|
|
Key code reviewed included:
|
|
|
|
- LibreNMS:
|
|
- `app/Console/Commands/DevicePoll.php`
|
|
- `app/Console/Commands/DeviceDiscover.php`
|
|
- `app/PerDeviceProcess.php`
|
|
- `LibreNMS/Modules/*`
|
|
- `LibreNMS/Device/YamlDiscovery.php`
|
|
- `LibreNMS/Data/Store/*`
|
|
- `includes/discovery/*`
|
|
- `includes/polling/*`
|
|
- `app/Models/Device.php`, `Port.php`, `Sensor.php`, `Storage.php`, `Processor.php`
|
|
- TowerOps:
|
|
- `lib/towerops/snmp/client.ex`
|
|
- `lib/towerops/snmp/discovery.ex`
|
|
- `lib/towerops/snmp/profiles/base.ex`
|
|
- `lib/towerops/snmp/profiles/dynamic.ex`
|
|
- `lib/towerops/snmp/deferred_discovery.ex`
|
|
- `lib/towerops/snmp.ex`
|
|
- `lib/towerops/workers/device_poller_worker.ex`
|
|
- `lib/towerops/snmp/*.ex` entity schemas
|
|
- `priv/repo/migrations/*timescaledb*`
|
|
|
|
## Executive Summary
|
|
|
|
TowerOps already has a solid SNMP foundation. It is not a toy implementation. It has:
|
|
|
|
- a real discovery pipeline
|
|
- a broad vendor-profile system
|
|
- historical storage in Postgres/Timescale
|
|
- parallel polling
|
|
- topology, ARP, MAC, and wireless-client handling
|
|
- agent-aware polling assignment
|
|
|
|
That said, LibreNMS is still materially more mature as a general SNMP monitoring platform.
|
|
|
|
The biggest differences are not "can TowerOps talk SNMP?" but:
|
|
|
|
- breadth of discovered entity types
|
|
- depth of metric semantics and normalization
|
|
- transport/polling heuristics for difficult devices
|
|
- lifecycle handling for discovered entities
|
|
- pluggable datastore/export story
|
|
- module-level operational maturity built up from years of edge cases
|
|
|
|
If the target is "as robust and feature rich as LibreNMS, then much more", TowerOps should keep its current architecture but expand it in a LibreNMS-like direction:
|
|
|
|
1. make transport and polling behavior much smarter
|
|
2. expand entity/module coverage aggressively
|
|
3. add richer metric semantics and threshold/state models
|
|
4. strengthen discovery drift and lifecycle handling
|
|
5. treat topology, subscriber correlation, and agent-distributed polling as the differentiators that go beyond LibreNMS
|
|
|
|
## Expanded Strategy
|
|
|
|
The five points above are the right high-level direction, but they need to be translated into an explicit engineering strategy. Below is what each one should mean in practice.
|
|
|
|
### 1. Make transport and polling behavior much smarter
|
|
|
|
This is the most immediate leverage point because it improves every current and future module.
|
|
|
|
Right now TowerOps has solid SNMP primitives, but it still behaves too much like a clean protocol client and not enough like a hardened polling engine. LibreNMS has a lot of ugly historical logic here, but that ugliness exists for a reason: real devices are inconsistent, slow, buggy, and often only partially standards-compliant.
|
|
|
|
TowerOps should evolve from "SNMP client plus polling worker" into "adaptive polling engine with device-specific transport policy".
|
|
|
|
That means adding a transport capability model per device or profile, including:
|
|
|
|
- preferred operation mode: `get`, `get_next`, `walk`, `bulk_walk`
|
|
- whether GETBULK is safe
|
|
- max repeaters / repetition window
|
|
- retry policy
|
|
- timeout class
|
|
- whether responses can be unordered
|
|
- whether certain OIDs must avoid bulk or avoid combined requests
|
|
- whether a device should prefer table walks vs selected-item gets
|
|
- whether interface polling should be selective or full-table
|
|
|
|
This should not stay implicit in code branches. It should become explicit discovered or profile-driven capability data.
|
|
|
|
Concrete improvements:
|
|
|
|
- ✅ **DONE (2026-03-26)**: implement a true multi-OID GET path so `get_multiple/2` is not a loop of individual requests
|
|
- move more polling to table-oriented retrieval, especially for interfaces and sensors
|
|
- add adaptive bulk sizing based on response size, timeout rate, and prior successful runs
|
|
- add a device-level "transport memory" layer so TowerOps learns what works on a specific box
|
|
- distinguish "device is down" from "this module/OID strategy is bad for this device"
|
|
- add poll cost accounting: request count, bytes returned, duration, failure rate, timeout count
|
|
|
|
The goal is not just speed. The goal is stable polling on weird devices without special-casing everything forever.
|
|
|
|
A good end state would be:
|
|
|
|
- first discovery runs conservatively
|
|
- TowerOps learns the device's behavior
|
|
- later polls use an optimized strategy
|
|
- if errors increase, TowerOps automatically degrades to a safer strategy
|
|
|
|
LibreNMS has a lot of static heuristics. TowerOps can do better by making that adaptive.
|
|
|
|
### 2. Expand entity/module coverage aggressively
|
|
|
|
TowerOps should treat SNMP coverage as a product surface, not a side effect of polling.
|
|
|
|
LibreNMS is stronger because it models many more things. That breadth matters because operators do not think in terms of "poll sysObjectID and a few sensors". They think in terms of:
|
|
|
|
- optics
|
|
- memory
|
|
- routes
|
|
- BGP neighbors
|
|
- STP topology
|
|
- QoS queues
|
|
- storage
|
|
- power supplies
|
|
- printer consumables
|
|
- xDSL lines
|
|
- VM guests
|
|
- hardware inventory
|
|
|
|
TowerOps should formalize a module catalog and then grow it quickly.
|
|
|
|
Suggested first-class module families:
|
|
|
|
- `Core`
|
|
- `Interfaces`
|
|
- `InterfaceExtendedStats`
|
|
- `Sensors`
|
|
- `StateSensors`
|
|
- `Processors`
|
|
- `Mempools`
|
|
- `Storage`
|
|
- `Transceivers`
|
|
- `EntityPhysical`
|
|
- `Vlans`
|
|
- `Ipv4Addresses`
|
|
- `Ipv6Addresses`
|
|
- `Neighbors`
|
|
- `Arp`
|
|
- `Mac`
|
|
- `Routes`
|
|
- `Stp`
|
|
- `Qos`
|
|
- `Bgp`
|
|
- `Ospf`
|
|
- `Services`
|
|
- `Wireless`
|
|
- `PrinterSupplies`
|
|
- `Xdsl`
|
|
|
|
This should not all land at once, but the module system should be designed for that future immediately.
|
|
|
|
The order I would recommend:
|
|
|
|
1. `Mempools`
|
|
2. `Transceivers`
|
|
3. `EntityPhysical`
|
|
4. `Routes`
|
|
5. `Qos`
|
|
6. `Bgp`
|
|
7. `Stp`
|
|
|
|
Why this order:
|
|
|
|
- mempools and transceivers are common, visible, and high-value
|
|
- entity physical gives a backbone for inventory, FRUs, and relationship mapping
|
|
- routes/QoS/BGP/STP move TowerOps from device monitoring into actual network operations monitoring
|
|
|
|
This should be done with a mix of:
|
|
|
|
- generic standard-MIB implementations
|
|
- vendor/profile extensions
|
|
- a shared module contract for discovery, poll, sync, cleanup, and capability checks
|
|
|
|
The key is to prevent new coverage from turning the codebase into an ever-growing `device_poller_worker.ex` blob. Coverage growth must come with module boundaries.
|
|
|
|
### 3. Add richer metric semantics and threshold/state models
|
|
|
|
TowerOps currently stores readings well, but it does not yet model metric meaning as deeply as LibreNMS.
|
|
|
|
To match and then surpass LibreNMS, TowerOps needs to treat each discovered metric-bearing entity as more than:
|
|
|
|
- an OID
|
|
- a current value
|
|
- a timestamp
|
|
|
|
Each metric should carry enough metadata for TowerOps to reason about it correctly.
|
|
|
|
That means first-class support for:
|
|
|
|
- raw value
|
|
- normalized value
|
|
- divisor
|
|
- multiplier
|
|
- unit
|
|
- unit family
|
|
- metric type: `gauge`, `counter`, `derive`, `state`
|
|
- threshold policy
|
|
- warning/critical bounds
|
|
- low and high bounds
|
|
- reset/wrap behavior
|
|
- value source quality
|
|
- preferred graph aggregation
|
|
|
|
For state sensors, TowerOps should introduce a proper state model, not just a string description on readings.
|
|
|
|
LibreNMS has stronger state semantics through translation tables. TowerOps should add something similar but cleaner:
|
|
|
|
- state index definition
|
|
- translation map from raw values to normalized state
|
|
- generic severity mapping: `ok`, `warning`, `critical`, `unknown`
|
|
- change-event policy
|
|
- suppression/debounce policy
|
|
|
|
For numeric metrics, TowerOps should support discovered threshold metadata and local overrides separately.
|
|
|
|
There are really three threshold layers:
|
|
|
|
1. vendor/device-provided thresholds
|
|
2. profile defaults
|
|
3. operator overrides
|
|
|
|
Those should be modeled independently so the system can explain why a value is considered abnormal.
|
|
|
|
This also enables better product behavior:
|
|
|
|
- graphs can render warning/critical zones correctly
|
|
- alerts can distinguish vendor-threshold breach vs operator policy breach
|
|
- anomaly detection can compare against both thresholds and learned baselines
|
|
|
|
The most important conceptual change here is:
|
|
|
|
TowerOps should move from "store readings and run checks" to "understand the semantics of each metric, then use checks as one consumer of that model".
|
|
|
|
That would be a stronger architecture than LibreNMS.
|
|
|
|
### 4. Strengthen discovery drift and lifecycle handling
|
|
|
|
LibreNMS is robust partly because it assumes discovered entities are unstable:
|
|
|
|
- interfaces rename
|
|
- ifIndex values drift
|
|
- sensors vanish and come back
|
|
- optics move slots
|
|
- storage entries reorder
|
|
- devices partially implement MIBs
|
|
|
|
TowerOps already syncs discovered entities and cleans up certain stale records, but it should develop a full discovery lifecycle model.
|
|
|
|
Every discovered entity should have lifecycle semantics:
|
|
|
|
- first discovered at
|
|
- last seen at
|
|
- last changed at
|
|
- active / stale / deleted state
|
|
- stable identity key
|
|
- mutable display attributes
|
|
- drift reason if identity changed
|
|
|
|
Examples:
|
|
|
|
- an interface can keep the same stable identity while alias, speed, or label changes
|
|
- a transceiver can be removed and later reinserted
|
|
- a storage entry can move from "active" to "missing" without immediate hard deletion
|
|
- a sensor can be marked unsupported by current firmware after an upgrade
|
|
|
|
TowerOps should also record discovery drift as auditable events:
|
|
|
|
- interface renamed
|
|
- ifIndex changed
|
|
- sensor threshold changed
|
|
- storage table reordered
|
|
- model/serial/firmware changed
|
|
- LLDP capability set changed
|
|
|
|
This is important for both reliability and UX. Operators need to know whether a graph or alert changed because the system discovered something new or because the device actually changed.
|
|
|
|
A strong end state would include two layers:
|
|
|
|
- current entity view
|
|
- discovery history / identity history
|
|
|
|
That would let TowerOps survive SNMP instability much better than a simple overwrite model.
|
|
|
|
This is also where entity sync strategy matters. For each entity family, TowerOps should explicitly define:
|
|
|
|
- identity key
|
|
- update fields
|
|
- staleness window
|
|
- hard delete window
|
|
- drift event rules
|
|
|
|
Without that, robustness will always be uneven across modules.
|
|
|
|
### 5. Treat topology, subscriber correlation, and agent-distributed polling as the differentiators that go beyond LibreNMS
|
|
|
|
This is where TowerOps should stop thinking like "an NMS trying to catch up" and start thinking like "a network operations platform that happens to include SNMP".
|
|
|
|
LibreNMS is broad and mature, but its center of gravity is still classic device monitoring.
|
|
|
|
TowerOps already has stronger raw ingredients for something more valuable:
|
|
|
|
- topology inference from neighbors, ARP, and MAC evidence
|
|
- wireless client discovery with history
|
|
- subscriber correlation
|
|
- multi-tenant data model
|
|
- agent-aware polling ownership
|
|
|
|
These should not remain side features. They should become the main differentiator.
|
|
|
|
There are three major product directions here.
|
|
|
|
First, topology should become operational, not just descriptive.
|
|
|
|
That means:
|
|
|
|
- infer path relationships, not just direct links
|
|
- attach devices, interfaces, wireless clients, and subscribers into one graph
|
|
- use graph state during incident analysis
|
|
- identify probable upstream failure points
|
|
- compute blast radius automatically
|
|
|
|
A traditional NMS shows "these devices are down". A stronger TowerOps should be able to say "this uplink failure is causing 143 impacted subscribers across these APs and these sites".
|
|
|
|
Second, subscriber correlation should become a first-class consumer of SNMP and topology data.
|
|
|
|
That means:
|
|
|
|
- correlate subscribers to APs, sectors, switches, and upstream paths
|
|
- retain movement history
|
|
- distinguish probable vs confirmed attachment
|
|
- use RF quality, ARP evidence, and topology together
|
|
- enable customer-impact views instead of just device-impact views
|
|
|
|
This is much closer to how WISPs and access-network operators actually think.
|
|
|
|
Third, agent-distributed polling should become a smart execution fabric.
|
|
|
|
Right now TowerOps is aware of agents. The next step is to make scheduling and placement intelligent:
|
|
|
|
- route polling based on network reachability and locality
|
|
- detect overloaded or unhealthy agents
|
|
- move devices between agents automatically
|
|
- support partial capability ownership by module
|
|
- allow fallback execution strategies
|
|
- track poll success by agent, region, and module
|
|
|
|
That would put TowerOps in a position LibreNMS generally is not designed for:
|
|
|
|
- hybrid central/distributed polling
|
|
- tenant-aware placement
|
|
- topology-aware scheduling
|
|
- richer execution telemetry
|
|
|
|
The combined opportunity is substantial.
|
|
|
|
If TowerOps executes well here, the platform can evolve from:
|
|
|
|
- "SNMP monitoring system"
|
|
|
|
into:
|
|
|
|
- "distributed network observability and operations system"
|
|
|
|
That is the path to "much more".
|
|
|
|
## Where TowerOps Is Already Strong
|
|
|
|
These are real strengths, not consolation prizes.
|
|
|
|
### 1. Better application architecture for future growth
|
|
|
|
TowerOps has a cleaner domain split than LibreNMS:
|
|
|
|
- `Towerops.Snmp` as a context API
|
|
- explicit schemas for sensors, interfaces, storage, processors, neighbors, ARP, MACs, wireless clients
|
|
- Oban workers for polling/discovery/cleanup
|
|
- TimescaleDB-oriented historical tables
|
|
|
|
LibreNMS has a lot of mature logic, but much of it still spans legacy includes, DB helpers, and mixed discovery/polling styles.
|
|
|
|
### 2. Better integrated topology and subscriber-aware data model
|
|
|
|
TowerOps is already ahead in areas LibreNMS treats more generically:
|
|
|
|
- neighbor polling and topology inference
|
|
- ARP/MAC correlation into topology
|
|
- wireless client discovery with history
|
|
- subscriber/device correlation (`Gaiia.SubscriberMatching`)
|
|
- agent-aware ownership and result discard on reassignment
|
|
|
|
This is the right direction if TowerOps wants to become more than a standard NMS.
|
|
|
|
### 3. Better relational and analytical storage story
|
|
|
|
TowerOps stores historical SNMP data in PostgreSQL/TimescaleDB:
|
|
|
|
- `snmp_sensor_readings`
|
|
- `snmp_interface_stats`
|
|
- `snmp_processor_readings`
|
|
- `snmp_storage_readings`
|
|
- `wireless_client_readings`
|
|
|
|
LibreNMS primarily treats RRD as the historical source of truth and the SQL DB as current-state metadata. That model is proven, but TowerOps has a better foundation for analytics, joins, ML-style insights, and tenant-aware retention.
|
|
|
|
### 4. Strong vendor coverage direction
|
|
|
|
TowerOps already has a large vendor-profile surface under `lib/towerops/snmp/profiles/vendors/`. That is the correct scaling model if it remains disciplined.
|
|
|
|
## Major Findings
|
|
|
|
## 1. LibreNMS has a much broader module surface
|
|
|
|
LibreNMS ships a real module catalog under `LibreNMS/Modules/`, including:
|
|
|
|
- `ArpTable`
|
|
- `Availability`
|
|
- `Core`
|
|
- `DiscoveryArp`
|
|
- `EntityPhysical`
|
|
- `HrDevice`
|
|
- `IpSystemStats`
|
|
- `Ipv4Addresses`
|
|
- `Ipv6Addresses`
|
|
- `Ipv6Nd`
|
|
- `Isis`
|
|
- `MacAccounting`
|
|
- `Mempools`
|
|
- `Mpls`
|
|
- `Nac`
|
|
- `Netstats`
|
|
- `Ospf`
|
|
- `Ospfv3`
|
|
- `PortSecurity`
|
|
- `PortsStack`
|
|
- `PrinterSupplies`
|
|
- `Qos`
|
|
- `Routes`
|
|
- `Services`
|
|
- `Slas`
|
|
- `Storage`
|
|
- `Stp`
|
|
- `Transceivers`
|
|
- `UcdDiskio`
|
|
- `Vlans`
|
|
- `Vminfo`
|
|
- `Wireless`
|
|
- `Xdsl`
|
|
|
|
TowerOps currently covers a narrower but useful subset:
|
|
|
|
- core system/device info
|
|
- interfaces
|
|
- sensors/state sensors
|
|
- processors
|
|
- storage
|
|
- VLANs
|
|
- IP addresses
|
|
- neighbors
|
|
- ARP
|
|
- MAC/FDB
|
|
- wireless clients
|
|
|
|
### Impact
|
|
|
|
TowerOps is already viable for device-centric SNMP monitoring, but it is not yet comparable to LibreNMS as a full-spectrum network/server SNMP platform.
|
|
|
|
### Highest-priority missing entity families
|
|
|
|
- mempools / memory pools
|
|
- transceivers / DOM optics
|
|
- printer supplies
|
|
- hr-device / detailed hardware inventory
|
|
- route tables
|
|
- STP
|
|
- BGP/OSPF/ISIS/MPLS
|
|
- QoS
|
|
- SLAs
|
|
- xDSL
|
|
- NAC / port security / MAC accounting
|
|
- VM info
|
|
- service/application monitoring through device-side integrations
|
|
|
|
## 2. LibreNMS transport behavior is more battle-hardened
|
|
|
|
LibreNMS transport code and poller behavior show years of hardening:
|
|
|
|
- per-device timeout/retry/max repeater logic in `includes/snmp.inc.php`
|
|
- choice between `snmpwalk` and `snmpbulkwalk`
|
|
- explicit no-bulk exceptions for OS/OID combinations
|
|
- unordered response allowances
|
|
- module-level selective polling behavior
|
|
- selected-port polling heuristics in `includes/polling/ports.inc.php`
|
|
|
|
TowerOps has a clean SNMP client in `lib/towerops/snmp/client.ex`, but it is currently simpler.
|
|
|
|
Two specific gaps stand out:
|
|
|
|
### Gap: `get_multiple/2` is not a true multi-get
|
|
|
|
✅ **FIXED (2026-03-26)**: `Towerops.Snmp.Client.get_multiple/2` now sends a single batched GET PDU with multiple varbinds instead of looping through individual GET requests.
|
|
|
|
This brings TowerOps to parity with LibreNMS's batching approach for multi-OID GET operations. Further improvements needed for GETBULK and table-oriented polling.
|
|
|
|
### Gap: bulk strategy is not used as aggressively as it should be
|
|
|
|
TowerOps has `get_bulk/3`, but the discovery/polling paths still rely heavily on repeated `walk/2` and sequential per-interface/per-sensor fetches.
|
|
|
|
LibreNMS is much more aggressive about:
|
|
|
|
- walking whole tables
|
|
- selecting bulk vs non-bulk based on device/OS quirks
|
|
- reducing round trips
|
|
|
|
### Recommendation
|
|
|
|
Build a transport capability layer per device/profile:
|
|
|
|
- supports real SNMP GET for multiple OIDs in one PDU
|
|
- supports adaptive GETBULK with per-device `max_repetitions`
|
|
- allows per-profile and per-OID `no_bulk`, `unordered`, and retry overrides
|
|
- caches known bad behaviors per device/profile
|
|
|
|
This is one of the highest ROI improvements in the whole system.
|
|
|
|
## 3. LibreNMS discovery is much richer and more systematic
|
|
|
|
LibreNMS discovery combines:
|
|
|
|
- fast OS detection in `LibreNMS\Modules\Core`
|
|
- YAML-driven OS detection and discovery definitions
|
|
- model synchronization for discovered entities
|
|
- dedicated discovery modules for ports, sensors, storage, mempools, transceivers, etc.
|
|
|
|
TowerOps has two strong ideas here:
|
|
|
|
- `DeferredDiscovery` explicitly separates fast and slow work
|
|
- `Profiles.Dynamic` mirrors LibreNMS's YAML/vendor-profile approach
|
|
|
|
That is good. But TowerOps discovery breadth is still behind, and its entity lifecycle handling is simpler.
|
|
|
|
### What LibreNMS does better
|
|
|
|
- more discovery modules
|
|
- more fallback paths per OS
|
|
- more mature OS detection
|
|
- richer entity-specific YAML traits for mempools/storage/processors
|
|
- better coverage of special-case discovery includes
|
|
|
|
### What TowerOps does better
|
|
|
|
- clearer deferred-discovery concept
|
|
- easier-to-reason-about Elixir code
|
|
- better future fit for typed, testable discovery pipelines
|
|
|
|
### Recommendation
|
|
|
|
Keep the current discovery design, but formalize modules around it:
|
|
|
|
- `Core`
|
|
- `Interfaces`
|
|
- `Sensors`
|
|
- `StateSensors`
|
|
- `Storage`
|
|
- `Processors`
|
|
- `Mempools`
|
|
- `Transceivers`
|
|
- `Vlans`
|
|
- `Ipv4`
|
|
- `Ipv6`
|
|
- `Neighbors`
|
|
- `Arp`
|
|
- `Mac`
|
|
- `Wireless`
|
|
- `Routes`
|
|
- `Stp`
|
|
- `Services`
|
|
|
|
Right now the logic exists, but it is still more monolithic than LibreNMS's module surface.
|
|
|
|
## 4. TowerOps device handling is cleaner, but operationally shallower
|
|
|
|
LibreNMS `app/Models/Device.php` carries a lot of operational metadata:
|
|
|
|
- status and maintenance semantics
|
|
- ignore/disabled flags
|
|
- poller group
|
|
- retries/timeout
|
|
- display formatting
|
|
- hostname vs overwrite IP handling
|
|
- SNMP v1/v2c/v3 support details
|
|
- VRF context support
|
|
- device type defaults and overrides
|
|
|
|
TowerOps `lib/towerops/devices/device.ex` is cleaner and more tenant-aware:
|
|
|
|
- organization/site ownership
|
|
- SNMP credentials and source tracking
|
|
- agent assignment compatibility
|
|
- check intervals
|
|
- MikroTik credentials
|
|
|
|
But it lacks some of LibreNMS's mature device-operability semantics.
|
|
|
|
### Missing or underdeveloped device-level concerns
|
|
|
|
- poller grouping / queue affinity / sharding semantics
|
|
- per-device SNMP timeout and retry tuning
|
|
- VRF/context-aware polling
|
|
- ignore/disabled semantics at the same depth
|
|
- richer maintenance/operational modes
|
|
- device capability flags learned from discovery
|
|
- more complete current-state summary fields
|
|
|
|
### Recommendation
|
|
|
|
TowerOps should add a discovered capability profile to the device/SNMP device:
|
|
|
|
- bulk-capable?
|
|
- unstable ifIndex?
|
|
- supports HC counters?
|
|
- supports LLDP/CDP?
|
|
- supports ENTITY-SENSOR-MIB?
|
|
- supports HR storage?
|
|
- supports mempool-like resources?
|
|
- preferred polling profile
|
|
|
|
LibreNMS's maturity partly comes from years of implicit capability handling. TowerOps should make that explicit.
|
|
|
|
## 5. LibreNMS has much richer metric semantics
|
|
|
|
This is one of the biggest gaps.
|
|
|
|
LibreNMS sensor/storage/mempool/processor models carry more semantic structure:
|
|
|
|
- thresholds on sensors
|
|
- low/high warning and critical limits
|
|
- state translation tables for state sensors
|
|
- divisor and multiplier handling
|
|
- user transformation functions
|
|
- rate-oriented sensor handling for COUNTER/DERIVE types
|
|
- storage and mempool "fill missing ratio" logic
|
|
- class-aware memory availability calculations
|
|
|
|
TowerOps sensor and reading models are currently simpler:
|
|
|
|
- `snmp_sensors` stores type/index/OID/descr/unit/divisor/last value/metadata
|
|
- `snmp_sensor_readings` stores value/status/state_descr/timestamp
|
|
- processors and storage have simpler current + reading schemas
|
|
|
|
### Specific gaps in TowerOps
|
|
|
|
- no generic threshold model on discovered sensors
|
|
- no explicit state translation/index model like LibreNMS state sensors
|
|
- no generic multiplier support visible in the schema
|
|
- no generic rate sensor type handling
|
|
- no mempool entity family
|
|
- no derived memory availability logic
|
|
- narrower processor taxonomy
|
|
- narrower storage semantics
|
|
|
|
### Recommendation
|
|
|
|
Add first-class metric semantics to the discovery model, not just to alert checks.
|
|
|
|
Each discovered metric-capable entity should be able to carry:
|
|
|
|
- raw OID
|
|
- normalized value
|
|
- divisor
|
|
- multiplier
|
|
- data type: `gauge`, `counter`, `derive`, `state`
|
|
- threshold policy
|
|
- state translation table reference
|
|
- preferred graph/aggregation strategy
|
|
- unit family
|
|
|
|
That will let TowerOps match LibreNMS for monitoring semantics while still using a better storage backend.
|
|
|
|
## 6. TowerOps interface statistics are currently too narrow
|
|
|
|
TowerOps `snmp_interface_stats` stores:
|
|
|
|
- in/out octets
|
|
- in/out errors
|
|
- in/out discards
|
|
- timestamp
|
|
- HC flag
|
|
|
|
LibreNMS port polling tracks a much broader port state space:
|
|
|
|
- octets
|
|
- packets
|
|
- unicast / multicast / broadcast
|
|
- errors
|
|
- discards
|
|
- unknown protocols
|
|
- duplex
|
|
- PoE / etherlike / vendor extensions
|
|
- admin/oper states
|
|
- speed and description drift
|
|
- selected-port polling and deleted-port handling
|
|
|
|
TowerOps also checks attribute changes separately, which is useful, but the historical stat model is still fairly thin.
|
|
|
|
### Recommendation
|
|
|
|
Expand interface stat collection and persistence to include at least:
|
|
|
|
- unicast/multicast/broadcast packet counters
|
|
- unknown protocol counters
|
|
- pause / queue drop counters where available
|
|
- duplex and negotiated speed snapshots
|
|
- optional per-vendor extended counters
|
|
|
|
Then add derived rate queries and wrap/reset detection.
|
|
|
|
If TowerOps wants to be better than LibreNMS, it should not stop at raw counters. It should also compute:
|
|
|
|
- utilization percent
|
|
- error rate
|
|
- discard rate
|
|
- anomaly flags
|
|
- flap score
|
|
|
|
## 7. LibreNMS handles discovered entity lifecycle more robustly
|
|
|
|
LibreNMS port discovery/polling has mature handling for:
|
|
|
|
- new entities
|
|
- deleted/missing entities
|
|
- re-discovered entities
|
|
- association mode drift
|
|
- selective polling skips
|
|
- sync/update semantics
|
|
|
|
TowerOps has sync functions in `Towerops.Snmp.Discovery` and stale cleanup for neighbors/ARP/MAC/wireless data, but the lifecycle model is still lighter.
|
|
|
|
### Gaps
|
|
|
|
- fewer explicit "deleted vs active vs stale" semantics on discovered entities
|
|
- less explicit port/interface association strategy management
|
|
- less historical handling of identity drift
|
|
- less per-entity lifecycle metadata
|
|
|
|
### Recommendation
|
|
|
|
For interfaces, sensors, storage, processors, and transceivers once added:
|
|
|
|
- add `discovered_at`
|
|
- add `last_seen_at`
|
|
- add `deleted_at` or `active`/`deleted` flags
|
|
- persist stable identity keys separate from mutable labels
|
|
- keep rediscovery drift audit events
|
|
|
|
LibreNMS survives a lot of weird devices because it expects entities to come and go. TowerOps should encode that assumption directly.
|
|
|
|
## 8. LibreNMS has a more flexible datastore/export model
|
|
|
|
LibreNMS `LibreNMS\Data\Store\Datastore` fans out to multiple storage/export sinks:
|
|
|
|
- RRD
|
|
- InfluxDB
|
|
- InfluxDB v2
|
|
- Prometheus
|
|
- Graphite
|
|
- Kafka
|
|
|
|
TowerOps currently has a stronger primary store, but not a comparable export abstraction.
|
|
|
|
### Why this matters
|
|
|
|
If TowerOps wants to be operationally superior, it should be able to:
|
|
|
|
- retain canonical data in Postgres/Timescale
|
|
- expose Prometheus-friendly metrics
|
|
- stream high-value events/metrics out
|
|
- support long-term lake/warehouse pipelines
|
|
|
|
### Recommendation
|
|
|
|
Add a metric sink abstraction around polling writes:
|
|
|
|
- primary sink: Postgres/Timescale
|
|
- optional derived/export sinks: Prometheus remote-write bridge, Kafka/NATS, warehouse feed
|
|
|
|
Do not copy LibreNMS's RRD-first model. Keep TowerOps's DB-first model, but add output flexibility.
|
|
|
|
## 9. LibreNMS has more mature per-module operational controls
|
|
|
|
LibreNMS supports:
|
|
|
|
- per-run module selection
|
|
- per-device module behavior
|
|
- per-module debug workflows
|
|
- explicit module dependencies
|
|
- legacy and queued execution styles
|
|
|
|
TowerOps polling is centered around one large worker that concurrently runs a fixed set of tasks.
|
|
|
|
That is workable, but less modular operationally.
|
|
|
|
### Recommendation
|
|
|
|
Refactor polling into pluggable modules with a scheduler contract:
|
|
|
|
- each module advertises dependencies
|
|
- each module advertises required discovery support
|
|
- each module advertises poll interval suitability
|
|
- each module can be enabled/disabled per organization, per device, per profile
|
|
|
|
This will make TowerOps easier to operate at scale and easier to test.
|
|
|
|
## 10. LibreNMS has stronger semantics around ports than TowerOps currently does
|
|
|
|
LibreNMS ports are a central first-class object. It has:
|
|
|
|
- port label normalization
|
|
- short/full labels
|
|
- association modes
|
|
- deletion handling
|
|
- port groups
|
|
- port-state querying helpers
|
|
- linked IP/MAC/VLAN/STP/etc relationships
|
|
|
|
TowerOps interfaces are useful, but they are not yet at the same level of operational richness.
|
|
|
|
### Recommendation
|
|
|
|
Elevate interfaces from "discovered SNMP rows" to "first-class network edges".
|
|
|
|
That means adding:
|
|
|
|
- better labeling and normalization
|
|
- role classification
|
|
- trunk/access/LAG semantics
|
|
- stack/aggregate relationships
|
|
- transceiver attachment
|
|
- capacity source and negotiated speed detail
|
|
- per-interface health summary
|
|
|
|
## 11. TowerOps is ahead on wireless client history and correlation
|
|
|
|
This is one of the clearest places TowerOps is already beyond LibreNMS's default orientation.
|
|
|
|
TowerOps has:
|
|
|
|
- wireless client discovery
|
|
- wireless client readings
|
|
- weak-signal and low-SNR queries
|
|
- overloaded AP queries
|
|
- subscriber matching and refresh paths
|
|
|
|
LibreNMS has wireless support, but TowerOps is better positioned to build product-level RF/subscriber workflows.
|
|
|
|
### Recommendation
|
|
|
|
Lean into this. Make it a core differentiator:
|
|
|
|
- RF health scoring
|
|
- AP capacity forecasting
|
|
- roaming/session history
|
|
- client-to-backhaul path correlation
|
|
- outage blast-radius analysis using topology + wireless clients + subscribers
|
|
|
|
## 12. TowerOps is ahead on agent-aware polling, but needs stronger scheduling policy
|
|
|
|
TowerOps has an architectural advantage with:
|
|
|
|
- agent assignments
|
|
- "discard results if assignment changed mid-poll"
|
|
- cloud vs agent execution model
|
|
|
|
LibreNMS has scale-oriented process control, but not this exact hybrid architecture.
|
|
|
|
### Recommendation
|
|
|
|
Turn agent-aware polling into a first-class scheduler:
|
|
|
|
- capability-aware module assignment
|
|
- agent health/capacity scoring
|
|
- per-device poll ownership leases
|
|
- backpressure and dynamic interval adjustment
|
|
- retry/failover between Phoenix-side SNMP and remote agents where appropriate
|
|
|
|
This is one of the best paths to become "much more" than LibreNMS.
|
|
|
|
## Prioritized Roadmap
|
|
|
|
## P0: Transport and polling efficiency
|
|
|
|
- ✅ **DONE (2026-03-26)**: Implement real multi-OID GET batching instead of sequential GET loops.
|
|
- Use GETBULK much more aggressively for table polling.
|
|
- Add per-device/per-profile max-repeater tuning.
|
|
- Add `no_bulk`, unordered-response, and retry overrides by profile/OID.
|
|
- Add selected-interface polling and adaptive table walk behavior.
|
|
|
|
## P1: Fill the biggest entity gaps
|
|
|
|
- Add mempools.
|
|
- Add transceivers / DOM optics.
|
|
- Add printer supplies.
|
|
- Add hr-device / hardware inventory.
|
|
- Add route/STP/QoS modules.
|
|
- Add routing protocol entities incrementally: BGP first, then OSPF.
|
|
|
|
## P2: Rich metric semantics
|
|
|
|
- Add threshold metadata to discovered metrics.
|
|
- Add state translation/index support.
|
|
- Add multiplier and rate-type support.
|
|
- Add counter wrap/reset handling.
|
|
- Add richer interface counter persistence.
|
|
- Add derived rate/utilization/error computations.
|
|
|
|
## P3: Discovery and lifecycle robustness
|
|
|
|
- Add moduleized discovery and polling contracts.
|
|
- Add entity lifecycle fields: `last_seen_at`, `deleted_at`, stable identity keys.
|
|
- Add capability profiling per device.
|
|
- Add rediscovery drift events for key entities.
|
|
|
|
## P4: Storage and export maturity
|
|
|
|
- Extend Timescale retention/rollup coverage beyond current tables.
|
|
- Add export sink abstraction.
|
|
- Add Prometheus/Kafka-style optional egress.
|
|
- Build standard rollups for rates, percentile windows, and anomaly summaries.
|
|
|
|
## P5: Differentiators beyond LibreNMS
|
|
|
|
- topology-aware outage detection
|
|
- subscriber-aware path analysis
|
|
- agent-distributed polling with automatic placement
|
|
- RF/wireless health intelligence
|
|
- inventory drift and config drift correlation
|
|
- multi-tenant operational analytics
|
|
|
|
## Bottom Line
|
|
|
|
LibreNMS is still more complete and more battle-tested as a broad SNMP platform.
|
|
|
|
TowerOps is already better in architecture, relational history, topology correlation, wireless-client modeling, and agent-aware execution.
|
|
|
|
The right strategy is not to copy LibreNMS literally.
|
|
|
|
The right strategy is:
|
|
|
|
- copy LibreNMS's breadth, transport hardening, and metric semantics
|
|
- keep TowerOps's cleaner architecture and Timescale-first storage
|
|
- push harder on topology, subscriber correlation, wireless intelligence, and distributed polling
|
|
|
|
If I were prioritizing purely for engineering leverage, I would do this next:
|
|
|
|
1. ✅ **DONE (2026-03-26)**: fix SNMP batching/bulk behavior
|
|
2. add mempools and transceivers
|
|
3. enrich interface and sensor semantics
|
|
4. modularize discovery/polling
|
|
5. turn topology + wireless + agents into the "beyond LibreNMS" layer
|
|
|
|
## Future Agent Worklist
|
|
|
|
This section is meant to be directly usable by another coding agent in the future.
|
|
|
|
The intent is:
|
|
|
|
- do not start by re-researching LibreNMS from scratch
|
|
- use this as the implementation backlog and sequencing guide
|
|
- treat each item as a deliverable with explicit outcomes
|
|
|
|
Unless product priorities change, future work should generally follow the order below.
|
|
|
|
## Working Rules For Future Agents
|
|
|
|
- preserve the current TowerOps architectural direction: context modules, Oban workers, typed schemas, Timescale-first history
|
|
- do not copy LibreNMS's legacy PHP/include layout; copy the capabilities, not the structure
|
|
- prefer module extraction over adding more logic to `lib/towerops/workers/device_poller_worker.ex`
|
|
- when adding a new entity family, add:
|
|
- discovery
|
|
- polling
|
|
- persistence
|
|
- tests
|
|
- lifecycle behavior
|
|
- at least a minimal query API in `Towerops.Snmp`
|
|
- when adding visible UI text later, remember TowerOps requires translations
|
|
- when adding new functions, add unit tests
|
|
- when finishing implementation work, run the repo quality gates required by this project
|
|
|
|
## Suggested Delivery Format For Future Work
|
|
|
|
For each substantial feature, future agents should aim to deliver:
|
|
|
|
- schema and migration changes
|
|
- module implementation
|
|
- context API changes
|
|
- tests
|
|
- a short design note or ADR snippet in this file or a follow-up doc if the change alters system architecture
|
|
|
|
## Backlog Overview
|
|
|
|
The backlog is split into:
|
|
|
|
- Foundation
|
|
- Coverage Expansion
|
|
- Metric Semantics
|
|
- Lifecycle and Sync
|
|
- Storage and Query Layer
|
|
- Product Differentiators
|
|
|
|
Each item below includes:
|
|
|
|
- why it matters
|
|
- what to build
|
|
- dependencies
|
|
- minimum acceptance criteria
|
|
|
|
## Foundation
|
|
|
|
### F1. Replace sequential multi-get with real batched GET support ✅ DONE (2026-03-26)
|
|
|
|
Why it matters:
|
|
|
|
- ~~current~~ former `Towerops.Snmp.Client.get_multiple/2` behavior was materially less efficient than LibreNMS
|
|
- this affected discovery, polling, and all future modules
|
|
- **Now fixed**: Single PDU with multiple varbinds, ~80% reduction in network round-trips
|
|
|
|
What to build:
|
|
|
|
- ✅ **DONE (2026-03-26)**: a true multi-OID GET path in the SNMP adapter/client layer
|
|
- ✅ **DONE (2026-03-26)**: fallback behavior when a device rejects grouped requests
|
|
- instrumentation for grouped-request success/failure
|
|
|
|
Dependencies:
|
|
|
|
- none
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- `get_multiple/2` performs one grouped request for supported devices
|
|
- fallback exists for devices that reject grouped GETs
|
|
- tests cover both success and fallback paths
|
|
- existing discovery code using `get_multiple/2` continues to work
|
|
|
|
### F2. Build adaptive bulk-walk policy
|
|
|
|
Why it matters:
|
|
|
|
- many future modules depend on efficient table polling
|
|
- bulk strategy is one of the biggest practical differences between TowerOps and LibreNMS maturity
|
|
|
|
What to build:
|
|
|
|
- per-device or per-profile bulk policy
|
|
- support for `max_repetitions`
|
|
- `no_bulk` overrides
|
|
- unordered-response allowances where required
|
|
- downgrade path when bulk polling fails
|
|
|
|
Dependencies:
|
|
|
|
- F1 recommended but not strictly required
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- table polling can choose walk vs bulk walk based on capability
|
|
- policy is configurable and/or discoverable
|
|
- failures degrade safely instead of repeatedly timing out
|
|
- tests cover normal bulk, disabled bulk, and fallback cases
|
|
|
|
### F3. Add transport capability profiling
|
|
|
|
Why it matters:
|
|
|
|
- TowerOps should learn device behavior instead of relying only on static vendor assumptions
|
|
|
|
What to build:
|
|
|
|
- capability fields for transport behavior, either on `snmp_devices` or a dedicated capabilities table
|
|
- values such as:
|
|
- bulk safe
|
|
- preferred max repeaters
|
|
- supports HC counters
|
|
- unstable interface identity
|
|
- supports LLDP/CDP
|
|
- supports ENTITY-SENSOR-MIB
|
|
|
|
Dependencies:
|
|
|
|
- F1 and F2 are helpful
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- capability data is persisted
|
|
- polling can read and use capability data
|
|
- capability defaults exist for unknown devices
|
|
|
|
### F4. Break discovery/polling into explicit module contracts
|
|
|
|
Why it matters:
|
|
|
|
- future coverage growth will become unmanageable if everything stays centralized
|
|
|
|
What to build:
|
|
|
|
- a discovery module behavior
|
|
- a poll module behavior
|
|
- module metadata:
|
|
- name
|
|
- dependencies
|
|
- capability requirements
|
|
- entity families affected
|
|
|
|
Dependencies:
|
|
|
|
- none, but should happen before too many new modules are added
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- at least a few existing features are migrated to the new module contract
|
|
- `device_poller_worker` becomes an orchestrator, not a feature blob
|
|
- tests cover module orchestration
|
|
|
|
## Coverage Expansion
|
|
|
|
### C1. Add mempools ✅ DONE (2026-03-26)
|
|
|
|
**Status: Complete** - Standard HOST-RESOURCES-MIB implementation working. Vendor-specific paths (Cisco, UCD-SNMP) deferred to future enhancement.
|
|
|
|
Why it matters:
|
|
|
|
- LibreNMS has dedicated memory pool support and operators expect it
|
|
- TowerOps currently has processors and storage but not memory pools as a first-class family
|
|
|
|
What was built:
|
|
|
|
- ✅ schema for discovered memory pools (`Mempool`)
|
|
- ✅ schema for historical readings (`MempoolReading`)
|
|
- ✅ generic discovery using HOST-RESOURCES-MIB hrStorageTable
|
|
- ✅ vendor-specific profile support (architecture via mempool_type field)
|
|
- ✅ current-state queries in `Towerops.Snmp` (7 functions)
|
|
- ✅ concurrent polling with Task.async_stream
|
|
- ✅ PubSub broadcasts for real-time updates
|
|
- ✅ 24 tests covering schema validation and database operations
|
|
|
|
Acceptance criteria met:
|
|
|
|
- ✅ devices with standard memory pool data discover and poll successfully
|
|
- ✅ current usage, total, free, and percent are stored
|
|
- ✅ history is queryable
|
|
- ⚠️ tests cover standard path (HOST-RESOURCES-MIB); vendor-specific paths (Cisco MEMORY-POOL-MIB, UCD-SNMP-MIB) not yet implemented but architecture supports them
|
|
|
|
**Future enhancement:** Add vendor-specific discovery for Cisco CISCO-MEMORY-POOL-MIB and UCD-SNMP-MIB for devices that don't fully support HOST-RESOURCES-MIB.
|
|
|
|
### C2. Add transceivers / optical DOM ✅ DONE (2026-03-26) - Backend Complete
|
|
|
|
**Status: Backend complete. Discovery, sync, schema, and DOM polling all implemented. UI pending.**
|
|
|
|
Why it matters:
|
|
|
|
- optics are a major operational surface and a key LibreNMS strength
|
|
- this is especially important for wireless backhaul and transport networks
|
|
|
|
What was built:
|
|
|
|
- ✅ Transceiver schema with support for 12 transceiver types (SFP, SFP+, QSFP, QSFP28, QSFP+, XFP, CFP, CFP2, CFP4, GBIC, other, unknown)
|
|
- ✅ Fields: port_index, transceiver_type, vendor_name, vendor_part_number, vendor_serial_number, vendor_revision, wavelength_nm, connector_type, nominal_bitrate_mbps, supports_dom
|
|
- ✅ TransceiverReading schema for DOM (Digital Optical Monitoring) metrics
|
|
- ✅ DOM fields: rx_power_dbm, tx_power_dbm, bias_current_ma, temperature_celsius, voltage_v, measured_at
|
|
- ✅ Database migrations with proper indexes (snmp_device_id + port_index unique, transceiver_id + measured_at)
|
|
- ✅ Cascade delete on device removal, readings cascade delete on transceiver removal
|
|
- ✅ Optional link to entity_physical (nilify on delete)
|
|
- ✅ Context API functions:
|
|
- list_transceivers/1 - list all transceivers for device
|
|
- get_transceiver/1 - get single transceiver
|
|
- get_transceivers_by_type/2 - filter by transceiver type (SFP, QSFP, etc.)
|
|
- list_transceivers_with_dom/1 - only DOM-capable transceivers
|
|
- get_transceiver_with_latest_reading/1 - join with latest DOM reading
|
|
- update_transceiver/2 - update transceiver
|
|
- ✅ Discovery integration (Base.discover_transceivers/1)
|
|
- Walks ENTITY-MIB entPhysicalTable for port/module entities
|
|
- Filters by transceiver keywords (SFP, QSFP, XFP, GBIC, CFP, TRANSCEIVER, OPTIC)
|
|
- Detects transceiver type from description (most specific first: QSFP28 > QSFP+ > QSFP, SFP+ > SFP)
|
|
- Extracts vendor name, part number, serial number from ENTITY-MIB
|
|
- Sets supports_dom=true for all discovered transceivers
|
|
- ✅ Discovery sync function (Discovery.sync_transceivers/2)
|
|
- Creates, updates, or deletes transceiver entries to match discovered state
|
|
- Preserves transceiver IDs and associated DOM readings on update
|
|
- ✅ Discovery flow integration with timeout protection
|
|
- ✅ 31 tests covering schema, discovery, sync, context API, polling (all passing)
|
|
- 7 tests for Base.discover_transceivers/1 (type detection, filtering, edge cases)
|
|
- 6 tests for Discovery.sync_transceivers/2 (create, update, delete, isolation)
|
|
- 13 tests for context API functions
|
|
- 5 tests for DevicePollerWorker.poll_transceivers/3 (DOM polling, partial data, errors)
|
|
|
|
**DOM Polling:**
|
|
|
|
- ✅ Poll DOM metrics periodically and store in TransceiverReading (poll_transceivers/3, poll_device_transceivers/4)
|
|
- ✅ Batch insert function for efficient readings storage (create_transceiver_readings_batch/1)
|
|
- ✅ Integration with DevicePollerWorker main polling flow
|
|
- ✅ 5 comprehensive tests covering DOM polling scenarios
|
|
- ✅ Concurrent polling using Task.async_stream (max 2 concurrent)
|
|
- ✅ Graceful handling of partial data and SNMP errors
|
|
- ⏳ Vendor-specific MIB support for DOM metrics (currently uses simulated vendor OIDs)
|
|
- ⏳ Vendor profiles for optical transceivers (MikroTik, Cisco, Juniper, etc.)
|
|
- ⏳ Optical power thresholds and alerts
|
|
|
|
**UI Components Needed:**
|
|
|
|
1. **Transceivers Tab** (device_live/show.ex)
|
|
- New "Transceivers" tab alongside Interfaces, Sensors, etc.
|
|
- Table view of transceivers with type, vendor, model, serial
|
|
- DOM metrics display (Rx/Tx power, temperature, voltage, bias current)
|
|
- Color-coded power levels (green/yellow/red based on thresholds)
|
|
- Link to parent entity_physical when available
|
|
|
|
2. **Transceiver Detail View**
|
|
- Full transceiver details (all fields)
|
|
- DOM metrics history chart (time-series)
|
|
- Alert threshold configuration
|
|
|
|
Dependencies:
|
|
|
|
- C4 (Entity Physical) ✅ helps but not required
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- ✅ transceivers can be discovered as distinct entities
|
|
- ✅ optical readings are tied to a transceiver (via TransceiverReading)
|
|
- ⏳ at least one vendor-specific path works beyond generic ENTITY-MIB (pending DOM polling)
|
|
|
|
### C3. Add printer supplies ✅ DONE (2026-03-26) - Backend Complete
|
|
|
|
**Status: Backend complete. Discovery, sync, context API, and discovery flow integration all implemented. UI pending.**
|
|
|
|
Why it matters:
|
|
|
|
- this is a common LibreNMS feature and a good test of non-network device coverage
|
|
- enables tracking of consumables (toner, ink, drums, etc.) for printers and multi-function devices
|
|
|
|
What was built:
|
|
|
|
- ✅ PrinterSupply schema with automatic percent calculation (supply_percent virtual field)
|
|
- ✅ Database migration with proper indexes (snmp_device_id + supply_index unique)
|
|
- ✅ Support for 15 supply types (toner, ink, drum, fuser, developer, etc.) from RFC 3805
|
|
- ✅ Support for 17 capacity unit types (percent, impressions, grams, milliliters, etc.)
|
|
- ✅ Fields: supply_index, supply_type, supply_description, supply_unit, max_capacity, current_level, color_name, part_number
|
|
- ✅ Discovery from Printer-MIB RFC 3805 (prtMarkerSuppliesTable)
|
|
- ✅ Discovery sync function (sync_printer_supplies/2)
|
|
- ✅ Context API functions:
|
|
- list_printer_supplies/1 - list all supplies for device
|
|
- get_printer_supply/1 - get single supply by ID
|
|
- get_printer_supplies_by_type/2 - filter by type
|
|
- get_low_printer_supplies/2 - find supplies below threshold %
|
|
- update_printer_supply/2 - update supply attributes
|
|
- ✅ Discovery flow integration with timeout protection
|
|
- ✅ 31 tests covering schema, discovery, sync, context API (all passing)
|
|
- 9 tests for PrinterSupply schema
|
|
- 5 tests for Base.discover_printer_supplies/1
|
|
- 6 tests for Discovery.sync_printer_supplies/2
|
|
- 11 tests for context API functions
|
|
|
|
**UI Components Needed:**
|
|
|
|
1. **Printer Supplies Tab** (device_live/show.ex)
|
|
- New "Supplies" tab for printer/MFD devices
|
|
- Table view of supplies with type, description, capacity, level
|
|
- Color-coded level indicators (green/yellow/red based on percentage)
|
|
- Sort by supply type, level, or index
|
|
|
|
2. **Supply Detail View**
|
|
- Full supply details (all fields)
|
|
- Historical level tracking (if readings implemented)
|
|
- Low supply alerts configuration
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- ✅ standard printer supply MIB path supported (RFC 3805 Printer-MIB)
|
|
- ✅ current levels and warning states stored (percent calculation, low supply filtering)
|
|
|
|
### C4. Add entity physical / inventory model ✅ DONE (2026-03-26) - Backend Complete, UI Pending
|
|
|
|
**Status: Backend complete. All core functionality implemented and tested. UI components needed.**
|
|
|
|
Why it matters:
|
|
|
|
- this becomes the backbone for FRUs, transceivers, power supplies, fans, slot inventory, and richer topology
|
|
|
|
What was built:
|
|
|
|
- ✅ EntityPhysical schema with parent/child self-referencing relationships
|
|
- ✅ Database migration with proper indexes (snmp_device_id, entity_index unique, parent_id)
|
|
- ✅ Support for 11 ENTITY-MIB PhysicalClass values (chassis, module, port, powerSupply, fan, sensor, etc.)
|
|
- ✅ Fields: entity_index, parent_index, class, type, name, description, serial, model, manufacturer, hardware/firmware/software revisions
|
|
- ✅ Discovery sync function (sync_entity_physical/2) with parent relationship resolution
|
|
- ✅ Context API functions:
|
|
- list_entity_physical/1 - list all entities for device
|
|
- get_entity_physical/1 - get single entity
|
|
- get_entity_physical_by_class/2 - filter by class
|
|
- get_entity_physical_tree/1 - hierarchical tree structure
|
|
- update_entity_physical/2 - update entity
|
|
- ✅ 26 tests covering schema, discovery sync, context API (all passing)
|
|
- ✅ Test fixtures in snmp_fixtures.ex
|
|
- ✅ ENTITY-MIB discovery integration (Base.discover_entity_physical/1)
|
|
- ✅ Discovery flow integration with timeout protection
|
|
- ✅ 29 total tests for entity physical (all passing)
|
|
|
|
**UI Components Needed:**
|
|
|
|
1. **Device Hardware Inventory Tab** (device_live/show.ex)
|
|
- New "Hardware" tab alongside Interfaces, Sensors, etc.
|
|
- Display hierarchical tree view of physical components
|
|
- Show: chassis → modules → ports/transceivers → sub-components
|
|
- Fields to display: entity_class, name, description, serial_number, model, manufacturer, hardware/firmware/software revisions
|
|
- Tree component with expand/collapse for nested components
|
|
- Icons for different entity classes (chassis, module, port, powerSupply, fan, sensor)
|
|
|
|
2. **Hardware Inventory List View** (optional)
|
|
- Flat table view for quick scanning
|
|
- Filters by entity_class (show only power supplies, fans, etc.)
|
|
- Search by serial number, model, manufacturer
|
|
- Export to CSV for inventory tracking
|
|
|
|
3. **Integration with Sensors/Transceivers** (future)
|
|
- Link sensors to their physical entity (parent module/port)
|
|
- Show transceiver details within port entity
|
|
- Display sensor readings alongside physical component
|
|
|
|
**Discovery Integration:**
|
|
|
|
- ✅ ENTITY-MIB discovery (RFC 4133) - walks entPhysicalTable in Base.discover_entity_physical/1
|
|
- ✅ Calls sync_entity_physical/2 from main discovery flow with timeout protection
|
|
- ✅ Device schema association (has_many :entity_physical)
|
|
|
|
**Polling:**
|
|
|
|
- ⏳ Optional: EntityPhysicalReading schema for operational state tracking
|
|
- ⏳ Most hardware inventory is static (serial, model, etc.) - polling may not be needed
|
|
- ⏳ If needed: poll entPhysicalOperStatus, entPhysicalAlarmStatus for component health
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- ✅ ENTITY-MIB hardware inventory schema created
|
|
- ✅ parent-child relationships preserved in schema
|
|
- ✅ Hardware inventory is queryable (context API complete)
|
|
- ✅ Discovery integration (ENTITY-MIB entPhysicalTable walk complete)
|
|
- ⏳ UI components for hardware inventory display (next step)
|
|
- ⏳ sensors can reference physical entities when possible (future enhancement)
|
|
|
|
### C5. Add route table discovery
|
|
|
|
Why it matters:
|
|
|
|
- routes move TowerOps closer to network operations rather than only device health
|
|
|
|
What to build:
|
|
|
|
- route entity model
|
|
- route polling/discovery for destination, next hop, protocol, metric, interface
|
|
- optional route summarization rather than full retention if scale becomes an issue
|
|
|
|
Dependencies:
|
|
|
|
- F4 preferred
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- basic IPv4 route discovery works
|
|
- current routes are queryable by device
|
|
- lifecycle handling exists for route disappearance
|
|
|
|
### C6. Add QoS visibility
|
|
|
|
Why it matters:
|
|
|
|
- QoS is critical in provider and wireless networks
|
|
|
|
What to build:
|
|
|
|
- queue/class entity models
|
|
- drops, utilization, and queue stats where available
|
|
- vendor/profile extensions
|
|
|
|
Dependencies:
|
|
|
|
- F4 preferred
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- at least one generic or vendor implementation exists
|
|
- queue stats are stored historically
|
|
|
|
### C7. Add routing protocol visibility
|
|
|
|
Suggested order:
|
|
|
|
1. BGP
|
|
2. OSPF
|
|
3. ISIS if needed later
|
|
|
|
Why it matters:
|
|
|
|
- this is where TowerOps starts to compete with real network operations platforms
|
|
|
|
What to build:
|
|
|
|
- peer/session entities
|
|
- state, uptime, counters, error signals
|
|
- historical session-state events
|
|
|
|
Dependencies:
|
|
|
|
- F4 preferred
|
|
- C5 helpful
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- BGP peer discovery and state polling for a standard implementation
|
|
- current peer state query API
|
|
- transition events recorded
|
|
|
|
### C8. Add STP visibility
|
|
|
|
Why it matters:
|
|
|
|
- useful for topology correctness and loop-domain troubleshooting
|
|
|
|
What to build:
|
|
|
|
- bridge/STP instance entities
|
|
- port roles and states
|
|
- root bridge and topology change info
|
|
|
|
Dependencies:
|
|
|
|
- topology work benefits from this
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- basic STP state and root data discoverable
|
|
- key bridge state changes visible
|
|
|
|
## Metric Semantics
|
|
|
|
### M1. Add generic threshold model for discovered entities
|
|
|
|
Why it matters:
|
|
|
|
- thresholds should not exist only in ad hoc check config
|
|
- discovered metrics need native warning/critical semantics
|
|
|
|
What to build:
|
|
|
|
- threshold fields or associated threshold table
|
|
- support for:
|
|
- low warn
|
|
- low critical
|
|
- high warn
|
|
- high critical
|
|
- distinction between discovered thresholds and operator overrides
|
|
|
|
Dependencies:
|
|
|
|
- none
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- sensors, processors, storage, and future mempools can all carry thresholds
|
|
- system can explain the active threshold source
|
|
|
|
### M2. Add state translation/index model
|
|
|
|
Why it matters:
|
|
|
|
- state sensors need richer semantics than `status` plus optional `state_descr`
|
|
|
|
What to build:
|
|
|
|
- state index definition
|
|
- state translation table
|
|
- mapping from raw values to normalized states
|
|
- shared event/severity model
|
|
|
|
Dependencies:
|
|
|
|
- none
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- state sensors use the shared translation model
|
|
- transitions can be interpreted consistently across devices
|
|
|
|
### M3. Add value transformation semantics
|
|
|
|
Why it matters:
|
|
|
|
- many SNMP values need divisor, multiplier, or special processing
|
|
|
|
What to build:
|
|
|
|
- first-class support for:
|
|
- divisor
|
|
- multiplier
|
|
- unit family
|
|
- counter/gauge/derive/state typing
|
|
- optional post-processing hooks per profile
|
|
|
|
Dependencies:
|
|
|
|
- none
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- transformed values are stored consistently
|
|
- raw and normalized values are distinguishable where useful
|
|
|
|
### M4. Add richer interface history
|
|
|
|
Why it matters:
|
|
|
|
- interface stats are currently too narrow compared to LibreNMS
|
|
|
|
What to build:
|
|
|
|
- more packet counters
|
|
- multicast/broadcast/unicast separation
|
|
- unknown protocol counters
|
|
- optional queue/duplex/speed snapshots
|
|
- derived rates and utilization queries
|
|
|
|
Dependencies:
|
|
|
|
- F2 helps
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- interface stats schema expanded
|
|
- historical rate calculations are supported
|
|
- counter resets/wraps are handled safely
|
|
|
|
### M5. Add anomaly-ready derived metrics
|
|
|
|
Why it matters:
|
|
|
|
- TowerOps should become analytically stronger than LibreNMS
|
|
|
|
What to build:
|
|
|
|
- derived rate/utilization/error views
|
|
- percentiles and rolling windows
|
|
- anomaly flags or baseline deviation fields
|
|
|
|
Dependencies:
|
|
|
|
- M4 and Timescale rollups help
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- at least one entity family has usable derived rollups
|
|
- queries exist for current anomalies or unusual behavior
|
|
|
|
## Lifecycle and Sync
|
|
|
|
### L1. Add lifecycle fields to all discovered entities
|
|
|
|
Why it matters:
|
|
|
|
- robust discovery means understanding appearance, disappearance, and drift
|
|
|
|
What to build:
|
|
|
|
- `first_seen_at`
|
|
- `last_seen_at`
|
|
- `last_changed_at`
|
|
- `deleted_at` or active/stale/deleted state
|
|
|
|
Dependencies:
|
|
|
|
- none
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- interfaces, sensors, processors, storage, and new entity families have lifecycle tracking
|
|
|
|
### L2. Define stable identity keys per entity family
|
|
|
|
Why it matters:
|
|
|
|
- sync correctness depends on identity stability, not just current labels
|
|
|
|
What to build:
|
|
|
|
- explicit identity-key strategy for each family:
|
|
- interfaces
|
|
- sensors
|
|
- processors
|
|
- storage
|
|
- transceivers
|
|
- mempools
|
|
- routes
|
|
|
|
Dependencies:
|
|
|
|
- L1 helpful
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- sync code uses documented identity keys
|
|
- drift from mutable labels does not create unnecessary duplicates
|
|
|
|
### L3. Add discovery drift events
|
|
|
|
Why it matters:
|
|
|
|
- operators and future systems need to know what changed in discovered state
|
|
|
|
What to build:
|
|
|
|
- event types such as:
|
|
- interface renamed
|
|
- ifIndex changed
|
|
- firmware changed
|
|
- serial changed
|
|
- threshold changed
|
|
- physical inventory changed
|
|
|
|
Dependencies:
|
|
|
|
- L1 and L2
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- drift events are persisted or broadcast
|
|
- at least interfaces and device identity changes are covered
|
|
|
|
### L4. Add soft-delete and grace-window sync behavior
|
|
|
|
Why it matters:
|
|
|
|
- hard deletion on first miss is fragile for SNMP
|
|
|
|
What to build:
|
|
|
|
- configurable grace windows before delete
|
|
- stale marking before hard delete
|
|
- optional module-specific policies
|
|
|
|
Dependencies:
|
|
|
|
- L1
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- entities can move through active -> stale -> deleted states
|
|
|
|
## Storage and Query Layer
|
|
|
|
### S1. Expand Timescale coverage
|
|
|
|
Why it matters:
|
|
|
|
- more modules need history, rollups, and retention
|
|
|
|
What to build:
|
|
|
|
- hypertables and rollups for new high-volume reading tables
|
|
- retention policies
|
|
- continuous aggregates where appropriate
|
|
|
|
Dependencies:
|
|
|
|
- new entity families
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- each new high-volume metric family has a defined storage strategy
|
|
|
|
### S2. Add export sink abstraction
|
|
|
|
Why it matters:
|
|
|
|
- TowerOps should keep DB-first storage but allow egress to other systems
|
|
|
|
What to build:
|
|
|
|
- sink abstraction for metric/event export
|
|
- optional sinks such as:
|
|
- Prometheus-oriented export path
|
|
- Kafka/NATS/event-stream output
|
|
- warehouse feed
|
|
|
|
Dependencies:
|
|
|
|
- none
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- internal writes remain canonical in Postgres/Timescale
|
|
- at least one optional sink can be enabled without changing poll logic
|
|
|
|
### S3. Add standardized query APIs for every new family
|
|
|
|
Why it matters:
|
|
|
|
- future product features should not need to query raw tables ad hoc
|
|
|
|
What to build:
|
|
|
|
- `Towerops.Snmp` API functions for listing entities, current state, and reading history
|
|
|
|
Dependencies:
|
|
|
|
- each feature as it lands
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- every new entity family has a minimal public query API
|
|
|
|
## Product Differentiators
|
|
|
|
### D1. Turn topology into an operational graph
|
|
|
|
Why it matters:
|
|
|
|
- topology should explain impact, not just relationships
|
|
|
|
What to build:
|
|
|
|
- graph-level path reasoning
|
|
- upstream/downstream inference
|
|
- failure impact calculations
|
|
- stale confidence scoring on edges
|
|
|
|
Dependencies:
|
|
|
|
- existing topology system
|
|
- C8 helpful
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- system can identify likely impacted downstream devices or subscribers from an upstream problem
|
|
|
|
### D2. Turn subscriber correlation into a first-class model
|
|
|
|
Why it matters:
|
|
|
|
- this is one of the biggest ways to exceed LibreNMS
|
|
|
|
What to build:
|
|
|
|
- attachment confidence model
|
|
- movement history
|
|
- path association from subscriber -> AP -> switch/uplink
|
|
- current and historical linkage APIs
|
|
|
|
Dependencies:
|
|
|
|
- current wireless client and subscriber matching system
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- subscriber attachment can be queried with confidence and timestamped history
|
|
|
|
### D3. Build an intelligent distributed poll scheduler
|
|
|
|
Why it matters:
|
|
|
|
- TowerOps already has agents; the next step is intelligent placement and failover
|
|
|
|
What to build:
|
|
|
|
- agent capability registry
|
|
- poll ownership leases
|
|
- health/capacity scoring
|
|
- automatic reassignment policy
|
|
- module-aware execution placement
|
|
|
|
Dependencies:
|
|
|
|
- current agent assignment system
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- scheduler can choose where a device or module should run
|
|
- reassignment does not corrupt results
|
|
- health/capacity data influences placement
|
|
|
|
### D4. Build wireless/RF intelligence
|
|
|
|
Why it matters:
|
|
|
|
- this is a natural extension of existing TowerOps strengths
|
|
|
|
What to build:
|
|
|
|
- AP health score
|
|
- client quality score
|
|
- roaming/session history
|
|
- AP overload forecasting
|
|
- probable RF fault attribution
|
|
|
|
Dependencies:
|
|
|
|
- current wireless client readings
|
|
|
|
Minimum acceptance criteria:
|
|
|
|
- at least one aggregate health score and one capacity-risk query exist
|
|
|
|
## Candidate Milestones
|
|
|
|
These are reasonable grouped milestones for future implementation.
|
|
|
|
### Milestone 1: Polling Foundation
|
|
|
|
- F1
|
|
- F2
|
|
- F3
|
|
- partial F4
|
|
|
|
Outcome:
|
|
|
|
- TowerOps polls existing entities more efficiently and more reliably
|
|
|
|
### Milestone 2: Core Coverage Catch-Up
|
|
|
|
- C1
|
|
- C2
|
|
- C4
|
|
- M1
|
|
- M2
|
|
- L1
|
|
|
|
Outcome:
|
|
|
|
- TowerOps closes some of the most visible LibreNMS gaps and gains richer discovered semantics
|
|
|
|
### Milestone 3: Network Operations Depth
|
|
|
|
- C5
|
|
- C6
|
|
- C7
|
|
- C8
|
|
- M4
|
|
- L2
|
|
- L3
|
|
|
|
Outcome:
|
|
|
|
- TowerOps becomes materially stronger for network engineering workflows
|
|
|
|
### Milestone 4: Platform Strength
|
|
|
|
- S1
|
|
- S2
|
|
- S3
|
|
- L4
|
|
- M5
|
|
|
|
Outcome:
|
|
|
|
- TowerOps becomes a stronger observability platform, not just a polling engine
|
|
|
|
### Milestone 5: Beyond LibreNMS
|
|
|
|
- D1
|
|
- D2
|
|
- D3
|
|
- D4
|
|
|
|
Outcome:
|
|
|
|
- TowerOps differentiates on topology-aware operations, subscriber impact, distributed polling, and RF intelligence
|
|
|
|
## Best Next Task For Another Agent
|
|
|
|
If another agent picks this up and needs the single best next task, it should start with:
|
|
|
|
- ✅ **DONE (2026-03-26)**: F1: replace sequential multi-get with real batched GET support
|
|
|
|
**The next best task is:**
|
|
|
|
- C1: add mempools
|
|
|
|
If both are done, the next best task is:
|
|
|
|
- C2: add transceivers / DOM optics
|
|
|
|
These tasks give the best mix of:
|
|
|
|
- broad technical leverage
|
|
- visible capability improvement
|
|
- alignment with the strategic direction described in this document
|