towerops/findings_librenms.md

LibreNMS vs TowerOps SNMP Review

Scope

This review compares the local LibreNMS checkout in ~/dev/librenms/ against the current TowerOps codebase in /Users/graham/dev/towerops/towerops-web.

I focused on:

• SNMP transport and polling behavior
• Device discovery and device lifecycle handling
• Metric/entity gathering
• Data normalization and threshold semantics
• Persistence and historical storage
• Operational robustness
• Areas where TowerOps is already better positioned

Key code reviewed included:

• LibreNMS:
  • app/Console/Commands/DevicePoll.php
  • app/Console/Commands/DeviceDiscover.php
  • app/PerDeviceProcess.php
  • LibreNMS/Modules/*
  • LibreNMS/Device/YamlDiscovery.php
  • LibreNMS/Data/Store/*
  • includes/discovery/*
  • includes/polling/*
  • app/Models/Device.php, Port.php, Sensor.php, Storage.php, Processor.php
• TowerOps:
  • lib/towerops/snmp/client.ex
  • lib/towerops/snmp/discovery.ex
  • lib/towerops/snmp/profiles/base.ex
  • lib/towerops/snmp/profiles/dynamic.ex
  • lib/towerops/snmp/deferred_discovery.ex
  • lib/towerops/snmp.ex
  • lib/towerops/workers/device_poller_worker.ex
  • lib/towerops/snmp/*.ex entity schemas
  • priv/repo/migrations/*timescaledb*

Executive Summary

TowerOps already has a solid SNMP foundation. It is not a toy implementation. It has:

• a real discovery pipeline
• a broad vendor-profile system
• historical storage in Postgres/Timescale
• parallel polling
• topology, ARP, MAC, and wireless-client handling
• agent-aware polling assignment

That said, LibreNMS is still materially more mature as a general SNMP monitoring platform.

The biggest differences are not "can TowerOps talk SNMP?" but:

• breadth of discovered entity types
• depth of metric semantics and normalization
• transport/polling heuristics for difficult devices
• lifecycle handling for discovered entities
• pluggable datastore/export story
• module-level operational maturity built up from years of edge cases

If the target is "as robust and feature rich as LibreNMS, then much more", TowerOps should keep its current architecture but expand it in a LibreNMS-like direction:

1. make transport and polling behavior much smarter
2. expand entity/module coverage aggressively
3. add richer metric semantics and threshold/state models
4. strengthen discovery drift and lifecycle handling
5. treat topology, subscriber correlation, and agent-distributed polling as the differentiators that go beyond LibreNMS

Expanded Strategy

The five points above are the right high-level direction, but they need to be translated into an explicit engineering strategy. Below is what each one should mean in practice.

1. Make transport and polling behavior much smarter

This is the most immediate leverage point because it improves every current and future module.

Right now TowerOps has solid SNMP primitives, but it still behaves too much like a clean protocol client and not enough like a hardened polling engine. LibreNMS has a lot of ugly historical logic here, but that ugliness exists for a reason: real devices are inconsistent, slow, buggy, and often only partially standards-compliant.

TowerOps should evolve from "SNMP client plus polling worker" into "adaptive polling engine with device-specific transport policy".

That means adding a transport capability model per device or profile, including:

• preferred operation mode: get, get_next, walk, bulk_walk
• whether GETBULK is safe
• max repeaters / repetition window
• retry policy
• timeout class
• whether responses can be unordered
• whether certain OIDs must avoid bulk or avoid combined requests
• whether a device should prefer table walks vs selected-item gets
• whether interface polling should be selective or full-table

This should not stay implicit in code branches. It should become explicit discovered or profile-driven capability data.

Concrete improvements:

• implement a true multi-OID GET path so get_multiple/2 is not a loop of individual requests
• move more polling to table-oriented retrieval, especially for interfaces and sensors
• add adaptive bulk sizing based on response size, timeout rate, and prior successful runs
• add a device-level "transport memory" layer so TowerOps learns what works on a specific box
• distinguish "device is down" from "this module/OID strategy is bad for this device"
• add poll cost accounting: request count, bytes returned, duration, failure rate, timeout count

The goal is not just speed. The goal is stable polling on weird devices without special-casing everything forever.

A good end state would be:

• first discovery runs conservatively
• TowerOps learns the device's behavior
• later polls use an optimized strategy
• if errors increase, TowerOps automatically degrades to a safer strategy

LibreNMS has a lot of static heuristics. TowerOps can do better by making that adaptive.

2. Expand entity/module coverage aggressively

TowerOps should treat SNMP coverage as a product surface, not a side effect of polling.

LibreNMS is stronger because it models many more things. That breadth matters because operators do not think in terms of "poll sysObjectID and a few sensors". They think in terms of:

• optics
• memory
• routes
• BGP neighbors
• STP topology
• QoS queues
• storage
• power supplies
• printer consumables
• xDSL lines
• VM guests
• hardware inventory

TowerOps should formalize a module catalog and then grow it quickly.

Suggested first-class module families:

• Core
• Interfaces
• InterfaceExtendedStats
• Sensors
• StateSensors
• Processors
• Mempools
• Storage
• Transceivers
• EntityPhysical
• Vlans
• Ipv4Addresses
• Ipv6Addresses
• Neighbors
• Arp
• Mac
• Routes
• Stp
• Qos
• Bgp
• Ospf
• Services
• Wireless
• PrinterSupplies
• Xdsl

This should not all land at once, but the module system should be designed for that future immediately.

The order I would recommend:

1. Mempools
2. Transceivers
3. EntityPhysical
4. Routes
5. Qos
6. Bgp
7. Stp

Why this order:

• mempools and transceivers are common, visible, and high-value
• entity physical gives a backbone for inventory, FRUs, and relationship mapping
• routes/QoS/BGP/STP move TowerOps from device monitoring into actual network operations monitoring

This should be done with a mix of:

• generic standard-MIB implementations
• vendor/profile extensions
• a shared module contract for discovery, poll, sync, cleanup, and capability checks

The key is to prevent new coverage from turning the codebase into an ever-growing device_poller_worker.ex blob. Coverage growth must come with module boundaries.

3. Add richer metric semantics and threshold/state models

TowerOps currently stores readings well, but it does not yet model metric meaning as deeply as LibreNMS.

To match and then surpass LibreNMS, TowerOps needs to treat each discovered metric-bearing entity as more than:

• an OID
• a current value
• a timestamp

Each metric should carry enough metadata for TowerOps to reason about it correctly.

That means first-class support for:

• raw value
• normalized value
• divisor
• multiplier
• unit
• unit family
• metric type: gauge, counter, derive, state
• threshold policy
• warning/critical bounds
• low and high bounds
• reset/wrap behavior
• value source quality
• preferred graph aggregation

For state sensors, TowerOps should introduce a proper state model, not just a string description on readings.

LibreNMS has stronger state semantics through translation tables. TowerOps should add something similar but cleaner:

• state index definition
• translation map from raw values to normalized state
• generic severity mapping: ok, warning, critical, unknown
• change-event policy
• suppression/debounce policy

For numeric metrics, TowerOps should support discovered threshold metadata and local overrides separately.

There are really three threshold layers:

1. vendor/device-provided thresholds
2. profile defaults
3. operator overrides

Those should be modeled independently so the system can explain why a value is considered abnormal.

This also enables better product behavior:

• graphs can render warning/critical zones correctly
• alerts can distinguish vendor-threshold breach vs operator policy breach
• anomaly detection can compare against both thresholds and learned baselines

The most important conceptual change here is:

TowerOps should move from "store readings and run checks" to "understand the semantics of each metric, then use checks as one consumer of that model".

That would be a stronger architecture than LibreNMS.

4. Strengthen discovery drift and lifecycle handling

LibreNMS is robust partly because it assumes discovered entities are unstable:

• interfaces rename
• ifIndex values drift
• sensors vanish and come back
• optics move slots
• storage entries reorder
• devices partially implement MIBs

TowerOps already syncs discovered entities and cleans up certain stale records, but it should develop a full discovery lifecycle model.

Every discovered entity should have lifecycle semantics:

• first discovered at
• last seen at
• last changed at
• active / stale / deleted state
• stable identity key
• mutable display attributes
• drift reason if identity changed

Examples:

• an interface can keep the same stable identity while alias, speed, or label changes
• a transceiver can be removed and later reinserted
• a storage entry can move from "active" to "missing" without immediate hard deletion
• a sensor can be marked unsupported by current firmware after an upgrade

TowerOps should also record discovery drift as auditable events:

• interface renamed
• ifIndex changed
• sensor threshold changed
• storage table reordered
• model/serial/firmware changed
• LLDP capability set changed

This is important for both reliability and UX. Operators need to know whether a graph or alert changed because the system discovered something new or because the device actually changed.

A strong end state would include two layers:

• current entity view
• discovery history / identity history

That would let TowerOps survive SNMP instability much better than a simple overwrite model.

This is also where entity sync strategy matters. For each entity family, TowerOps should explicitly define:

• identity key
• update fields
• staleness window
• hard delete window
• drift event rules

Without that, robustness will always be uneven across modules.

5. Treat topology, subscriber correlation, and agent-distributed polling as the differentiators that go beyond LibreNMS

This is where TowerOps should stop thinking like "an NMS trying to catch up" and start thinking like "a network operations platform that happens to include SNMP".

LibreNMS is broad and mature, but its center of gravity is still classic device monitoring.

TowerOps already has stronger raw ingredients for something more valuable:

• topology inference from neighbors, ARP, and MAC evidence
• wireless client discovery with history
• subscriber correlation
• multi-tenant data model
• agent-aware polling ownership

These should not remain side features. They should become the main differentiator.

There are three major product directions here.

First, topology should become operational, not just descriptive.

That means:

• infer path relationships, not just direct links
• attach devices, interfaces, wireless clients, and subscribers into one graph
• use graph state during incident analysis
• identify probable upstream failure points
• compute blast radius automatically

A traditional NMS shows "these devices are down". A stronger TowerOps should be able to say "this uplink failure is causing 143 impacted subscribers across these APs and these sites".

Second, subscriber correlation should become a first-class consumer of SNMP and topology data.

That means:

• correlate subscribers to APs, sectors, switches, and upstream paths
• retain movement history
• distinguish probable vs confirmed attachment
• use RF quality, ARP evidence, and topology together
• enable customer-impact views instead of just device-impact views

This is much closer to how WISPs and access-network operators actually think.

Third, agent-distributed polling should become a smart execution fabric.

Right now TowerOps is aware of agents. The next step is to make scheduling and placement intelligent:

• route polling based on network reachability and locality
• detect overloaded or unhealthy agents
• move devices between agents automatically
• support partial capability ownership by module
• allow fallback execution strategies
• track poll success by agent, region, and module

That would put TowerOps in a position LibreNMS generally is not designed for:

• hybrid central/distributed polling
• tenant-aware placement
• topology-aware scheduling
• richer execution telemetry

The combined opportunity is substantial.

If TowerOps executes well here, the platform can evolve from:

• "SNMP monitoring system"

into:

• "distributed network observability and operations system"

That is the path to "much more".

Where TowerOps Is Already Strong

These are real strengths, not consolation prizes.

1. Better application architecture for future growth

TowerOps has a cleaner domain split than LibreNMS:

• Towerops.Snmp as a context API
• explicit schemas for sensors, interfaces, storage, processors, neighbors, ARP, MACs, wireless clients
• Oban workers for polling/discovery/cleanup
• TimescaleDB-oriented historical tables

LibreNMS has a lot of mature logic, but much of it still spans legacy includes, DB helpers, and mixed discovery/polling styles.

2. Better integrated topology and subscriber-aware data model

TowerOps is already ahead in areas LibreNMS treats more generically:

• neighbor polling and topology inference
• ARP/MAC correlation into topology
• wireless client discovery with history
• subscriber/device correlation (Gaiia.SubscriberMatching)
• agent-aware ownership and result discard on reassignment

This is the right direction if TowerOps wants to become more than a standard NMS.

3. Better relational and analytical storage story

TowerOps stores historical SNMP data in PostgreSQL/TimescaleDB:

• snmp_sensor_readings
• snmp_interface_stats
• snmp_processor_readings
• snmp_storage_readings
• wireless_client_readings

LibreNMS primarily treats RRD as the historical source of truth and the SQL DB as current-state metadata. That model is proven, but TowerOps has a better foundation for analytics, joins, ML-style insights, and tenant-aware retention.

4. Strong vendor coverage direction

TowerOps already has a large vendor-profile surface under lib/towerops/snmp/profiles/vendors/. That is the correct scaling model if it remains disciplined.

Major Findings

1. LibreNMS has a much broader module surface

LibreNMS ships a real module catalog under LibreNMS/Modules/, including:

• ArpTable
• Availability
• Core
• DiscoveryArp
• EntityPhysical
• HrDevice
• IpSystemStats
• Ipv4Addresses
• Ipv6Addresses
• Ipv6Nd
• Isis
• MacAccounting
• Mempools
• Mpls
• Nac
• Netstats
• Ospf
• Ospfv3
• PortSecurity
• PortsStack
• PrinterSupplies
• Qos
• Routes
• Services
• Slas
• Storage
• Stp
• Transceivers
• UcdDiskio
• Vlans
• Vminfo
• Wireless
• Xdsl

TowerOps currently covers a narrower but useful subset:

• core system/device info
• interfaces
• sensors/state sensors
• processors
• storage
• VLANs
• IP addresses
• neighbors
• ARP
• MAC/FDB
• wireless clients

Impact

TowerOps is already viable for device-centric SNMP monitoring, but it is not yet comparable to LibreNMS as a full-spectrum network/server SNMP platform.

Highest-priority missing entity families

• mempools / memory pools
• transceivers / DOM optics
• printer supplies
• hr-device / detailed hardware inventory
• route tables
• STP
• BGP/OSPF/ISIS/MPLS
• QoS
• SLAs
• xDSL
• NAC / port security / MAC accounting
• VM info
• service/application monitoring through device-side integrations

2. LibreNMS transport behavior is more battle-hardened

LibreNMS transport code and poller behavior show years of hardening:

• per-device timeout/retry/max repeater logic in includes/snmp.inc.php
• choice between snmpwalk and snmpbulkwalk
• explicit no-bulk exceptions for OS/OID combinations
• unordered response allowances
• module-level selective polling behavior
• selected-port polling heuristics in includes/polling/ports.inc.php

TowerOps has a clean SNMP client in lib/towerops/snmp/client.ex, but it is currently simpler.

Two specific gaps stand out:

Gap: get_multiple/2 is not a true multi-get

Towerops.Snmp.Client.get_multiple/2 currently loops and issues one GET per OID when using the default adapter.

That is materially less efficient and less robust than LibreNMS's batching and bulkwalk-heavy approach.

Gap: bulk strategy is not used as aggressively as it should be

TowerOps has get_bulk/3, but the discovery/polling paths still rely heavily on repeated walk/2 and sequential per-interface/per-sensor fetches.

LibreNMS is much more aggressive about:

• walking whole tables
• selecting bulk vs non-bulk based on device/OS quirks
• reducing round trips

Recommendation

Build a transport capability layer per device/profile:

• supports real SNMP GET for multiple OIDs in one PDU
• supports adaptive GETBULK with per-device max_repetitions
• allows per-profile and per-OID no_bulk, unordered, and retry overrides
• caches known bad behaviors per device/profile

This is one of the highest ROI improvements in the whole system.

3. LibreNMS discovery is much richer and more systematic

LibreNMS discovery combines:

• fast OS detection in LibreNMS\Modules\Core
• YAML-driven OS detection and discovery definitions
• model synchronization for discovered entities
• dedicated discovery modules for ports, sensors, storage, mempools, transceivers, etc.

TowerOps has two strong ideas here:

• DeferredDiscovery explicitly separates fast and slow work
• Profiles.Dynamic mirrors LibreNMS's YAML/vendor-profile approach

That is good. But TowerOps discovery breadth is still behind, and its entity lifecycle handling is simpler.

What LibreNMS does better

• more discovery modules
• more fallback paths per OS
• more mature OS detection
• richer entity-specific YAML traits for mempools/storage/processors
• better coverage of special-case discovery includes

What TowerOps does better

• clearer deferred-discovery concept
• easier-to-reason-about Elixir code
• better future fit for typed, testable discovery pipelines

Recommendation

Keep the current discovery design, but formalize modules around it:

• Core
• Interfaces
• Sensors
• StateSensors
• Storage
• Processors
• Mempools
• Transceivers
• Vlans
• Ipv4
• Ipv6
• Neighbors
• Arp
• Mac
• Wireless
• Routes
• Stp
• Services

Right now the logic exists, but it is still more monolithic than LibreNMS's module surface.

4. TowerOps device handling is cleaner, but operationally shallower

LibreNMS app/Models/Device.php carries a lot of operational metadata:

• status and maintenance semantics
• ignore/disabled flags
• poller group
• retries/timeout
• display formatting
• hostname vs overwrite IP handling
• SNMP v1/v2c/v3 support details
• VRF context support
• device type defaults and overrides

TowerOps lib/towerops/devices/device.ex is cleaner and more tenant-aware:

• organization/site ownership
• SNMP credentials and source tracking
• agent assignment compatibility
• check intervals
• MikroTik credentials

But it lacks some of LibreNMS's mature device-operability semantics.

Missing or underdeveloped device-level concerns

• poller grouping / queue affinity / sharding semantics
• per-device SNMP timeout and retry tuning
• VRF/context-aware polling
• ignore/disabled semantics at the same depth
• richer maintenance/operational modes
• device capability flags learned from discovery
• more complete current-state summary fields

Recommendation

TowerOps should add a discovered capability profile to the device/SNMP device:

• bulk-capable?
• unstable ifIndex?
• supports HC counters?
• supports LLDP/CDP?
• supports ENTITY-SENSOR-MIB?
• supports HR storage?
• supports mempool-like resources?
• preferred polling profile

LibreNMS's maturity partly comes from years of implicit capability handling. TowerOps should make that explicit.

5. LibreNMS has much richer metric semantics

This is one of the biggest gaps.

LibreNMS sensor/storage/mempool/processor models carry more semantic structure:

• thresholds on sensors
• low/high warning and critical limits
• state translation tables for state sensors
• divisor and multiplier handling
• user transformation functions
• rate-oriented sensor handling for COUNTER/DERIVE types
• storage and mempool "fill missing ratio" logic
• class-aware memory availability calculations

TowerOps sensor and reading models are currently simpler:

• snmp_sensors stores type/index/OID/descr/unit/divisor/last value/metadata
• snmp_sensor_readings stores value/status/state_descr/timestamp
• processors and storage have simpler current + reading schemas

Specific gaps in TowerOps

• no generic threshold model on discovered sensors
• no explicit state translation/index model like LibreNMS state sensors
• no generic multiplier support visible in the schema
• no generic rate sensor type handling
• no mempool entity family
• no derived memory availability logic
• narrower processor taxonomy
• narrower storage semantics

Recommendation

Add first-class metric semantics to the discovery model, not just to alert checks.

Each discovered metric-capable entity should be able to carry:

• raw OID
• normalized value
• divisor
• multiplier
• data type: gauge, counter, derive, state
• threshold policy
• state translation table reference
• preferred graph/aggregation strategy
• unit family

That will let TowerOps match LibreNMS for monitoring semantics while still using a better storage backend.

6. TowerOps interface statistics are currently too narrow

TowerOps snmp_interface_stats stores:

• in/out octets
• in/out errors
• in/out discards
• timestamp
• HC flag

LibreNMS port polling tracks a much broader port state space:

• octets
• packets
• unicast / multicast / broadcast
• errors
• discards
• unknown protocols
• duplex
• PoE / etherlike / vendor extensions
• admin/oper states
• speed and description drift
• selected-port polling and deleted-port handling

TowerOps also checks attribute changes separately, which is useful, but the historical stat model is still fairly thin.

Recommendation

Expand interface stat collection and persistence to include at least:

• unicast/multicast/broadcast packet counters
• unknown protocol counters
• pause / queue drop counters where available
• duplex and negotiated speed snapshots
• optional per-vendor extended counters

Then add derived rate queries and wrap/reset detection.

If TowerOps wants to be better than LibreNMS, it should not stop at raw counters. It should also compute:

• utilization percent
• error rate
• discard rate
• anomaly flags
• flap score

7. LibreNMS handles discovered entity lifecycle more robustly

LibreNMS port discovery/polling has mature handling for:

• new entities
• deleted/missing entities
• re-discovered entities
• association mode drift
• selective polling skips
• sync/update semantics

TowerOps has sync functions in Towerops.Snmp.Discovery and stale cleanup for neighbors/ARP/MAC/wireless data, but the lifecycle model is still lighter.

Gaps

• fewer explicit "deleted vs active vs stale" semantics on discovered entities
• less explicit port/interface association strategy management
• less historical handling of identity drift
• less per-entity lifecycle metadata

Recommendation

For interfaces, sensors, storage, processors, and transceivers once added:

• add discovered_at
• add last_seen_at
• add deleted_at or active/deleted flags
• persist stable identity keys separate from mutable labels
• keep rediscovery drift audit events

LibreNMS survives a lot of weird devices because it expects entities to come and go. TowerOps should encode that assumption directly.

8. LibreNMS has a more flexible datastore/export model

LibreNMS LibreNMS\Data\Store\Datastore fans out to multiple storage/export sinks:

• RRD
• InfluxDB
• InfluxDB v2
• Prometheus
• Graphite
• Kafka

TowerOps currently has a stronger primary store, but not a comparable export abstraction.

Why this matters

If TowerOps wants to be operationally superior, it should be able to:

• retain canonical data in Postgres/Timescale
• expose Prometheus-friendly metrics
• stream high-value events/metrics out
• support long-term lake/warehouse pipelines

Recommendation

Add a metric sink abstraction around polling writes:

• primary sink: Postgres/Timescale
• optional derived/export sinks: Prometheus remote-write bridge, Kafka/NATS, warehouse feed

Do not copy LibreNMS's RRD-first model. Keep TowerOps's DB-first model, but add output flexibility.

9. LibreNMS has more mature per-module operational controls

LibreNMS supports:

• per-run module selection
• per-device module behavior
• per-module debug workflows
• explicit module dependencies
• legacy and queued execution styles

TowerOps polling is centered around one large worker that concurrently runs a fixed set of tasks.

That is workable, but less modular operationally.

Recommendation

Refactor polling into pluggable modules with a scheduler contract:

• each module advertises dependencies
• each module advertises required discovery support
• each module advertises poll interval suitability
• each module can be enabled/disabled per organization, per device, per profile

This will make TowerOps easier to operate at scale and easier to test.

10. LibreNMS has stronger semantics around ports than TowerOps currently does

LibreNMS ports are a central first-class object. It has:

• port label normalization
• short/full labels
• association modes
• deletion handling
• port groups
• port-state querying helpers
• linked IP/MAC/VLAN/STP/etc relationships

TowerOps interfaces are useful, but they are not yet at the same level of operational richness.

Recommendation

Elevate interfaces from "discovered SNMP rows" to "first-class network edges".

That means adding:

• better labeling and normalization
• role classification
• trunk/access/LAG semantics
• stack/aggregate relationships
• transceiver attachment
• capacity source and negotiated speed detail
• per-interface health summary

11. TowerOps is ahead on wireless client history and correlation

This is one of the clearest places TowerOps is already beyond LibreNMS's default orientation.

TowerOps has:

• wireless client discovery
• wireless client readings
• weak-signal and low-SNR queries
• overloaded AP queries
• subscriber matching and refresh paths

LibreNMS has wireless support, but TowerOps is better positioned to build product-level RF/subscriber workflows.

Recommendation

Lean into this. Make it a core differentiator:

• RF health scoring
• AP capacity forecasting
• roaming/session history
• client-to-backhaul path correlation
• outage blast-radius analysis using topology + wireless clients + subscribers

12. TowerOps is ahead on agent-aware polling, but needs stronger scheduling policy

TowerOps has an architectural advantage with:

• agent assignments
• "discard results if assignment changed mid-poll"
• cloud vs agent execution model

LibreNMS has scale-oriented process control, but not this exact hybrid architecture.

Recommendation

Turn agent-aware polling into a first-class scheduler:

• capability-aware module assignment
• agent health/capacity scoring
• per-device poll ownership leases
• backpressure and dynamic interval adjustment
• retry/failover between Phoenix-side SNMP and remote agents where appropriate

This is one of the best paths to become "much more" than LibreNMS.

Prioritized Roadmap

P0: Transport and polling efficiency

• Implement real multi-OID GET batching instead of sequential GET loops.
• Use GETBULK much more aggressively for table polling.
• Add per-device/per-profile max-repeater tuning.
• Add no_bulk, unordered-response, and retry overrides by profile/OID.
• Add selected-interface polling and adaptive table walk behavior.

P1: Fill the biggest entity gaps

• Add mempools.
• Add transceivers / DOM optics.
• Add printer supplies.
• Add hr-device / hardware inventory.
• Add route/STP/QoS modules.
• Add routing protocol entities incrementally: BGP first, then OSPF.

P2: Rich metric semantics

• Add threshold metadata to discovered metrics.
• Add state translation/index support.
• Add multiplier and rate-type support.
• Add counter wrap/reset handling.
• Add richer interface counter persistence.
• Add derived rate/utilization/error computations.

P3: Discovery and lifecycle robustness

• Add moduleized discovery and polling contracts.
• Add entity lifecycle fields: last_seen_at, deleted_at, stable identity keys.
• Add capability profiling per device.
• Add rediscovery drift events for key entities.

P4: Storage and export maturity

• Extend Timescale retention/rollup coverage beyond current tables.
• Add export sink abstraction.
• Add Prometheus/Kafka-style optional egress.
• Build standard rollups for rates, percentile windows, and anomaly summaries.

P5: Differentiators beyond LibreNMS

• topology-aware outage detection
• subscriber-aware path analysis
• agent-distributed polling with automatic placement
• RF/wireless health intelligence
• inventory drift and config drift correlation
• multi-tenant operational analytics

Bottom Line

LibreNMS is still more complete and more battle-tested as a broad SNMP platform.

TowerOps is already better in architecture, relational history, topology correlation, wireless-client modeling, and agent-aware execution.

The right strategy is not to copy LibreNMS literally.

The right strategy is:

• copy LibreNMS's breadth, transport hardening, and metric semantics
• keep TowerOps's cleaner architecture and Timescale-first storage
• push harder on topology, subscriber correlation, wireless intelligence, and distributed polling

If I were prioritizing purely for engineering leverage, I would do this next:

1. fix SNMP batching/bulk behavior
2. add mempools and transceivers
3. enrich interface and sensor semantics
4. modularize discovery/polling
5. turn topology + wireless + agents into the "beyond LibreNMS" layer

Future Agent Worklist

This section is meant to be directly usable by another coding agent in the future.

The intent is:

• do not start by re-researching LibreNMS from scratch
• use this as the implementation backlog and sequencing guide
• treat each item as a deliverable with explicit outcomes

Unless product priorities change, future work should generally follow the order below.

Working Rules For Future Agents

• preserve the current TowerOps architectural direction: context modules, Oban workers, typed schemas, Timescale-first history
• do not copy LibreNMS's legacy PHP/include layout; copy the capabilities, not the structure
• prefer module extraction over adding more logic to lib/towerops/workers/device_poller_worker.ex
• when adding a new entity family, add:
  • discovery
  • polling
  • persistence
  • tests
  • lifecycle behavior
  • at least a minimal query API in Towerops.Snmp
• when adding visible UI text later, remember TowerOps requires translations
• when adding new functions, add unit tests
• when finishing implementation work, run the repo quality gates required by this project

Suggested Delivery Format For Future Work

For each substantial feature, future agents should aim to deliver:

• schema and migration changes
• module implementation
• context API changes
• tests
• a short design note or ADR snippet in this file or a follow-up doc if the change alters system architecture

Backlog Overview

The backlog is split into:

• Foundation
• Coverage Expansion
• Metric Semantics
• Lifecycle and Sync
• Storage and Query Layer
• Product Differentiators

Each item below includes:

• why it matters
• what to build
• dependencies
• minimum acceptance criteria

Foundation

F1. Replace sequential multi-get with real batched GET support

Why it matters:

• current Towerops.Snmp.Client.get_multiple/2 behavior is materially less efficient than LibreNMS
• this affects discovery, polling, and all future modules

What to build:

• a true multi-OID GET path in the SNMP adapter/client layer
• fallback behavior when a device rejects grouped requests
• instrumentation for grouped-request success/failure

Dependencies:

• none

Minimum acceptance criteria:

• get_multiple/2 performs one grouped request for supported devices
• fallback exists for devices that reject grouped GETs
• tests cover both success and fallback paths
• existing discovery code using get_multiple/2 continues to work

F2. Build adaptive bulk-walk policy

Why it matters:

• many future modules depend on efficient table polling
• bulk strategy is one of the biggest practical differences between TowerOps and LibreNMS maturity

What to build:

• per-device or per-profile bulk policy
• support for max_repetitions
• no_bulk overrides
• unordered-response allowances where required
• downgrade path when bulk polling fails

Dependencies:

• F1 recommended but not strictly required

Minimum acceptance criteria:

• table polling can choose walk vs bulk walk based on capability
• policy is configurable and/or discoverable
• failures degrade safely instead of repeatedly timing out
• tests cover normal bulk, disabled bulk, and fallback cases

F3. Add transport capability profiling

Why it matters:

• TowerOps should learn device behavior instead of relying only on static vendor assumptions

What to build:

• capability fields for transport behavior, either on snmp_devices or a dedicated capabilities table
• values such as:
  • bulk safe
  • preferred max repeaters
  • supports HC counters
  • unstable interface identity
  • supports LLDP/CDP
  • supports ENTITY-SENSOR-MIB

Dependencies:

• F1 and F2 are helpful

Minimum acceptance criteria:

• capability data is persisted
• polling can read and use capability data
• capability defaults exist for unknown devices

F4. Break discovery/polling into explicit module contracts

Why it matters:

• future coverage growth will become unmanageable if everything stays centralized

What to build:

• a discovery module behavior
• a poll module behavior
• module metadata:
  • name
  • dependencies
  • capability requirements
  • entity families affected

Dependencies:

• none, but should happen before too many new modules are added

Minimum acceptance criteria:

• at least a few existing features are migrated to the new module contract
• device_poller_worker becomes an orchestrator, not a feature blob
• tests cover module orchestration

Coverage Expansion

C1. Add mempools

Why it matters:

• LibreNMS has dedicated memory pool support and operators expect it
• TowerOps currently has processors and storage but not memory pools as a first-class family

What to build:

• schema for discovered memory pools
• schema for historical readings
• generic discovery using standard resources MIBs where possible
• vendor-specific profile support
• current-state queries in Towerops.Snmp

Dependencies:

• F4 preferred

Minimum acceptance criteria:

• devices with standard memory pool data discover and poll successfully
• current usage, total, free, and percent are stored
• history is queryable
• tests cover standard and at least one vendor-specific path

C2. Add transceivers / optical DOM

Why it matters:

• optics are a major operational surface and a key LibreNMS strength
• this is especially important for wireless backhaul and transport networks

What to build:

• transceiver entity schema
• DOM sensor association model
• discovery from entity physical and vendor-specific optics tables
• optical metrics such as Rx power, Tx power, bias current, temperature, voltage

Dependencies:

• C4 strongly helps

Minimum acceptance criteria:

• transceivers can be discovered as distinct entities
• optical readings are tied to a transceiver or interface
• at least one vendor-specific path works beyond generic ENTITY-MIB

C3. Add printer supplies

Why it matters:

• this is a common LibreNMS feature and a good test of non-network device coverage

What to build:

• supply entity model
• toner/drum/etc levels
• thresholds and depletion state

Dependencies:

• F4 preferred

Minimum acceptance criteria:

• standard printer supply MIB path supported
• current levels and warning states stored

C4. Add entity physical / inventory model

Why it matters:

• this becomes the backbone for FRUs, transceivers, power supplies, fans, slot inventory, and richer topology

What to build:

• physical entity schema
• parent/child relationships
• class/type/name/serial/model/manufacturer where available
• association to discovered sensors and transceivers

Dependencies:

• none, but should happen early

Minimum acceptance criteria:

• ENTITY-MIB hardware inventory is discoverable and queryable
• parent-child relationships are preserved
• sensors can reference physical entities when possible

C5. Add route table discovery

Why it matters:

• routes move TowerOps closer to network operations rather than only device health

What to build:

• route entity model
• route polling/discovery for destination, next hop, protocol, metric, interface
• optional route summarization rather than full retention if scale becomes an issue

Dependencies:

• F4 preferred

Minimum acceptance criteria:

• basic IPv4 route discovery works
• current routes are queryable by device
• lifecycle handling exists for route disappearance

C6. Add QoS visibility

Why it matters:

• QoS is critical in provider and wireless networks

What to build:

• queue/class entity models
• drops, utilization, and queue stats where available
• vendor/profile extensions

Dependencies:

• F4 preferred

Minimum acceptance criteria:

• at least one generic or vendor implementation exists
• queue stats are stored historically

C7. Add routing protocol visibility

Suggested order:

1. BGP
2. OSPF
3. ISIS if needed later

Why it matters:

• this is where TowerOps starts to compete with real network operations platforms

What to build:

• peer/session entities
• state, uptime, counters, error signals
• historical session-state events

Dependencies:

• F4 preferred
• C5 helpful

Minimum acceptance criteria:

• BGP peer discovery and state polling for a standard implementation
• current peer state query API
• transition events recorded

C8. Add STP visibility

Why it matters:

• useful for topology correctness and loop-domain troubleshooting

What to build:

• bridge/STP instance entities
• port roles and states
• root bridge and topology change info

Dependencies:

• topology work benefits from this

Minimum acceptance criteria:

• basic STP state and root data discoverable
• key bridge state changes visible

Metric Semantics

M1. Add generic threshold model for discovered entities

Why it matters:

• thresholds should not exist only in ad hoc check config
• discovered metrics need native warning/critical semantics

What to build:

• threshold fields or associated threshold table
• support for:
  • low warn
  • low critical
  • high warn
  • high critical
• distinction between discovered thresholds and operator overrides

Dependencies:

• none

Minimum acceptance criteria:

• sensors, processors, storage, and future mempools can all carry thresholds
• system can explain the active threshold source

M2. Add state translation/index model

Why it matters:

• state sensors need richer semantics than status plus optional state_descr

What to build:

• state index definition
• state translation table
• mapping from raw values to normalized states
• shared event/severity model

Dependencies:

• none

Minimum acceptance criteria:

• state sensors use the shared translation model
• transitions can be interpreted consistently across devices

M3. Add value transformation semantics

Why it matters:

• many SNMP values need divisor, multiplier, or special processing

What to build:

• first-class support for:
  • divisor
  • multiplier
  • unit family
  • counter/gauge/derive/state typing
  • optional post-processing hooks per profile

Dependencies:

• none

Minimum acceptance criteria:

• transformed values are stored consistently
• raw and normalized values are distinguishable where useful

M4. Add richer interface history

Why it matters:

• interface stats are currently too narrow compared to LibreNMS

What to build:

• more packet counters
• multicast/broadcast/unicast separation
• unknown protocol counters
• optional queue/duplex/speed snapshots
• derived rates and utilization queries

Dependencies:

• F2 helps

Minimum acceptance criteria:

• interface stats schema expanded
• historical rate calculations are supported
• counter resets/wraps are handled safely

M5. Add anomaly-ready derived metrics

Why it matters:

• TowerOps should become analytically stronger than LibreNMS

What to build:

• derived rate/utilization/error views
• percentiles and rolling windows
• anomaly flags or baseline deviation fields

Dependencies:

• M4 and Timescale rollups help

Minimum acceptance criteria:

• at least one entity family has usable derived rollups
• queries exist for current anomalies or unusual behavior

Lifecycle and Sync

L1. Add lifecycle fields to all discovered entities

Why it matters:

• robust discovery means understanding appearance, disappearance, and drift

What to build:

• first_seen_at
• last_seen_at
• last_changed_at
• deleted_at or active/stale/deleted state

Dependencies:

• none

Minimum acceptance criteria:

• interfaces, sensors, processors, storage, and new entity families have lifecycle tracking

L2. Define stable identity keys per entity family

Why it matters:

• sync correctness depends on identity stability, not just current labels

What to build:

• explicit identity-key strategy for each family:
  • interfaces
  • sensors
  • processors
  • storage
  • transceivers
  • mempools
  • routes

Dependencies:

• L1 helpful

Minimum acceptance criteria:

• sync code uses documented identity keys
• drift from mutable labels does not create unnecessary duplicates

L3. Add discovery drift events

Why it matters:

• operators and future systems need to know what changed in discovered state

What to build:

• event types such as:
  • interface renamed
  • ifIndex changed
  • firmware changed
  • serial changed
  • threshold changed
  • physical inventory changed

Dependencies:

• L1 and L2

Minimum acceptance criteria:

• drift events are persisted or broadcast
• at least interfaces and device identity changes are covered

L4. Add soft-delete and grace-window sync behavior

Why it matters:

• hard deletion on first miss is fragile for SNMP

What to build:

• configurable grace windows before delete
• stale marking before hard delete
• optional module-specific policies

Dependencies:

• L1

Minimum acceptance criteria:

• entities can move through active -> stale -> deleted states

Storage and Query Layer

S1. Expand Timescale coverage

Why it matters:

• more modules need history, rollups, and retention

What to build:

• hypertables and rollups for new high-volume reading tables
• retention policies
• continuous aggregates where appropriate

Dependencies:

• new entity families

Minimum acceptance criteria:

• each new high-volume metric family has a defined storage strategy

S2. Add export sink abstraction

Why it matters:

• TowerOps should keep DB-first storage but allow egress to other systems

What to build:

• sink abstraction for metric/event export
• optional sinks such as:
  • Prometheus-oriented export path
  • Kafka/NATS/event-stream output
  • warehouse feed

Dependencies:

• none

Minimum acceptance criteria:

• internal writes remain canonical in Postgres/Timescale
• at least one optional sink can be enabled without changing poll logic

S3. Add standardized query APIs for every new family

Why it matters:

• future product features should not need to query raw tables ad hoc

What to build:

• Towerops.Snmp API functions for listing entities, current state, and reading history

Dependencies:

• each feature as it lands

Minimum acceptance criteria:

• every new entity family has a minimal public query API

Product Differentiators

D1. Turn topology into an operational graph

Why it matters:

• topology should explain impact, not just relationships

What to build:

• graph-level path reasoning
• upstream/downstream inference
• failure impact calculations
• stale confidence scoring on edges

Dependencies:

• existing topology system
• C8 helpful

Minimum acceptance criteria:

• system can identify likely impacted downstream devices or subscribers from an upstream problem

D2. Turn subscriber correlation into a first-class model

Why it matters:

• this is one of the biggest ways to exceed LibreNMS

What to build:

• attachment confidence model
• movement history
• path association from subscriber -> AP -> switch/uplink
• current and historical linkage APIs

Dependencies:

• current wireless client and subscriber matching system

Minimum acceptance criteria:

• subscriber attachment can be queried with confidence and timestamped history

D3. Build an intelligent distributed poll scheduler

Why it matters:

• TowerOps already has agents; the next step is intelligent placement and failover

What to build:

• agent capability registry
• poll ownership leases
• health/capacity scoring
• automatic reassignment policy
• module-aware execution placement

Dependencies:

• current agent assignment system

Minimum acceptance criteria:

• scheduler can choose where a device or module should run
• reassignment does not corrupt results
• health/capacity data influences placement

D4. Build wireless/RF intelligence

Why it matters:

• this is a natural extension of existing TowerOps strengths

What to build:

• AP health score
• client quality score
• roaming/session history
• AP overload forecasting
• probable RF fault attribution

Dependencies:

• current wireless client readings

Minimum acceptance criteria:

• at least one aggregate health score and one capacity-risk query exist

Candidate Milestones

These are reasonable grouped milestones for future implementation.

Milestone 1: Polling Foundation

• F1
• F2
• F3
• partial F4

Outcome:

• TowerOps polls existing entities more efficiently and more reliably

Milestone 2: Core Coverage Catch-Up

• C1
• C2
• C4
• M1
• M2
• L1

Outcome:

• TowerOps closes some of the most visible LibreNMS gaps and gains richer discovered semantics

Milestone 3: Network Operations Depth

• C5
• C6
• C7
• C8
• M4
• L2
• L3

Outcome:

• TowerOps becomes materially stronger for network engineering workflows

Milestone 4: Platform Strength

• S1
• S2
• S3
• L4
• M5

Outcome:

• TowerOps becomes a stronger observability platform, not just a polling engine

Milestone 5: Beyond LibreNMS

• D1
• D2
• D3
• D4

Outcome:

• TowerOps differentiates on topology-aware operations, subscriber impact, distributed polling, and RF intelligence

Best Next Task For Another Agent

If another agent picks this up and needs the single best next task, it should start with:

• F1: replace sequential multi-get with real batched GET support

If that is already done, the next best task is:

• C1: add mempools

If both are done, the next best task is:

• C2: add transceivers / DOM optics

These tasks give the best mix of:

• broad technical leverage
• visible capability improvement
• alignment with the strategic direction described in this document