towerops/docs/librenms-audit/towerops-detection.md

# Towerops Device Detection Algorithm

**Analysis Date**: 2026-02-11
**Towerops Version**: Current main branch
**Key Files**: `lib/towerops/profiles/yaml_profiles.ex`, `priv/profiles/os_detection/*.yaml`

## Overview

Towerops uses a YAML-based detection system adapted from LibreNMS but enhanced with:
- **4-phase matching** (unconditional/conditional × non-generic/generic)
- **ETS caching** with read concurrency optimization
- **Rust NIF MIB resolution** for fast, reliable OID lookups
- **Index templates** to prevent sensor deduplication issues

## Detection Flow

The detection process happens during SNMP discovery:

```
Device SNMP Query
    ↓
discover_system_info() - Collects sysObjectID, sysDescr, sysName
    ↓
select_profile() - Matches against YAML profiles
    ↓
YamlProfiles.match_profile() - 4-Pass LibreNMS-style matching
    ↓
Profile Selected → build_device_info() - Extracts manufacturer, model, firmware
    ↓
Dynamic.identify_device() - Uses profile to extract device metadata
    ↓
discover_sensors() - Loads sensors from profile
```

**Entry Point**: `Towerops.Snmp.Discovery.discover_device/1`

## YAML Profile Structure

Each profile file represents one device OS/type. Located: `priv/profiles/os_detection/*.yaml`

### Example Structure

```yaml
os: routeros                    # Profile identifier (maps to filename)
text: 'MikroTik RouterOS'       # Human-readable vendor name
type: network                   # Device type category
icon: mikrotik                  # UI icon reference
mib_dir: mikrotik               # MIB directory name

discovery:
  - sysObjectID:                # Primary detection: OID matching
      - .1.3.6.1.4.1.14988.1
    sysDescr:                   # Optional: compound matching (OID + string)
      - "RouterOS"
    sysDescr_regex:             # Optional: regex patterns
      - '/RouterOS/i'           # Case-insensitive
    snmpget:                    # Optional: conditional SNMP query
      oid: 'MIKROTIK-MIB::mtxrLicVersion'
      op: '!='                  # Operators: '=', '!=', 'starts', 'contains', 'regex', '>='
      value: ''
    snmpwalk:                   # Optional: walk condition
      oid: 'MIKROTIK-MIB::mtxrInterfaceStatsName'
      op: 'contains'
      value: 'ether'
```

### Profile Composition

- **Multiple detection blocks** per profile (OR logic)
- **Multiple conditions** per block (AND logic)
- **First matching profile** wins
- **Best match selection** by longest OID when multiple blocks match

## 4-Phase Matching Algorithm

`YamlProfiles.match_profile()` implements priority-based matching (lines 653-673):

### Phase 1: Non-Generic Profiles (Unconditional Blocks)

**Goal**: Fast string-based matching for specific vendors

**Checks**:
- All profiles **except** generic fallbacks (`airos`, `freebsd`, `linux`)
- Only detection blocks **without** `snmpget`/`snmpwalk` conditions
- Pure string matching on sysObjectID and sysDescr
- Very fast: <1ms per device

**Example**: Cisco router with known OID prefix

**Code**:
```elixir
# Filter profiles
non_generic = Enum.filter(profiles, &(&1.name not in @generic_os))

# Filter unconditional blocks
Enum.filter(profile.detection_blocks, &(!&1.has_condition))

# Match blocks
Enum.filter(blocks, &block_matches?(&1, sys_object_id, sys_descr, opts, false))
```

### Phase 2: Non-Generic Profiles (Conditional Blocks)

**Goal**: Accurate matching for ambiguous devices requiring SNMP queries

**Checks**:
- Same profiles as Phase 1
- Only blocks **with** `snmpget`/`snmpwalk` conditions
- MIB names pre-resolved at startup (cached)
- Uses actual SNMP queries to verify conditions

**Example**: Ubiquiti AirOS AF60 variant (requires `snmpget: UI-AF60-MIB::af60Role.1 = false`)

**Code**:
```elixir
# Filter conditional blocks
Enum.filter(profile.detection_blocks, &(&1.has_condition))

# Match with conditional evaluation
Enum.filter(blocks, &block_matches?(&1, sys_object_id, sys_descr, opts, true))
```

### Phase 3: Generic OS Profiles (Unconditional)

**Goal**: Catch generic Linux/BSD devices without conditional checks

**Checks**:
- Generic profiles only: `airos`, `freebsd`, `linux`
- Unconditional blocks only
- Matches by sysDescr string

**Example**: Linux device with "Linux" in sysDescr

**Code**:
```elixir
generic = Enum.filter(profiles, &(&1.name in @generic_os))
Enum.filter(profile.detection_blocks, &(!&1.has_condition))
```

### Phase 4: Generic OS Profiles (Conditional)

**Goal**: Final fallback for devices requiring conditional checks

**Checks**:
- Generic profiles with `snmpget`/`snmpwalk` conditions
- Last resort matching

**Example**: Ubiquiti device with Linux + UI-AF60-MIB check

### Best Match Selection

When multiple blocks match within the same phase:

```elixir
Enum.max_by(matching_blocks, &String.length(&1.oid || ""))
```

**Logic**: Longest OID wins (more specific match)

**Example**:
- Block A OID: `.1.3.6.1.4.1.10002` (16 chars)
- Block B OID: `.1.3.6.1.4.1.10002.1.2.3` (27 chars)
- **Winner**: Block B (more specific)

## Detection Block Matching Logic

Each block is evaluated using `block_matches?/5` with four checks:

### 1. OID Check

```elixir
oid_match = block.oid && String.contains?(sys_object_id, block.oid)
```

**Behavior**: Substring matching (not exact equality)

**Rationale**: Device firmware sometimes truncates OID responses. Substring matching handles vendor inconsistencies.

**Example**:
- Block OID: `.1.3.6.1.4.1.10002`
- Device OID: `1.3.6.1.4.1.10002.1.2` (no leading dot)
- **Match**: YES (substring found)

### 2. Condition Mode Check

```elixir
condition_match = block.has_condition == match_conditional
```

**Behavior**: Filters blocks based on current phase

**Purpose**: Prevents `snmpget` blocks from being evaluated in Phase 1 (performance)

### 3. sysDescr Constraints

```elixir
defp block_matches_descr?(block, sys_descr) do
  case {patterns, regexes} do
    {[], []} -> true                                    # No constraints
    _ when is_nil(sys_descr) -> false                  # No sysDescr to match
    _ -> Enum.any?(patterns, fn p ->
           String.contains?(sys_descr, p)
         end) ||
         Enum.any?(regexes, fn r ->
           Regex.match?(r, sys_descr)
         end)
  end
end
```

**Logic**:
- **No constraints**: Block matches if OID matches (sufficient)
- **String patterns**: At least one must match (substring search, case-sensitive)
- **Regex patterns**: At least one must match (case-sensitive or insensitive based on flags)
- **Both types**: Treated as OR (string OR regex)

### 4. Conditional Checks (snmpget/snmpwalk)

#### snmpget Evaluation

```elixir
defp evaluate_snmpget_condition(%{oid: oid, op: op, value: expected}, client_opts) do
  numeric_oid = MibCache.lookup(oid)  # Cached MIB resolution
  case Client.get(client_opts, numeric_oid) do
    {:ok, actual} -> compare_snmp_values(actual, op, expected)
    {:error, _} -> op == "=" && expected == false  # Missing OID = false
  end
end
```

**Behavior**:
- Performs single SNMP GET query
- Resolves MIB symbolic names (e.g., `UI-AF60-MIB::af60Role.1`) to numeric OIDs
- Returns true if comparison succeeds
- **Special case**: If OID doesn't exist, treats as `false` (useful for `= false` checks)

#### snmpwalk Evaluation

```elixir
defp evaluate_snmpwalk_condition(%{oid: oid, op: op, value: expected}, client_opts) do
  numeric_oid = MibCache.lookup(oid)
  case Client.walk(client_opts, numeric_oid) do
    {:ok, results} when is_map(results) ->
      values = Map.values(results)
      Enum.any?(values, fn value -> compare_snmp_values(value, op, expected) end)
    {:error, _} -> false
  end
end
```

**Behavior**:
- Walks OID subtree, returns all values found
- Matches if **ANY** value satisfies condition
- Useful for checking table-based conditions

#### Comparison Operators

```elixir
"="       → exact string match
"!="      → string inequality
"starts"  → String.starts_with?
"contains"→ String.contains? (default)
"regex"   → Regex.match? (compiles pattern dynamically)
">="      → numeric comparison (with fallback to string)
```

## ETS Caching Strategy

### Profile ETS Table

**Table**: `:yaml_profiles`

**Initialized in**: `YamlProfiles.init/0` (GenServer)

```elixir
:ets.new(@table, [:named_table, :set, :public, read_concurrency: true])
```

**Cache keys** (composite indexing):
- `{:profile, name}` → Full profile map
- `{:oid, numeric_oid}` → Profile name (fast OID lookup)
- `{:pattern, string}` → Profile name (sysDescr pattern lookups)

**Why read_concurrency: true?**
- Multiple discovery processes read profiles simultaneously
- Write occurs only at startup or reload
- Extremely low contention (99% reads, 1% writes)

**Lookup performance**:
```elixir
def get_profile(name) do
  case :ets.lookup(@table, {:profile, name}) do
    [{_, profile}] -> profile  # O(1) constant time
    [] -> nil
  end
end
```

### MIB Cache ETS Table

**Table**: `:mib_cache`

**Stores**: MIB symbolic names → numeric OID strings
- `"SNMPv2-MIB::sysDescr"` → `"1.3.6.1.2.1.1.1"`
- `"UBNT-AirFIBER-MIB::fwVersion.1"` → `"1.3.6.1.4.1.10002.1.2.3.1.0"`

**Pre-resolution at startup**:

```elixir
# In YamlProfiles.init:
mib_names = extract_mib_names_from_profiles(raw_profiles)
MibCache.pre_resolve(mib_names)

# In MibCache.handle_cast:
Enum.reduce(mib_names, 0, fn mib_name, acc ->
  case resolve_oid_with_timeout(mib_name) do
    {:ok, oid} -> :ets.insert(@table, {mib_name, oid}); acc + 1
    {:error, _} -> Logger.warning("Failed to resolve #{mib_name}"); acc
  end
end)
```

**Timeout handling**: 5-second timeout per MIB name (prevents hanging on missing MIBs)

**Runtime fallback**:
```elixir
def lookup(mib_name) do
  case :ets.lookup(@table, mib_name) do
    [{^mib_name, oid}] -> oid              # Cache hit
    [] -> resolve_and_cache(mib_name)      # Cache miss - resolve at runtime
  end
end
```

## MIB Name Resolution via Rust NIF

### ToweropsNative Module

**Location**: `lib/towerops_native.ex` (wrapper for compiled NIF)

**Implementation**: C NIF that calls `libnetsnmp`
- Binary: `priv/towerops_nif` (compiled at build time)
- Wraps net-snmp library functions

### Resolution Process

```
MIB Name → ToweropsNative.resolve_oid/1 → snmptranslate → Numeric OID
```

**Call sequence**:
```elixir
# In conditional matching (Phase 2/4)
numeric_oid = MibCache.lookup("UBNT-AirFIBER-MIB::fwVersion.1")
  ↓
# Cache hit? Return immediately (<1µs)
# Cache miss? Fall back to:
ToweropsNative.resolve_oid("UBNT-AirFIBER-MIB::fwVersion.1")
  ↓
# C NIF calls internal net-snmp functions
# Returns: "1.3.6.1.4.1.10002.1.2.3.1.0"
```

### Performance Characteristics

- **MIB Loading**: ~1-2 seconds (one-time at app startup)
- **In-memory lookup**: ~1-20 microseconds (after pre-resolution)
- **Throughput**: ~50,000-100,000 resolutions per second per core
- **Cache hit rate**: ~95%+ for profiles (pre-resolved at startup)

### System Dependencies

**Development (macOS)**:
```bash
brew install net-snmp  # keg-only, requires PKG_CONFIG_PATH setup
```

**Production (Docker)**:
```dockerfile
apt-get install libsnmp-dev snmp-mibs-downloader
```

**Runtime requirement**: `snmptranslate` command in PATH

### Test Environment

- MIB resolution **disabled** in test mode (avoids hanging)
- Tests must stub `ToweropsNative` if they need MIB behavior

## YAML Profile Processing Pipeline

### Load Phase (`load_all_profiles/0`)

```elixir
# 1. Find all YAML files
yaml_files = Path.wildcard("priv/profiles/os_detection/*.yaml")

# 2. Parse each YAML file
{name, file, load_profile(profiles_path, name)}
  # read_yaml(file) → YamlElixir.read_from_string(content)
  # build_profile(name, detection, discovery)

# 3. Index into ETS
:ets.insert(@table, {{:profile, name}, profile})
:ets.insert(@table, {{:oid, oid}, name})
:ets.insert(@table, {{:pattern, pattern}, name})

# 4. Extract and pre-resolve MIB names
mib_names = extract_mib_names_from_profiles(raw_profiles)
MibCache.pre_resolve(mib_names)
```

### Profile Struct Building (`build_profile/3`)

Raw YAML is transformed into optimized detection structure:

```elixir
%{
  name: "airos-af",
  vendor: "Ubiquiti AirFiber",

  # Primary detection OID (first OID from first block)
  detection_oid: "1.3.6.1.4.1.10002.1",

  # Simple string patterns for fallback matching
  detection_patterns: ["Linux"],
  detection_descr_patterns: ["Linux"],
  detection_descr_regex: [compiled_regex],

  # All detection blocks with structured info
  detection_blocks: [
    %{
      oid: "1.3.6.1.4.1.10002.1",
      has_snmpget: true,
      has_snmpwalk: false,
      has_condition: true,
      descr_patterns: ["Linux"],
      descr_regex: [],
      snmpget: %{oid: "UBNT-AirFIBER-MIB::fwVersion.1", op: "!=", value: "false"},
      snmpwalk: nil
    }
  ],

  # Device metadata extraction
  device_oids: %{
    firmware_version: "SNMPv2-MIB::sysDescr.0",
    serial_number: "SOME-MIB::serialNumber.0"
  },
  hardware_regex: compiled_regex,

  # Sensor definitions for SNMP polling
  sensor_oids: [...],
  table_sensor_oids: [...],
  processor_oids: [...],
  count_sensor_oids: [...],
  state_sensor_oids: [...]
}
```

## Index Templates (Sensor Deduplication)

### Problem

MikroTik firewall sensors (total/ipv4/ipv6 connections) shared `sensor_index: "count_1"` because each walked separately with `Enum.with_index(1)`.

### Solution: {{ $index }} Placeholders

For table-based sensors, YAML uses `{{ $index }}` placeholder:

```yaml
modules:
  sensors:
    count:
      data:
        - oid: 'MIKROTIK-MIB::mtxrGaugeName'
          num_oid: '.1.3.6.1.4.1.14988.1.1.3.10.{{ $index }}'
          index: 'count_{{ $index }}'  # custom sensor_index format
```

**Runtime substitution**:
```elixir
# OID walk returns: {"1.3.6.1.4.1.14988.1.1.3.10.1" => 100, ...}
# Extract index: "1"
# Substitute: "count_{{ $index }}" → "count_1"
```

**Prevents**: All sensors getting same index when walking different OID branches

## Complete Detection Flow Example

### Scenario: Ubiquiti AirFiber device

**Step 1: SNMP Query**
```
sysObjectID: "1.3.6.1.4.1.10002.1.2.1"
sysDescr: "Linux airfiber 5.4.0 #1 armv5tejl GNU/Linux"
```

**Step 2: Phase 1 - Non-Generic Unconditional**
```
Profiles checked: [all except airos, freebsd, linux]
Blocks checked: [blocks WITHOUT snmpget/snmpwalk]

airos-af profile, block 1:
  ✓ OID check: "10002.1" in "10002.1.2.1"? YES
  ✓ Condition mode: no snmpget? YES (unconditional)
  ✓ sysDescr pattern "Linux" in sysDescr? YES
  → MATCH FOUND!

Best match: airos-af (block 1 OID length: 10 chars)
→ Return profile immediately
```

**Step 3: Device Identification**
```
Dynamic.identify_device(profile, client_opts, system_info):
  1. Fetch device_oids from profile
     - firmware_version → SNMP GET
     - serial_number → SNMP GET

  2. Extract hardware from sysDescr_regex
     - Pattern: "/armv5tejl/" matches
     - Result: "5.4.0"

  3. Return device metadata
```

**Step 4: Sensor Discovery**
```
Dynamic.discover_sensors(profile, client_opts):
  1. Base sensors via ENTITY-SENSOR-MIB
  2. Profile scalar sensors (direct OID queries)
  3. Profile table sensors (SNMP walks with index templates)
  4. Return sensor list
```

## Comparison with LibreNMS

| Feature | LibreNMS | Towerops |
|---------|----------|----------|
| Profile source | YAML files | YAML files |
| Number of phases | 2 (fast/slow) | 4 (unconditional/conditional × non-generic/generic) |
| OID matching | Prefix match (startsWith) | Substring match (contains) |
| Conditional checks | snmpget in PHP | Pre-resolved MIB cache (ETS) |
| Generic OS handling | Deferred to pass 2 | Separate phases 3-4 |
| MIB resolution | PHP SNMP module (unreliable) | Rust NIF + libnetsnmp (fast, reliable) |
| Caching | In-memory PHP arrays | ETS with read_concurrency + pre-warming |
| Index deduplication | None | `{{ $index }}` template substitution |
| Performance (lookups) | Depends on PHP | ~microseconds per lookup |
| Configuration loading | ~1-2 seconds | ~1-2 seconds |

## Key Design Decisions

### Decision 1: 4-Phase Matching vs 2-Phase

**Why?** Separates conditional checks into separate phases instead of mixing with fast checks. Clearer separation of concerns.

**Benefit**: Easier to debug, more predictable performance

### Decision 2: Pre-resolved MIB Cache

**Why?** Conditional snmpget checks need MIB names, but net-snmp resolution is slow. Pre-resolving at startup (1-2 seconds) enables microsecond lookups during matching.

**Benefit**: Fast conditional matching without repeated MIB resolution

### Decision 3: ETS with read_concurrency

**Why?** Multiple agent discovery processes query profiles simultaneously during scaling. Read-heavy workload (99% reads, 1% startup writes).

**Benefit**: Concurrent reads without locking

### Decision 4: OID Substring Matching

**Why?** Device firmware sometimes truncates OID responses. Substring matching is more forgiving than exact equality.

**Benefit**: Handles vendor inconsistencies and leading dot variations

### Decision 5: Index Templates

**Why?** Some sensors (MikroTik) walk in ways that create duplicate indices.

**Benefit**: Prevents sensor deduplication errors

## Debugging Profile Matching

### Enable Logging

```elixir
# In config:
config :towerops, log_level: :debug

# In iex:
Logger.configure(level: :debug)
```

### Common Issues

**Profile not matching?**
1. Check sysObjectID in device output
2. Verify OID in YAML (leading dot optional)
3. Check sysDescr patterns/regexes
4. Test conditional snmpget manually

**snmpget condition fails?**
1. Test MIB resolution: `ToweropsNative.resolve_oid("MIB::object")`
2. Test SNMP query manually: `snmpget -v2c -c public 192.168.1.1 1.3.6.1.4.1.10002...`
3. Check expected value (string comparison, not numeric)

**Sensors not discovering?**
1. Verify `modules.sensors` section in discovery YAML
2. Check base_oid contains `{{ $index }}` for table sensors
3. Verify OID is walkable

## Current Status

**Total Profiles**: 784 OS detection profiles in `priv/profiles/os_detection/`

**Key Files**:
- `lib/towerops/profiles/yaml_profiles.ex` - Profile matching logic
- `lib/towerops/profiles/mib_cache.ex` - MIB resolution cache
- `lib/towerops/snmp/profiles/dynamic.ex` - Device identification and sensor discovery
- `lib/towerops_native.ex` - Rust NIF wrapper for MIB resolution