From 2f7f6370e3547e848a541e6c3e84718e7d8045a5 Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Sun, 18 Jan 2026 09:24:38 -0600 Subject: [PATCH] docs: add comprehensive profile management documentation Document the complete workflow for exporting and importing device profiles: - Local export process with mix task - Production import via API endpoint - Profile structure and database schema - Troubleshooting guide - Security and authentication details File location: PROFILES.md (intentionally not in API docs) --- PROFILES.md | 244 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 244 insertions(+) create mode 100644 PROFILES.md diff --git a/PROFILES.md b/PROFILES.md new file mode 100644 index 00000000..eb411bf9 --- /dev/null +++ b/PROFILES.md @@ -0,0 +1,244 @@ +# Device Profile Management + +This document describes how to manage SNMP device profiles in Towerops. Device profiles define how to discover and monitor different types of network equipment via SNMP. + +## Overview + +Towerops uses a database-driven profile system that supports 671+ device types dynamically without code changes. Profiles are imported from external YAML definitions and stored in the database. + +## Architecture + +**Profile Discovery Flow:** +1. Device performs SNMP discovery +2. System reads sysDescr and sysObjectID from device +3. Matches against detection rules in database +4. Uses matched profile to discover sensors, CPU, memory, etc. +5. Saves discovered data for monitoring and graphing + +**Profile Components:** +- **Detection Rules**: sysObjectID prefix matching, sysDescr regex patterns +- **Sensor Definitions**: Temperature, voltage, current, power, fan speed, state sensors +- **Processor Definitions**: CPU usage monitoring (converted to percent sensors) +- **Memory Pool Definitions**: Memory usage monitoring +- **OS Definitions**: Firmware version, serial number, location data + +## Exporting Profiles Locally + +Use the `mix export_profiles` task to export profiles from an external source to JSON format: + +```bash +# Export all profiles +mix export_profiles --librenms-path ~/dev/librenms --output profiles.json + +# Export specific profiles only +mix export_profiles --librenms-path ~/dev/librenms --profiles epmp,unifi,airos-af --output profiles.json +``` + +**Options:** +- `--librenms-path PATH` - Path to external source installation (required) +- `--output PATH` - Output JSON file path (default: profiles.json) +- `--profiles LIST` - Comma-separated list of specific profiles to export (optional) + +**Output:** +The task generates a JSON file containing all profile data (detection rules, sensor definitions, etc.) suitable for uploading to production. + +## Importing Profiles to Production + +**Requirements:** +- Superuser account (user with `is_superuser = true`) +- Active browser session +- Session cookie from browser + +### Step 1: Get Session Cookie + +1. Log in to Towerops as a superuser +2. Open browser Developer Tools (F12) +3. Navigate to: Application > Cookies > https://towerops.net +4. Copy the value of `_towerops_key` cookie + +### Step 2: Upload Profiles + +```bash +curl -X POST https://towerops.net/api/v1/admin/profiles/import \ + -H "Cookie: _towerops_key=YOUR_SESSION_COOKIE" \ + -H "Content-Type: application/json" \ + -d @profiles.json +``` + +**Response:** +```json +{ + "status": "queued", + "job_id": "uuid", + "profile_count": 15, + "message": "Import job queued successfully. 15 profiles will be imported in the background." +} +``` + +### Step 3: Monitor Import Progress + +The import runs as a background job in the `maintenance` queue. Monitor progress via: + +1. **Server logs:** + ```bash + kubectl logs -n towerops deployment/towerops | grep "profile import" + ``` + +2. **Exq dashboard:** + Navigate to `/dashboard` and check the Maintenance queue + +**Log messages:** +- `Starting profile import job with N profiles` - Job started +- `[] Successfully imported profile: ` - Profile imported +- `[] Failed to import profile : ` - Import error +- `[] Import complete: N succeeded, M failed` - Job finished + +## Profile Structure + +### Detection YAML (os_detection/*) +```yaml +os: epmp +text: Cambium epmp +type: wireless +group: cambium +icon: cambium +discovery: + - sysObjectID: + - ".1.3.6.1.4.1.17713.21" +``` + +### Discovery YAML (os_discovery/*) +```yaml +mib: CAMBIUM-PMP80211-MIB +modules: + os: + version: CAMBIUM-PMP80211-MIB::cambiumCurrentuImageVersion.0 + serial: CAMBIUM-PMP80211-MIB::cambiumEPMPMSN.0 + + processors: + data: + - oid: sysCPUUsage + num_oid: ".1.3.6.1.4.1.17713.21.2.1.64.{{ $index }}" + type: cambium-cpu + precision: 10 + + sensors: + temperature: + data: + - oid: boardTemp + num_oid: ".1.3.6.1.4.1.17713.21.1.2.1.0" + descr: Board Temperature + precision: 10 +``` + +## Database Schema + +**device_profiles** - Main profile table +- `os` - Operating system identifier (e.g., "epmp", "unifi") +- `text` - Display name (e.g., "Cambium epmp") +- `type` - Device type (wireless, switch, router, etc.) +- `group` - Manufacturer group (cambium, ubiquiti, cisco, etc.) +- `priority` - Matching priority (lower = higher priority) +- `enabled` - Whether profile is active + +**profile_detection_rules** - Device detection patterns +- `rule_type` - "sysObjectID", "sysDescr", "sysDescr_regex" +- `pattern` - OID prefix or regex pattern +- `value` - Exact match value + +**profile_sensor_definitions** - Sensor discovery definitions +- `sensor_class` - temperature, voltage, current, power, fanspeed, humidity, dbm, snr, percent, count, frequency, state +- `oid` - Symbolic OID name +- `num_oid` - Numeric OID (e.g., ".1.3.6.1.4.1.17713.21.2.1.36.{{ $index }}") +- `descr` - Sensor description +- `divisor` - Value divisor for unit conversion +- `precision` - Number of decimal places (stored in divisor field) + +**profile_processor_definitions** - CPU discovery definitions +- `oid` - Symbolic OID name +- `num_oid` - Numeric OID +- `precision` - Divisor for CPU percentage (e.g., 10 means divide by 10) +- `type` - Processor type identifier + +## Force Replacement + +All profile imports use force replacement - existing profiles with the same `os` name are deleted before importing. This ensures clean updates without conflicts. + +## Troubleshooting + +**Import fails immediately:** +- Check you're logged in as a superuser (`is_superuser = true`) +- Verify session cookie is correct and not expired +- Check JSON format is valid: `jq . profiles.json` + +**Profiles import but discovery doesn't work:** +- Check profile detection rules match device sysObjectID/sysDescr +- Verify sensor OIDs are numeric (not symbolic MIB names) +- Check logs for SNMP walk errors +- Test SNMP manually: `snmpwalk -v2c -c public ` + +**Sensor indices conflict:** +- Sensor indices must be unique per device +- Scalar sensors (index .0) use descriptive names (e.g., "dfs_detected_count") +- Table sensors use class prefix (e.g., "temperature_1", "voltage_2") + +**CPU not showing:** +- Processor definitions are converted to "percent" type sensors +- Check divisor/precision is correct (usually 10 or 100) +- Verify num_oid is accessible via SNMP + +## Examples + +### Export ePMP and UniFi profiles +```bash +mix export_profiles \ + --librenms-path ~/dev/librenms \ + --profiles epmp,unifi \ + --output production_profiles.json +``` + +### Import to production +```bash +# Get session cookie from browser first +SESSION_COOKIE="SFMyNTY.g3QAAAACbQ..." + +curl -X POST https://towerops.net/api/v1/admin/profiles/import \ + -H "Cookie: _towerops_key=${SESSION_COOKIE}" \ + -H "Content-Type: application/json" \ + -d @production_profiles.json +``` + +### Check import status +```bash +kubectl logs -n towerops deployment/towerops --tail=100 | grep "Import complete" +``` + +## Related Files + +- `lib/mix/tasks/export_profiles.ex` - Export task +- `lib/mix/tasks/import_profiles.ex` - Local import task (for development) +- `lib/towerops/device_profiles/importer.ex` - Profile import logic +- `lib/towerops/snmp/profiles/dynamic.ex` - Dynamic profile discovery engine +- `lib/towerops/workers/profile_import_worker.ex` - Background import worker +- `lib/towerops_web/controllers/api/v1/profiles_controller.ex` - API endpoint +- `lib/towerops_web/plugs/require_superuser.ex` - Superuser authorization + +## Security + +- Profile import requires superuser authentication +- Uses browser session cookies (not API tokens) +- Runs in background to prevent timeouts +- Validates profile data before importing +- Logs all import operations + +## Supported Profile Count + +As of January 2026, the system supports **671+ device profiles** including: +- Cambium ePMP, PMP, cnPilot +- Ubiquiti AirFiber, airMAX, UniFi +- Cisco IOS, IOS-XE, NX-OS +- MikroTik RouterOS +- Juniper JunOS +- And many more... + +New profiles can be added without code changes by exporting and importing.