add preseem integration design and stage 1 plan docs

docs/plans/2026-02-12-preseem-integration-design.md

@@ -0,0 +1,305 @@
+# Preseem API Integration Design
+
+## Overview
+
+Integrate Preseem's QoE (Quality of Experience) data with Towerops' SNMP-based device monitoring to provide network engineers with combined infrastructure health + traffic quality intelligence.
+
+**Primary persona (Phase 1):** Network engineer — planning capacity, upgrading towers, optimizing RF.
+
+**Future personas:** NOC technician (alert correlation), ISP owner/manager (investment prioritization). See `TODO.md` at project root.
+
+## What Preseem Provides
+
+Preseem sits inline on WISP networks and measures per-subscriber, per-AP, per-tower:
+- **QoE metrics**: latency, loss, jitter, throughput (per subscriber and aggregate)
+- **AP capacity scores**: airtime utilization, busy hours, subscriber density
+- **RF metrics**: RSSI, modulation, frequency, channel width
+- **Subscriber data**: plan rates, actual throughput, data usage
+
+Preseem has a Model API (beta, in production use) and REST APIs.
+
+## What Towerops Provides
+
+- SNMP-based device inventory (hardware, firmware, serial numbers)
+- 570+ vendor device profiles
+- Network topology (LLDP/CDP neighbors, ARP discovery)
+- Sensor readings (temperature, voltage, power, signal strength)
+- Interface statistics (traffic counters, errors, discards)
+- MikroTik configuration tracking
+
+## The Value of Combining Both
+
+Towerops knows *about devices* (hardware, config, topology, sensor health).
+Preseem knows *about traffic* (QoE, capacity, subscriber experience).
+
+Together: "This AF5XHD at Tower 3 is running hot (Towerops sensor), serving 45 subscribers (Preseem), with QoE dropping during busy hours (Preseem), and its firmware is 2 versions behind the best-performing instances in your fleet (both)."
+
+---
+
+## Data Model
+
+### Integration Settings
+
+`integrations` — Per-organization, generic for future integrations.
+
+| Column | Type | Notes |
+|--------|------|-------|
+| id | binary_id | PK |
+| organization_id | binary_id | FK to organizations |
+| provider | string | "preseem" (future: "splynx", "sonar", etc.) |
+| enabled | boolean | default false |
+| credentials | encrypted JSONB | API key, base URL (Cloak AES-256-GCM) |
+| sync_interval_minutes | integer | default 10 |
+| last_synced_at | utc_datetime | nullable |
+| last_sync_status | string | "success", "partial", "failed", "never" |
+| timestamps | utc_datetime | |
+
+Unique constraint on `{organization_id, provider}`.
+
+### Preseem Access Points
+
+`preseem_access_points` — Central entity synced from Preseem API.
+
+| Column | Type | Notes |
+|--------|------|-------|
+| id | binary_id | PK |
+| organization_id | binary_id | FK, tenant isolation |
+| preseem_id | string | Preseem's identifier, unique per org |
+| name | string | AP name from Preseem |
+| mac_address | string | for device matching |
+| ip_address | string | for device matching |
+| model | string | for fleet analysis |
+| firmware | string | for fleet analysis |
+| capacity_score | float | Preseem computed score |
+| qoe_score | float | Preseem computed score |
+| rf_score | float | Preseem computed score |
+| busy_hours | integer | hours per day at/near capacity |
+| airtime_utilization | float | percentage |
+| subscriber_count | integer | connected subscribers |
+| device_id | binary_id | nullable FK to devices (matched) |
+| match_confidence | string | "auto_mac", "auto_ip", "auto_hostname", "manual", "ambiguous", "unmatched" |
+| raw_data | map (JSONB) | full Preseem API response |
+| timestamps | utc_datetime | |
+
+Unique constraint on `{organization_id, preseem_id}`.
+
+### Preseem Subscriber Metrics (Time-Series)
+
+`preseem_subscriber_metrics` — Aggregate QoE per AP, stored as time-series.
+
+| Column | Type | Notes |
+|--------|------|-------|
+| id | binary_id | PK |
+| preseem_access_point_id | binary_id | FK |
+| avg_latency | float | ms |
+| avg_jitter | float | ms |
+| avg_loss | float | percentage |
+| avg_throughput | float | Mbps |
+| p95_latency | float | ms, key QoE indicator |
+| subscriber_count | integer | at time of measurement |
+| recorded_at | utc_datetime | hypertable partition key |
+
+### Preseem Sync Logs
+
+`preseem_sync_logs` — Audit trail per sync operation.
+
+| Column | Type | Notes |
+|--------|------|-------|
+| id | binary_id | PK |
+| organization_id | binary_id | FK |
+| integration_id | binary_id | FK |
+| status | string | "success", "partial", "failed" |
+| records_synced | integer | |
+| errors | map (JSONB) | error details |
+| duration_ms | integer | |
+| inserted_at | utc_datetime | |
+
+---
+
+## Device Matching Engine
+
+When sync pulls Preseem APs, a matching pipeline runs against Towerops devices (same org).
+
+### Match Strategy (ordered by confidence)
+
+1. **MAC address** — Normalize (lowercase, strip delimiters), match against `devices.mac_address` and `interfaces.mac_address`. Unique match → `:auto_mac`.
+2. **IP address** — Match against `devices.ip_address`. Unique match → `:auto_ip`.
+3. **Hostname** — Normalize (lowercase, strip domain suffix), match against `devices.name`. Unique match → `:auto_hostname`.
+4. **No match** → `:unmatched`.
+
+### Conflict Handling
+
+If multiple Towerops devices match a single Preseem AP → `:ambiguous`, surfaced for manual review.
+
+### Manual Override
+
+- Link/unlink from integrations page (unmatched/ambiguous list)
+- Link/unlink from device detail page
+- Manual links stored as `:manual`, never overwritten by auto-matching
+
+---
+
+## Analysis Engine
+
+### Individual Device Baselines
+
+For each matched device, compute a rolling 14-day baseline (nightly Oban worker):
+
+- **QoE baseline**: typical qoe_score, p95_latency, subscriber_count (busy vs off-peak)
+- **Capacity baseline**: normal airtime_utilization, busy_hours, throughput ranges
+- **RF baseline**: typical rf_score, Towerops SNMP signal sensors
+
+Stored in `preseem_device_baselines`:
+- Per-metric: mean, stddev, p5, p95
+- Separate busy-hour vs off-peak profiles
+
+When fresh sync data arrives, compare against baseline → flag deviations.
+
+### Fleet Intelligence
+
+Aggregate across all matched devices per org, grouped by device model (manufacturer + model from SNMP profile):
+
+Stored in `preseem_fleet_profiles` keyed by `{organization_id, manufacturer, model}`:
+
+- **Model performance profiles**: avg subscribers, avg QoE per model
+- **Capacity ceilings**: subscriber count where QoE degrades per model
+- **Firmware correlation**: QoE/capacity grouped by firmware version
+- **Configuration patterns**: correlate config differences with performance (using MikroTik backup tracking)
+
+Computed nightly.
+
+---
+
+## Insight Generation & Delivery
+
+### Insight Types
+
+| Insight | Source | Delivery |
+|---------|--------|----------|
+| QoE degradation on specific AP | Baseline deviation | Proactive alert |
+| AP approaching capacity saturation | Baseline trend + fleet ceiling | Proactive alert |
+| Firmware upgrade opportunity | Fleet firmware correlation | Contextual guidance |
+| Frequency/interference conflict | Preseem RF + Towerops topology | Contextual guidance |
+| Device model underperforming fleet avg | Fleet profile comparison | Passive feed |
+| Subscriber growth toward capacity ceiling | Baseline trend + fleet intel | Passive feed |
+| Config drift from best-performing peers | Config tracking + fleet correlation | Passive feed |
+
+### Delivery Channels
+
+1. **Proactive alerts** — Generated as Towerops monitoring checks. Uses existing alerting infrastructure (soft/hard state transitions, flapping detection, notifications).
+
+2. **Contextual guidance** — On device detail page, new "Insights" section when device is matched to a Preseem AP. Dismissable recommendation cards.
+
+3. **Passive insights feed** — New LiveView page under network section. Filterable, sortable list of all insights across the org.
+
+---
+
+## UI Components
+
+### Org Settings > Integrations
+
+- List of available integration providers with enable/disable toggle
+- Per-integration config panel:
+  - Preseem: API key field, optional base URL override
+  - "Test Connection" button (validates API key)
+  - Sync status, last sync time, record counts
+  - Sync history log
+
+### Device Detail > Preseem Tab (when matched)
+
+- Current QoE score, capacity score, RF score
+- Subscriber count and busy hours
+- Trend charts (from preseem_subscriber_metrics time-series)
+- Contextual insights/recommendations
+- Link to Preseem dashboard for this AP
+
+### Unmatched Devices Page (under Integrations)
+
+- List of Preseem APs not yet matched to Towerops devices
+- Ambiguous matches needing resolution
+- Search/select to manually link
+
+### Network Insights Page
+
+- All generated insights across the org
+- Filter by: type, urgency, device, site
+- Sortable by urgency, date, device
+- Bulk dismiss
+
+---
+
+## Sync Architecture
+
+### PreseemSyncWorker (Oban Cron)
+
+Runs every 10 minutes (configurable per org). For each org with enabled Preseem integration:
+
+1. Fetch API key from encrypted credentials
+2. Pull AP list from Preseem Model API
+3. Upsert `preseem_access_points` (by preseem_id)
+4. Pull QoE metrics per AP
+5. Insert `preseem_subscriber_metrics` time-series records
+6. Run device matching pipeline for unmatched/new APs
+7. Log sync result to `preseem_sync_logs`
+
+### PreseemBaselineWorker (Oban Cron, nightly)
+
+1. Compute individual device baselines from 14-day window
+2. Compute fleet profiles grouped by model
+3. Compare current data against baselines
+4. Generate insights for deviations
+
+### Preseem API Client
+
+`Towerops.Preseem.Client` — Uses `:req` (only approved HTTP client).
+
+- `list_access_points/1` — Pull all APs
+- `get_access_point_metrics/2` — QoE metrics for specific AP
+- `test_connection/1` — Validate API key
+
+---
+
+## Context Structure
+
+```
+Towerops.Integrations          — Generic integration CRUD (settings, credentials)
+Towerops.Preseem               — Preseem-specific context
+Towerops.Preseem.Client        — API client (Req-based)
+Towerops.Preseem.Sync          — Sync logic (upsert, metrics insert)
+Towerops.Preseem.DeviceMatcher — Matching pipeline
+Towerops.Preseem.Baseline      — Individual device baselining
+Towerops.Preseem.FleetIntelligence — Fleet-wide model analysis
+Towerops.Preseem.Insights      — Insight generation and storage
+```
+
+---
+
+## Implementation Stages
+
+### Stage 1: Foundation
+- `integrations` schema + migration
+- Org Settings > Integrations UI (generic)
+- Preseem API client with test connection
+- Encrypted credential storage
+
+### Stage 2: Data Sync
+- `preseem_access_points` + `preseem_subscriber_metrics` schemas + migrations
+- `preseem_sync_logs` schema + migration
+- PreseemSyncWorker
+- Device matching engine
+
+### Stage 3: UI — Device Integration
+- Preseem tab on device detail page
+- Unmatched devices management page
+- Manual link/unlink UI
+
+### Stage 4: Analysis Engine
+- `preseem_device_baselines` + `preseem_fleet_profiles` schemas
+- PreseemBaselineWorker (nightly)
+- Fleet intelligence computation
+
+### Stage 5: Insights & Guidance
+- Insight generation from baselines + fleet data
+- Proactive alerts (monitoring check integration)
+- Contextual guidance on device pages
+- Network insights feed page