towerops/docs/ARCHITECTURE.md

# TowerOps Architecture

> Network infrastructure monitoring platform for ISPs/WISPs built with Phoenix LiveView, Oban, and TimescaleDB.

---

## 1. Project Structure

```
towerops/
├── assets/                    # Frontend: JS, CSS, Tailwind
├── config/                    # Runtime & env config (dev/test/prod/runtime)
├── c_src/                     # C NIFs (SNMP native interop)
├── docs/                      # Documentation
├── e2e/                       # End-to-end tests (Playwright)
├── flux/                      # FluxCD / GitOps manifests
├── k8s/                       # Kubernetes manifests
├── lib/
│   ├── mix/                   # Custom Mix tasks
│   ├── snmpkit/               # SNMP toolkit (native extension)
│   ├── towerops/              # Core business logic (contexts, schemas, workers)
│   └── towerops_web/          # Web layer (LiveView, controllers, channels, GraphQL)
├── mibs/                      # SNMP MIB files
├── priv/
│   ├── repo/migrations/       # Ecto migrations
│   ├── static/                # Static assets
│   └── gettext/               # i18n translations
├── rel/                       # Release configuration
├── scripts/                   # Operational scripts
├── test/                      # Test suite
├── towerops/                  # Agent-related files
└── vendor/                    # Vendored dependencies
```

### Key `lib/towerops/` Subdirectories

| Directory | Purpose |
|-----------|---------|
| `accounts/` | User, auth tokens, TOTP, recovery codes, sessions, consent |
| `admin/` | Superuser audit logs |
| `agents/` | Remote polling agent tokens & assignments |
| `alerts/` | Alert schema and queries |
| `api_tokens/` | API token management (hashed bearer tokens) |
| `billing/` | Stripe integration, usage metering |
| `capacity/` | Interface throughput & utilization calculations |
| `config_changes/` | MikroTik config change event tracking |
| `contexts/` | Shared context helpers |
| `devices/` | Device schema, backups, firmware, events |
| `ecto_types/` | Custom Ecto types (IP address, MAC address, SNMP OID) |
| `gaiia/` | Gaiia billing/CRM integration (accounts, subscriptions, impact analysis) |
| `geoip/` | GeoIP lookup (blocks, locations) |
| `integrations/` | Integration schema (multi-provider) |
| `job_monitoring/` | Oban job health events |
| `maintenance/` | Maintenance windows |
| `mobile_sessions/` | Mobile app sessions & QR login |
| `monitoring/` | Check definitions & check results |
| `netbox/` | NetBox DCIM sync |
| `on_call/` | Schedules, escalation policies, incidents, notifications |
| `organizations/` | Org, membership, invitations, policies |
| `pagerduty/` | PagerDuty client & notifier |
| `preseem/` | Preseem QoE integration (APs, baselines, fleet intelligence) |
| `profiles/` | SNMP device profiles (OID mappings) |
| `proto/` | Protobuf definitions (agent communication) |
| `security/` | Brute-force protection, IP blocks/whitelists |
| `settings/` | Global application settings (key-value store) |
| `sites/` | Site schema & tree structure |
| `snmp/` | SNMP device data (interfaces, sensors, readings, wireless, topology) |
| `sonar/` | Sonar billing integration |
| `splynx/` | Splynx billing integration |
| `topology/` | Network topology discovery (LLDP, CDP, ARP, MAC) |
| `visp/` | VISP billing integration |
| `weather/` | Weather observations & alerts per site |
| `workers/` | All Oban background workers |

### Key `lib/towerops_web/` Subdirectories

| Directory | Purpose |
|-----------|---------|
| `channels/` | Phoenix Channels (agent WebSocket) |
| `components/` | Reusable UI components (breadcrumbs, tables, forms) |
| `controllers/` | Traditional controllers (auth, webhooks, health, API) |
| `graphql/` | Absinthe GraphQL schema, types, resolvers, subscriptions |
| `helpers/` | View helpers (time formatting, status display) |
| `live/` | All LiveView pages |
| `plugs/` | Plug middleware (auth, rate limiting, security headers) |

---

## 2. Data Model

All primary keys are UUIDs (`:binary_id`). Multi-tenancy is organization-scoped.

### Core Entities

```
┌──────────────┐     ┌──────────────────┐     ┌──────────────┐
│     User     │────<│   Membership     │>────│ Organization │
│              │     │ (role-based)     │     │              │
│ email        │     │ role: owner|     │     │ name, slug   │
│ hashed_pass  │     │  admin|executive │     │ subscription │
│ totp_secret  │     │  |technician|    │     │ snmp_config  │
│ is_superuser │     │  member|viewer   │     │ mikrotik_cfg │
│ timezone     │     │ is_default       │     │ stripe_*     │
│ default_org  │     └──────────────────┘     │ alert_routing│
└──────────────┘                               └──────┬───────┘
                                                      │
                              ┌────────────────┬──────┴──────┬───────────────┐
                              │                │             │               │
                        ┌─────┴─────┐   ┌──────┴──────┐ ┌───┴────┐  ┌──────┴──────┐
                        │   Site    │   │   Device    │ │ Alert  │  │ Integration │
                        │           │   │             │ │        │  │             │
                        │ name      │   │ name        │ │ type   │  │ provider    │
                        │ lat/lng   │   │ ip_address  │ │ sev    │  │ credentials │
                        │ address   │   │ status      │ │ msg    │  │ sync_status │
                        │ snmp_cfg  │   │ snmp_cfg    │ │ device │  │ (encrypted) │
                        │ parent_id │   │ mikrotik_cfg│ │ check  │  └─────────────┘
                        │ agent_tok │   │ site_id     │ └────────┘
                        └───────────┘   │ agent_assign│
                                        │ check_intv  │
                                        └─────────────┘
```

### On-Call / Incident Management

```
Schedule ──has_many──> Layer ──has_many──> LayerMember ──belongs_to──> User
    │
    └──has_many──> Override (user, start_at, end_at)

EscalationPolicy ──has_many──> EscalationRule ──has_many──> EscalationTarget
    │                              │                            │
    │                              │ position, delay_minutes    ├── schedule_id
    │                              └────────────────────────    └── user_id
    │
Incident ──belongs_to──> Alert
    │                     EscalationPolicy
    │                     Organization
    └──has_many──> Notification ──belongs_to──> User
```

### SNMP Data Model

```
Device (towerops) ──1:1──> Snmp.Device
    │
    ├── Snmp.Interface ──> InterfaceStat (time-series)
    ├── Snmp.Sensor ──> SensorReading (time-series)
    ├── Snmp.StateSensor
    ├── Snmp.Processor ──> ProcessorReading
    ├── Snmp.Storage ──> StorageReading
    ├── Snmp.WirelessClient ──> WirelessClientReading
    ├── Snmp.Neighbor (LLDP/CDP)
    ├── Snmp.ArpEntry
    ├── Snmp.Vlan
    └── Snmp.PhysicalEntity
```

### Billing / CRM Schemas

- **Gaiia**: `Account`, `BillingSubscription`, `NetworkSite`, `InventoryItem`, `DeviceSubscriberLink`
- **Preseem**: `AccessPoint`, `SubscriberMetric`, `FleetProfile`, `DeviceBaseline`, `Insight`, `SyncLog`
- **Billing**: `StripeWebhookEvent`

### Other Schemas

| Schema | Table | Purpose |
|--------|-------|---------|
| `AgentToken` | `agent_tokens` | Remote polling agent credentials |
| `AgentAssignment` | `agent_assignments` | Device → agent mapping |
| `ApiToken` | `api_tokens` | REST/GraphQL API bearer tokens |
| `BrowserSession` | `browser_sessions` | Active login sessions |
| `LoginAttempt` | `login_attempts` | Auth attempt audit log |
| `UserConsent` | `user_consents` | Privacy/terms consent records |
| `PolicyVersion` | `policy_versions` | Privacy policy versions |
| `UserTotpDevice` | `user_totp_devices` | Multi-device TOTP |
| `UserRecoveryCode` | `user_recovery_codes` | MFA recovery codes |
| `MaintenanceWindow` | `maintenance_windows` | Scheduled maintenance |
| `ConfigChangeEvent` | `config_change_events` | MikroTik config diffs |
| `DeviceFirmwareHistory` | `device_firmware_histories` | Firmware change tracking |
| `FirmwareRelease` | `firmware_releases` | Known firmware versions |
| `BackupRequest` | `backup_requests` | MikroTik backup queue |
| `MikrotikBackup` | `mikrotik_backups` | Stored config backups |
| `Check` | `checks` | Monitoring check definitions |
| `CheckResult` | `check_results` | Check execution results |
| `MonitoringCheck` | `monitoring_checks` | Check configuration |
| `DeviceLink` | `device_links` | Discovered topology links |
| `DeviceLinkEvidence` | `device_link_evidence` | Link confidence scores |
| `DeviceNeighbor` | `device_neighbors` | Raw neighbor data |
| `ApplicationSetting` | `application_settings` | Global config KV store |
| `AuditLog` | `audit_logs` | Admin action audit trail |
| `IpBlock` | `ip_blocks` | Blocked IPs |
| `IpWhitelist` | `ip_whitelists` | Whitelisted IPs |
| `MobileSession` | `mobile_sessions` | Mobile app sessions |
| `QrLoginToken` | `qr_login_tokens` | QR-based mobile auth |
| `Weather.Observation` | `weather_observations` | Site weather data |
| `Weather.Alert` | `weather_alerts` | Active weather alerts |
| `GeoIp.Block` / `Location` | `geoip_*` | IP geolocation |

---

## 3. Context Modules

### `Towerops.Accounts`
User management, authentication, TOTP/MFA, password management, email changes, session tracking.

```elixir
# User lookup
def get_user_by_email(email)
def get_user_by_email_and_password(email, password)
def get_user(id)
def get_user!(id)

# Registration
def register_user(attrs)
def register_user_with_organization(attrs)
def change_user_registration(user, attrs \\ %{})

# TOTP / MFA
def generate_totp_secret()
def generate_totp_uri(user, secret)
def generate_totp_qr_code(user, secret)
def verify_totp(secret, code)
def enable_totp(user, secret)
def totp_enabled?(user)
def verify_user_totp(user, code)
def list_user_totp_devices(user_id)
def create_totp_device(user_id, device_name)
def verify_user_totp_any_device(user, code)
def delete_totp_device(device_id, user_id)
def rename_totp_device(device_id, user_id, new_name)
def generate_recovery_codes(user_id)
def verify_recovery_code(user_id, code)
def count_unused_recovery_codes(user_id)
def verify_user_mfa(user, code)

# Profile & Settings
def change_user_profile(user, attrs \\ %{})
def update_user_profile(user, attrs)
def sudo_mode?(user, minutes \\ -10)
def grant_sudo_mode(user)

# Email & Password
def change_user_email(user, attrs \\ %{}, opts \\ [])
def update_user_email(user, token)
def change_user_password(user, attrs \\ %{}, opts \\ [])

# Sessions (+ more token functions not listed)
```

### `Towerops.Organizations`
Multi-tenant org management, memberships, invitations, SNMP/agent propagation.

```elixir
def list_user_organizations(user_id)
def user_has_access?(user_id, organization_id)
def get_organization!(id)
def get_organization_by_slug!(slug)
def create_organization(attrs, user_id, opts \\ [])
def update_organization(organization, attrs)
def delete_organization(organization)
def change_organization(organization, attrs \\ %{})

# Memberships
def get_membership(organization_id, user_id)
def list_organization_memberships(organization_id)
def list_organization_members(organization_id)
def remove_member(organization_id, user_id)
def update_member_role(organization_id, user_id, new_role)
def create_membership(attrs)
def get_default_membership(user_id)
def set_default_organization(user_id, organization_id)
def list_organization_notification_recipients(organization_id)

# Invitations
def list_pending_invitations(organization_id)
def create_invitation(attrs)
def get_invitation_by_token(token)
def accept_invitation(invitation, user_id)
def delete_invitation(invitation)

# Config propagation
def apply_snmp_config_to_all_equipment(organization_id)
def apply_agent_to_all_equipment(organization_id)
def clear_all_site_assignments(organization_id)

# Billing lookups
def get_organization_by_stripe_customer_id(stripe_customer_id)
def list_organizations_with_active_subscriptions()
```

### `Towerops.Devices`
Device CRUD, SNMP/MikroTik config resolution (cascading: org → site → device), monitoring.

```elixir
def list_site_devices(site_id)
def list_organization_devices(organization_id, filters \\ %{})
def count_organization_devices(organization_id)
def search_devices(organization_id, query)
def get_device_status_counts(organization_id)
def list_monitored_devices()
def list_snmp_enabled_devices()
def list_mikrotik_devices_with_api()
def get_device(id)
def get_device!(id)
def get_device_with_details(id)
def get_device_by_ip(ip_address, site_id, exclude_id \\ nil)
def get_snmp_config(device)          # Resolves cascading config
def get_mikrotik_config(device)      # Resolves cascading config
def get_snmpv3_config(device)        # Resolves cascading config
def resolve_agent_token_id(device)   # Device → site → org → global fallback
def propagate_site_snmpv3_change(site_id, attrs)
def propagate_organization_mikrotik_change(organization_id, attrs)
# ... CRUD, reordering, etc.
```

### `Towerops.Sites`
Site CRUD, hierarchical tree structure, config propagation.

```elixir
def list_organization_sites(organization_id)
def list_root_sites(organization_id)
def list_child_sites(parent_site_id)
def build_site_tree(organization_id)
def list_sites_with_coordinates()
def search_sites(organization_id, query)
def get_site!(id)
def create_site(attrs)
def update_site(site, attrs)
def delete_site(site)
def apply_snmp_config_to_all_equipment(site_id)
def apply_agent_to_all_equipment(site_id)
def reorder_site(site_id, new_position)
```

### `Towerops.Alerts`
Alert lifecycle: creation, acknowledgement, resolution, Gaiia impact analysis.

```elixir
def create_alert(attrs)
def list_devices_alerts(device_id, limit \\ 100)
def list_organization_active_alerts(organization_id)
def list_organization_alerts(organization_id, filters)
def count_active_alerts(organization_id)
def get_alert!(id)
def get_alert(id)
def acknowledge_alert(alert, user_id)
def resolve_alert(alert)
def resolve_alert_silent(alert)         # No PubSub broadcast
def acknowledge_alert_silent(alert)     # No PubSub broadcast
def has_active_alert?(device_id, alert_type)
def get_active_alert(device_id, alert_type)
def has_active_check_alert?(check_id)
def resolve_check_alerts(check_id)
def store_gaiia_impact(alert, impact_data)
```

### `Towerops.Integrations`
Generic integration CRUD for all billing/CRM providers.

```elixir
def list_integrations(organization_id)
def get_integration(organization_id, provider)
def create_integration(organization_id, attrs)
def update_integration(integration, attrs)
def update_sync_status(integration, status, message \\ nil)
def delete_integration(integration)
def get_billing_summary(organization_id)
def list_enabled_integrations(provider)
def update_billing_totals(integration, subscriber_count, total_mrr)
```

**Valid providers:** `preseem`, `gaiia`, `pagerduty`, `netbox`, `sonar`, `splynx`, `visp`

### `Towerops.OnCall`
On-call schedules, escalation policies, incidents.

```elixir
# Schedules
def list_schedules(organization_id)
def get_schedule!(id) / get_schedule(id)
def create_schedule(attrs) / update_schedule(schedule, attrs) / delete_schedule(schedule)
def create_layer(attrs) / update_layer(layer, attrs) / delete_layer(layer)
def add_layer_member(attrs) / remove_layer_member(member)
def create_override(attrs) / delete_override(override)
def who_is_on_call(schedule_id, datetime \\ DateTime.utc_now())

# Escalation Policies
def list_escalation_policies(organization_id)
def get_escalation_policy!(id) / get_escalation_policy(id)
def create_escalation_policy(attrs) / update_escalation_policy(policy, attrs) / delete_escalation_policy(policy)
def create_escalation_rule(attrs) / update_escalation_rule(rule, attrs) / delete_escalation_rule(rule)
def create_escalation_target(attrs) / delete_escalation_target(target)
```

### `Towerops.Billing`
Stripe billing integration: checkout, portal, usage metering, subscription management.

```elixir
def default_free_devices()
def default_price_per_device()
def create_checkout_session(organization, opts \\ [])
def create_portal_session(organization, return_url)
def billable_device_count(organization)
def effective_free_device_count(org)
def effective_price_per_device(org)
def estimated_monthly_cost(organization)
def sync_usage_to_stripe(organization)
def update_subscription_from_stripe(organization, stripe_subscription)
def update_payment_method_status(organization, status)
def migrate_all_subscriptions_to_price(new_price_id)
```

### `Towerops.Agents`
Remote polling agent lifecycle: tokens, heartbeat, assignments, mass updates, debug mode.

```elixir
def create_agent_token(organization_id, name)
def create_cloud_poller(name)
def list_organization_agent_tokens(organization_id)
def list_cloud_pollers()
def verify_agent_token(token)
def update_agent_token_heartbeat(agent_token_id, ip_address, metadata \\ nil)
def update_agent_token(agent_token, attrs)
def restart_agent(agent_token_id)
def update_agent(agent_token_id, url, checksum)
def toggle_agent_debug(agent_token, enabled, user)
def revoke_agent_token(id)
def delete_agent_token(id)
def assign_device_to_agent(agent_token_id, device_id)
def unassign_device(device_id)
def list_agent_devices(agent_token_id)
def list_agent_polling_targets(agent_token_id)
def get_effective_agent_token(device)
def broadcast_mass_update()
```

### `Towerops.Monitoring`
Check definitions, execution, result tracking, state machine (soft/hard states).

```elixir
def list_checks(organization_id, opts \\ [])
def get_check(id) / get_check!(id)
def create_check(attrs) / update_check(check, attrs) / delete_check(check)
def schedule_check(check)
def create_check_result(attrs)
def get_latest_check_result(check_id)
def get_check_results(check_id, opts \\ [])
def get_check_graph_data(check_id, from_time, to_time)
def update_check_state(check, status, output)     # Soft/hard state machine
def stop_device_checks(device_id)
def disable_device_checks(device_id)
def get_device_health_summary(device_ids)
def get_device_latest_response_times(device_ids)
```

### `Towerops.Snmp`
SNMP device discovery, interface/sensor management, readings.

```elixir
def test_connection(ip, community, version, port \\ 161)
def discover_device(device)
def discover_all_for_org(org_id)
def create_checks_from_discovery(device, snmp_device)
def get_device(device_id) / get_device_with_associations(device_id)
def list_interfaces(device_id) / list_monitored_interfaces(device_id)
def list_sensors(device_id) / list_sensors_by_type(device_id)
def get_sensor_readings(sensor_id, opts \\ [])
def get_interface_stats(interface_id, opts \\ [])
def set_manual_capacity(interface_id, capacity_bps)
# ... many more reading/stat functions
```

### `Towerops.Gaiia`
Gaiia CRM/billing data: accounts, subscriptions, network sites, inventory, impact analysis.

```elixir
def list_accounts(organization_id)
def upsert_account(organization_id, attrs)
def count_active_subscribers(organization_id)
def sum_active_mrr(organization_id)
def list_network_sites(organization_id)
def upsert_network_site(organization_id, attrs)
def list_inventory_items(organization_id)
def list_billing_subscriptions(organization_id)
def get_device_subscriber_counts(device_ids)
def suggest_site_matches(organization_id, network_site)
def suggest_device_matches(organization_id, inventory_item)
```

Sub-modules: `Gaiia.Client`, `Gaiia.Webhooks`, `Gaiia.ImpactAnalysis`, `Gaiia.Reconciliation`, `Gaiia.SubscriberMatching`, `Gaiia.Actions`

### `Towerops.Preseem`
Preseem QoE integration: access points, subscriber metrics, fleet intelligence.

```elixir
def get_access_point_for_device(device_id)
def list_access_points(organization_id)
def list_unmatched_access_points(organization_id)
def list_matched_access_points(organization_id)
def list_subscriber_metrics(access_point_id, opts \\ [])
def link_access_point(access_point_id, device_id)
def unlink_access_point(access_point_id)
def get_site_qoe_summary(site_id)
```

Sub-modules: `Preseem.Client`, `Preseem.Sync`, `Preseem.Baseline`, `Preseem.DeviceMatcher`, `Preseem.FleetIntelligence`, `Preseem.Insights`

### Other Context Modules

| Module | Purpose |
|--------|---------|
| `Towerops.Maintenance` | Maintenance window CRUD, device/site in-maintenance checks |
| `Towerops.Topology` | Network topology discovery (LLDP/CDP/ARP/MAC), link inference |
| `Towerops.Dashboard` | Dashboard summary, health scores, impact analysis |
| `Towerops.Search` | Cross-entity full-text search (devices, sites, accounts, alerts) |
| `Towerops.ActivityFeed` | Aggregated activity feed (config changes, alerts, syncs, events) |
| `Towerops.ApiTokens` | API token CRUD, hashed token verification |
| `Towerops.Capacity` | Interface throughput & utilization calculations |
| `Towerops.ConfigChanges` | MikroTik config change diff tracking |
| `Towerops.MobileSessions` | Mobile app sessions, QR login flow |
| `Towerops.Weather` | Weather observations & alerts per site |
| `Towerops.Admin` | Superuser admin: user/org listing, audit logs, billing overrides |
| `Towerops.Profiles` | SNMP device profile matching & OID management |
| `Towerops.Settings` | Global application settings (key-value store) |
| `Towerops.Geocoding` | Address → lat/lng geocoding |
| `Towerops.GeoIp` | IP → location lookup |
| `Towerops.Http` | Shared HTTP client helpers |
| `Towerops.Trace` | Subscriber trace functionality |
| `Towerops.RateLimit` | Rate limiting logic |
| `Towerops.Numeric` | Numeric formatting utilities |

---

## 4. LiveView Pages

### Main Application (requires auth + org)

| Route | LiveView | Purpose |
|-------|----------|---------|
| `/dashboard` | `DashboardLive` | Org dashboard with health score, alert summary, site overview |
| `/alerts` | `AlertLive.Index` | Alert list with filtering (active/ack/resolved), real-time updates |
| `/agents` | `AgentLive.Index` | Agent list with health status |
| `/agents/:id` | `AgentLive.Show` | Agent detail: assigned devices, heartbeat, metadata |
| `/agents/:id/edit` | `AgentLive.Edit` | Edit agent settings |
| `/devices` | `DeviceLive.Index` | Device list with status, filtering, real-time updates |
| `/devices/new` | `DeviceLive.Form` | Create device form (with SNMP credential test) |
| `/devices/:id` | `DeviceLive.Show` | Device detail: status, interfaces, sensors, alerts, backups |
| `/devices/:id/edit` | `DeviceLive.Form` | Edit device form |
| `/devices/:id/graph/:sensor_type` | `GraphLive.Show` | Time-series graphs for device metrics |
| `/devices/:device_id/backups/compare` | `MikrotikBackupLive.Compare` | Config backup diff viewer |
| `/devices/:device_id/config-timeline` | `ConfigTimelineLive` | Config change timeline |
| `/sites` | `SiteLive.Index` | Site list with device counts |
| `/sites/new` | `SiteLive.Form` | Create site |
| `/sites/:id` | `SiteLive.Show` | Site detail: devices, capacity, weather |
| `/sites/:id/edit` | `SiteLive.Form` | Edit site |
| `/insights` | `InsightsLive.Index` | AI-generated operational insights |
| `/trace` | `TraceLive.Index` | Subscriber trace tool |
| `/activity` | `ActivityFeedLive` | Activity feed (config changes, alerts, syncs) |
| `/schedules` | `ScheduleLive.Index` | On-call schedule list |
| `/schedules/new` | `ScheduleLive.Form` | Create schedule |
| `/schedules/:id` | `ScheduleLive.Show` | Schedule detail with calendar view |
| `/schedules/:id/edit` | `ScheduleLive.Form` | Edit schedule |
| `/schedules/escalation-policies` | `EscalationPolicyLive.Index` | Escalation policy list |
| `/schedules/escalation-policies/new` | `EscalationPolicyLive.Form` | Create escalation policy |
| `/schedules/escalation-policies/:id` | `EscalationPolicyLive.Show` | Policy detail with rules |
| `/schedules/escalation-policies/:id/edit` | `EscalationPolicyLive.Form` | Edit policy |
| `/maintenance` | `MaintenanceLive.Index` | Maintenance window list |
| `/maintenance/new` | `MaintenanceLive.Form` | Create maintenance window |
| `/maintenance/:id` | `MaintenanceLive.Show` | Window detail |
| `/maintenance/:id/edit` | `MaintenanceLive.Form` | Edit window |
| `/changelog` | `ChangelogLive` | Product changelog |
| `/network-map` | `NetworkMapLive` | Network topology visualization |
| `/sites-map` | `MapLive.Index` | Geographic site map |

### Organization Settings (requires auth + specific org slug)

| Route | LiveView | Purpose |
|-------|----------|---------|
| `/orgs/:org_slug/settings` | `Org.SettingsLive` | Org settings, SNMP defaults, billing |
| `/orgs/:org_slug/settings/integrations` | `Org.IntegrationsLive` | Integration management |
| `/orgs/:org_slug/settings/integrations/preseem/devices` | `Org.PreseemDevicesLive` | Preseem AP ↔ device matching |
| `/orgs/:org_slug/settings/integrations/preseem/insights` | `Org.PreseemInsightsLive` | Preseem fleet insights |
| `/orgs/:org_slug/settings/integrations/gaiia/mapping` | `Org.GaiiaMappingLive` | Gaiia site/device mapping |
| `/orgs/:org_slug/settings/integrations/gaiia/reconciliation` | `Org.GaiiaReconciliationLive` | Gaiia data reconciliation |

### Account / User Pages

| Route | LiveView | Purpose |
|-------|----------|---------|
| `/orgs` | `OrgLive.Index` | Organization picker |
| `/orgs/new` | `OrgLive.New` | Create organization |
| `/users/settings` | `UserSettingsLive` | Profile, email, password, API tokens, sessions, TOTP |
| `/users/my-data` | `AccountLive.MyData` | GDPR data export |
| `/account/activity` | `AccountLive.Activity` | User login history |
| `/account/totp-enrollment` | `AccountLive.TotpEnrollment` | TOTP setup wizard |
| `/mobile/qr-login` | `MobileQRLive` | QR code for mobile app login |
| `/users/register` | `UserRegistrationLive` | Registration form |
| `/users/reset-password/:token` | `UserResetPasswordLive` | Password reset |
| `/help` | `HelpLive.Index` | Help / documentation page |

### Admin Pages (superuser only)

| Route | LiveView | Purpose |
|-------|----------|---------|
| `/admin/` | `Admin.DashboardLive` | Admin overview |
| `/admin/users` | `Admin.UserLive.Index` | All users management |
| `/admin/organizations` | `Admin.OrgLive.Index` | All orgs management |
| `/admin/audit` | `Admin.AuditLive.Index` | Audit log viewer |
| `/admin/monitoring` | `Admin.MonitoringLive` | System monitoring |
| `/admin/agents` | `Admin.AgentLive.Index` | All agents health |
| `/admin/security` | `Admin.SecurityLive.Index` | IP blocks/whitelists |

---

## 5. PubSub Topics

### Alert Topics

| Topic | Events | Subscribers |
|-------|--------|-------------|
| `alerts:org:{org_id}:new` | New alert created | `AlertLive.Index`, `DashboardLive`, `ActivityFeedLive` |
| `alerts:org:{org_id}:resolved` | Alert resolved | `AlertLive.Index`, `DashboardLive`, `ActivityFeedLive` |
| `organization:{org_id}:alerts` | Alert status updates | `UserAuth.subscribe_to_status_updates` (nav badge) |

### Device Topics

| Topic | Events | Subscribers |
|-------|--------|-------------|
| `device:{device_id}` | Device status changes, poll results, sensor data | `DeviceLive.Show`, `GraphLive.Show` |
| `device:events` | All device events (cross-device) | `Devices.EventLogger` |
| `device_events:org:{org_id}` | Org-scoped device events | `ActivityFeedLive` |
| `devices:org:{org_id}` | Device created/updated/deleted | `DeviceLive.Index`, `ActivityFeedLive` |
| `wireless_clients:device:{device_id}` | Wireless client updates | `DeviceLive.Show` |

### Agent Topics

| Topic | Events | Subscribers |
|-------|--------|-------------|
| `agents:health` | Agent heartbeat, stale detection | `AgentLive.Index`, `DeviceLive.Show`, `Admin.AgentLive` |
| `admin:agents` | Admin-level agent events | `Admin.AgentLive.Index` |
| `agent:{agent_id}:assignments` | Assignment changes | `AgentChannel` |
| `agent:{agent_id}:discovery` | Discovery commands | `AgentChannel` |
| `agent:{agent_id}:backup` | Backup commands | `AgentChannel` |
| `agent:{agent_id}:credential_test` | Credential test requests | `AgentChannel` |
| `agent:{agent_id}:live_poll` | Live poll requests | `AgentChannel` |
| `agent:{agent_id}:lifecycle` | Restart/update commands | `AgentChannel` |
| `agent:{agent_id}:latency_probe` | Latency probe requests | `AgentChannel` |

### Other Topics

| Topic | Events | Subscribers |
|-------|--------|-------------|
| `config:org:{org_id}` | Config change events | `ActivityFeedLive` |
| `sync:org:{org_id}` | Integration sync events | `ActivityFeedLive` |
| `topology:{org_id}` | Topology link changes | `NetworkMapLive` |
| `credential_test:{test_id}` | SNMP credential test results | `DeviceLive.Form` |
| `security:blocks` | IP block changes | `Admin.SecurityLive` |
| `security:whitelist` | IP whitelist changes | `Admin.SecurityLive` |
| `job_monitoring` | Oban job health events | `Admin.MonitoringLive` |

---

## 6. Workers (Oban)

### Queue: `pollers`

| Worker | Purpose |
|--------|---------|
| `DevicePollerWorker` | SNMP polling: interfaces, sensors, storage, processors, wireless, ARP, topology. The main data collection worker. |

### Queue: `monitors`

| Worker | Purpose |
|--------|---------|
| `DeviceMonitorWorker` | Device up/down monitoring via ICMP ping, triggers alerts on status change |

### Queue: `discovery`

| Worker | Purpose |
|--------|---------|
| `DiscoveryWorker` | SNMP device discovery: sysinfo, interfaces, sensors, profiles |

### Queue: `checks`

| Worker | Purpose |
|--------|---------|
| `CheckWorker` | Schedules and dispatches monitoring check execution |

### Queue: `check_executors`

| Worker | Purpose |
|--------|---------|
| `CheckExecutorWorker` | Executes individual monitoring checks (HTTP, ping, etc.) |

### Queue: `notifications`

| Worker | Purpose |
|--------|---------|
| `AlertNotificationWorker` | Sends alert notifications (email, PagerDuty, on-call) |
| `WelcomeEmailWorker` | Sends welcome email after registration |
| `EscalationCheckWorker` | Checks for escalation timeouts and advances incidents |

### Queue: `weather`

| Worker | Purpose |
|--------|---------|
| `WeatherSyncWorker` | Fetches weather data for sites with coordinates |

### Queue: `maintenance`

| Worker | Purpose |
|--------|---------|
| `BillingSyncWorker` | Syncs device count usage to Stripe |
| `GaiiaSyncWorker` | Syncs data from Gaiia billing/CRM |
| `GaiiaInsightWorker` | Generates Gaiia-based insights |
| `PreseemSyncWorker` | Syncs data from Preseem |
| `PreseemBaselineWorker` | Calculates Preseem performance baselines |
| `NetboxSyncWorker` | Syncs data from NetBox DCIM |
| `SonarSyncWorker` | Syncs data from Sonar billing |
| `SplynxSyncWorker` | Syncs data from Splynx billing |
| `VispSyncWorker` | Syncs data from VISP billing |
| `MikrotikBackupWorker` | Fetches MikroTik config backups |
| `BackupSummaryWorker` | Generates backup comparison summaries |
| `BackupTimeoutWorker` | Cleans up timed-out backup requests |
| `FirmwareVersionFetcherWorker` | Fetches latest firmware versions from vendors |
| `StaleAgentWorker` | Detects agents that stopped heartbeating |
| `AgentLatencyEvaluator` | Evaluates agent-to-device latency data |
| `CloudLatencyProbeWorker` | Probes cloud poller latency |
| `SessionCleanupWorker` | Cleans up expired browser/mobile sessions |
| `LoginHistoryCleanupWorker` | Purges old login attempt records |
| `ExpiredBanCleanupWorker` | Removes expired IP bans |
| `StaleViolationCleanupWorker` | Cleans up stale rate limit violations |
| `CloudflareBanWorker` | Syncs IP bans to Cloudflare WAF |
| `JobHealthCheckWorker` | Monitors Oban job queue health |
| `DeviceHealthInsightWorker` | Generates device health insights |
| `CapacityInsightWorker` | Generates capacity utilization insights |
| `SystemInsightWorker` | Generates system-level insights |
| `WirelessInsightWorker` | Generates wireless performance insights |

### Additional Worker

| Worker | Queue | Purpose |
|--------|-------|---------|
| `Snmp.NeighborCleanupWorker` | (in snmp module) | Cleans up stale SNMP neighbor records |

---

## 7. Integrations Architecture

### Overview

Integrations are stored as `Integration` records with encrypted credentials (`Cloak.Ecto`). Each provider has:
- A **client module** (`Client`) for API communication
- A **sync module** (`Sync`) for data synchronization
- An **Oban worker** for periodic sync

### Billing/CRM Providers

| Provider | Modules | Data Synced |
|----------|---------|-------------|
| **Gaiia** | `Gaiia.Client`, `Gaiia.Sync`, `Gaiia.Webhooks`, `Gaiia.ImpactAnalysis`, `Gaiia.Reconciliation`, `Gaiia.SubscriberMatching`, `Gaiia.Actions` | Accounts, billing subscriptions, network sites, inventory items, device-subscriber links |
| **Preseem** | `Preseem.Client`, `Preseem.Sync`, `Preseem.Baseline`, `Preseem.DeviceMatcher`, `Preseem.FleetIntelligence`, `Preseem.Insights` | Access points, subscriber metrics, fleet profiles, device baselines |
| **Sonar** | `Sonar.Client`, `Sonar.Sync` | Subscriber/billing data |
| **Splynx** | `Splynx.Client`, `Splynx.Sync` | Subscriber/billing data |
| **VISP** | `Visp.Client`, `Visp.Sync` | Subscriber/billing data |

### Infrastructure Providers

| Provider | Modules | Purpose |
|----------|---------|---------|
| **NetBox** | `Netbox.Client`, `Netbox.Sync` | DCIM inventory sync (devices, sites) |
| **PagerDuty** | `PagerDuty.Client`, `PagerDuty.Notifier` | Alert routing & incident escalation |

### Stripe Billing

Stripe integration is handled separately from the integration framework:
- `Towerops.Billing` — checkout sessions, portal sessions, usage metering
- `BillingSyncWorker` — periodic usage sync
- `StripeWebhookController` — receives Stripe webhook events
- `StripeWebhookEvent` schema — idempotent webhook processing
- Usage-based billing: free device tier, then per-device pricing
- Organization fields: `stripe_customer_id`, `stripe_subscription_id`, `subscription_status`

### Gaiia Specifics

Gaiia has the deepest integration:
- **Webhook receiver** at `/api/v1/webhooks/gaiia/:organization_id`
- **Impact analysis** — when a device goes down, calculates affected subscribers & MRR
- **Reconciliation** — UI for matching Gaiia network sites to TowerOps sites
- **Device-subscriber links** — maps which subscribers are served by which device

### Alert Routing

Organizations configure `alert_routing`:
- `"builtin"` — uses on-call schedules + escalation policies
- `"pagerduty"` — routes to PagerDuty
- `"both"` — routes to both

---

## 8. Auth Flow

### Authentication Stack

```
Request → Browser Pipeline → fetch_session → fetch_current_scope_for_user
                                                  │
                                                  ├─ Check session token
                                                  ├─ Check remember_me cookie
                                                  ├─ Build Scope (user + org + tz)
                                                  └─ Handle impersonation
```

### `Towerops.Accounts.Scope`

The `Scope` struct flows through the entire request lifecycle:

```elixir
%Scope{
  user: %User{},           # Current user (or impersonated user)
  superuser: %User{} | nil, # Actual superuser if impersonating
  impersonating?: boolean(),
  organization: %Organization{} | nil,
  timezone: "America/Chicago",
  time_format: "24h"
}
```

### Session Lifecycle

1. **Login** — `UserSessionController.create` validates email/password
2. **TOTP verification** — If TOTP enabled, redirect to `/users/log-in/totp`
3. **Session creation** — Token stored in session + optional remember_me cookie
4. **Organization loading** — Default org set from membership, or from URL slug
5. **Session renewal** — Tokens reissued after configurable interval
6. **Sudo mode** — Sensitive operations require recent password re-entry (`last_sudo_at`)

### LiveView `on_mount` Hooks

```elixir
:load_cookie_consent           # Load GDPR consent state
:require_authenticated_user    # Redirect to login if no session
:require_totp_enrollment       # Redirect to TOTP setup if required
:load_current_organization     # Load org from URL slug param
:load_default_organization     # Load user's default org
:subscribe_to_status_updates   # Subscribe to org alert PubSub
:require_superuser             # Admin-only pages
:require_sudo_mode             # Require recent password entry
:handle_policy_consent         # Check privacy/terms acceptance
:redirect_if_user_is_authenticated  # Login page redirect
:require_organization_owner    # Owner-only operations
```

### API Authentication

- **REST API** (`/api/v1/*`) — Bearer token via `ToweropsWeb.Plugs.ApiAuth`, verified against hashed `ApiToken` records
- **GraphQL** (`/api/graphql`) — Same bearer token auth + `GraphQL.Context` plug
- **Mobile API** (`/api/v1/mobile/*`) — Session token via `ToweropsWeb.Plugs.MobileAuth`
- **Webhooks** (`/api/v1/webhooks/*`) — Shared secret via `ToweropsWeb.Plugs.WebhookAuth`
- **Agent WebSocket** — Token-based auth in `AgentSocket`/`AgentChannel`

### Organization Switching

- `POST /orgs/switch` — changes the current organization in session
- User must have a membership in the target organization
- Default org stored on `Membership.is_default` flag
- URL-based: `/orgs/:org_slug/settings/*` uses slug from URL

### Security Features

- **Rate limiting** — Per-route rate limits (auth, API, admin) via `RateLimit` plug
- **Brute-force protection** — `Security.BruteForce` with IP blocking + Cloudflare WAF sync
- **TOTP/MFA** — Required enrollment, multi-device support, recovery codes
- **Sudo mode** — Time-limited elevated access for sensitive operations
- **IP whitelisting/blocking** — `IpBlock`, `IpWhitelist` schemas
- **Session tracking** — `BrowserSession` records with device/location info
- **Impersonation** — Superuser can impersonate users, tracked in `Scope`

---

## 9. Key Patterns

### Cascading Configuration

SNMP and MikroTik credentials cascade: **Organization → Site → Device**. Each level can override or inherit:

```elixir
# In Devices context
def resolve_agent_token_id(device) do
  get_direct_agent(device)       # Device-level assignment
  || get_site_agent(device)      # Site default agent
  || get_org_default_agent(device) # Organization default
  || get_global_default_agent()  # Global cloud poller fallback
end
```

### AccessControl Helpers

`ToweropsWeb.Live.Helpers.AccessControl` provides organization-scoped access verification:

```elixir
verify_device_access(device_id, organization_id)  # → {:ok, device} | {:error, :not_found | :unauthorized}
verify_site_access(site_id, organization_id)
verify_alert_access(alert_id, organization_id)
```

### Membership Roles

Ordered by privilege: `owner > admin > executive > technician > member > viewer`

- **owner** — Full access, can delete organization
- **admin** — Full access except deleting org
- **executive** — Read-only with financial visibility (MRR, revenue)
- **technician** — Field ops: devices, sites, alerts, no financials
- **viewer** — Read-only, no financials

### Breadcrumbs

`ToweropsWeb.Components.Breadcrumbs` provides a DaisyUI-based breadcrumb component:

```heex
<.breadcrumb items={[
  %{label: "Sites", path: ~p"/sites"},
  %{label: @site.name, path: ~p"/sites/#{@site.id}"},
  %{label: "Edit"}
]} />
```

### PubSub Pattern

Contexts broadcast changes; LiveViews subscribe on mount:

```elixir
# In context (e.g., Alerts)
Phoenix.PubSub.broadcast(Towerops.PubSub, "alerts:org:#{org_id}:new", {:new_alert, alert})

# In LiveView mount
Phoenix.PubSub.subscribe(Towerops.PubSub, "alerts:org:#{organization.id}:new")

# In LiveView handle_info
def handle_info({:new_alert, alert}, socket), do: ...
```

### Encrypted Fields

Sensitive data (SNMP passwords, API credentials, integration secrets) uses `Cloak.Ecto` encryption:

```elixir
field :snmpv3_auth_password, Towerops.Encrypted.Binary
field :credentials, Towerops.Encrypted.Map
```

### Multi-Tenancy

All queries are scoped by `organization_id`. The org context flows through:
1. `Scope` struct in conn/socket assigns
2. URL slug (`/orgs/:org_slug/...`)
3. Default organization from user membership

### Form Pattern

LiveViews use Phoenix changesets with `to_form/1`:

```elixir
# In mount/handle_params
changeset = Context.change_entity(entity, %{})
socket = assign(socket, form: to_form(changeset))

# In handle_event("save", ...)
case Context.update_entity(entity, params) do
  {:ok, entity} -> {:noreply, push_navigate(socket, to: ...)}
  {:error, changeset} -> {:noreply, assign(socket, form: to_form(changeset))}
end
```

### Flash Messages

Standard Phoenix flash with custom styling:

```elixir
socket |> put_flash(:info, "Device created successfully") |> push_navigate(to: ...)
socket |> put_flash(:error, "Failed to save device")
```

### Agent Communication

Agents connect via WebSocket (`AgentChannel`):
1. Agent authenticates with token via `AgentSocket`
2. Channel subscribes to agent-specific PubSub topics
3. Server pushes commands: poll, discover, backup, restart, update
4. Agent sends results: poll data, discovery results, backup content
5. Protobuf used for efficient data serialization (`lib/towerops/proto/`)

### GraphQL API

Absinthe-based GraphQL at `/api/graphql`:
- Schema: `ToweropsWeb.GraphQL.Schema`
- Types mirror core entities (devices, sites, alerts, agents, schedules, etc.)
- Complexity analysis enabled (max 500 complexity, max 10 depth)
- Subscriptions via `ToweropsWeb.GraphQL.Subscriptions`
- API token authentication via `GraphQL.Context` plug

### Health Checks

- `GET /health` — Kubernetes probe endpoint (no auth)
- `DeviceMonitorWorker` — ICMP ping monitoring
- `JobHealthCheckWorker` — Oban queue health
- `AgentLatencyEvaluator` — Agent responsiveness

---

*Generated from source code analysis. Last updated: 2026-03-13.*