graham/towerops

Fork 0

Graham McIntire aa46ff2bd3

docs: replace flux/fluxcd references with argocd

2026-04-29 11:15:53 -05:00

42 KiB

Raw Permalink Blame History

TowerOps Architecture

Network infrastructure monitoring platform for ISPs/WISPs built with Phoenix LiveView, Oban, and TimescaleDB.

1. Project Structure

towerops/
├── assets/                    # Frontend: JS, CSS, Tailwind
├── config/                    # Runtime & env config (dev/test/prod/runtime)
├── c_src/                     # C NIFs (SNMP native interop)
├── docs/                      # Documentation
├── e2e/                       # End-to-end tests (Playwright)
├── k8s/                       # Kubernetes manifests (reconciled by ArgoCD)
├── lib/
│   ├── mix/                   # Custom Mix tasks
│   ├── snmpkit/               # SNMP toolkit (native extension)
│   ├── towerops/              # Core business logic (contexts, schemas, workers)
│   └── towerops_web/          # Web layer (LiveView, controllers, channels, GraphQL)
├── mibs/                      # SNMP MIB files
├── priv/
│   ├── repo/migrations/       # Ecto migrations
│   ├── static/                # Static assets
│   └── gettext/               # i18n translations
├── rel/                       # Release configuration
├── scripts/                   # Operational scripts
├── test/                      # Test suite
├── towerops/                  # Agent-related files
└── vendor/                    # Vendored dependencies

Key `lib/towerops/` Subdirectories

Directory	Purpose
`accounts/`	User, auth tokens, TOTP, recovery codes, sessions, consent
`admin/`	Superuser audit logs
`agents/`	Remote polling agent tokens & assignments
`alerts/`	Alert schema and queries
`api_tokens/`	API token management (hashed bearer tokens)
`billing/`	Stripe integration, usage metering
`capacity/`	Interface throughput & utilization calculations
`config_changes/`	MikroTik config change event tracking
`contexts/`	Shared context helpers
`devices/`	Device schema, backups, firmware, events
`ecto_types/`	Custom Ecto types (IP address, MAC address, SNMP OID)
`gaiia/`	Gaiia billing/CRM integration (accounts, subscriptions, impact analysis)
`geoip/`	GeoIP lookup (blocks, locations)
`integrations/`	Integration schema (multi-provider)
`job_monitoring/`	Oban job health events
`maintenance/`	Maintenance windows
`mobile_sessions/`	Mobile app sessions & QR login
`monitoring/`	Check definitions & check results
`netbox/`	NetBox DCIM sync
`on_call/`	Schedules, escalation policies, incidents, notifications
`organizations/`	Org, membership, invitations, policies
`pagerduty/`	PagerDuty client & notifier
`preseem/`	Preseem QoE integration (APs, baselines, fleet intelligence)
`profiles/`	SNMP device profiles (OID mappings)
`proto/`	Protobuf definitions (agent communication)
`security/`	Brute-force protection, IP blocks/whitelists
`settings/`	Global application settings (key-value store)
`sites/`	Site schema & tree structure
`snmp/`	SNMP device data (interfaces, sensors, readings, wireless, topology)
`sonar/`	Sonar billing integration
`splynx/`	Splynx billing integration
`topology/`	Network topology discovery (LLDP, CDP, ARP, MAC)
`visp/`	VISP billing integration
`weather/`	Weather observations & alerts per site
`workers/`	All Oban background workers

Key `lib/towerops_web/` Subdirectories

Directory	Purpose
`channels/`	Phoenix Channels (agent WebSocket)
`components/`	Reusable UI components (breadcrumbs, tables, forms)
`controllers/`	Traditional controllers (auth, webhooks, health, API)
`graphql/`	Absinthe GraphQL schema, types, resolvers, subscriptions
`helpers/`	View helpers (time formatting, status display)
`live/`	All LiveView pages
`plugs/`	Plug middleware (auth, rate limiting, security headers)

2. Data Model

All primary keys are UUIDs (:binary_id). Multi-tenancy is organization-scoped.

Core Entities

┌──────────────┐     ┌──────────────────┐     ┌──────────────┐
│     User     │────<│   Membership     │>────│ Organization │
│              │     │ (role-based)     │     │              │
│ email        │     │ role: owner|     │     │ name, slug   │
│ hashed_pass  │     │  admin|executive │     │ subscription │
│ totp_secret  │     │  |technician|    │     │ snmp_config  │
│ is_superuser │     │  member|viewer   │     │ mikrotik_cfg │
│ timezone     │     │ is_default       │     │ stripe_*     │
│ default_org  │     └──────────────────┘     │ alert_routing│
└──────────────┘                               └──────┬───────┘
                                                      │
                              ┌────────────────┬──────┴──────┬───────────────┐
                              │                │             │               │
                        ┌─────┴─────┐   ┌──────┴──────┐ ┌───┴────┐  ┌──────┴──────┐
                        │   Site    │   │   Device    │ │ Alert  │  │ Integration │
                        │           │   │             │ │        │  │             │
                        │ name      │   │ name        │ │ type   │  │ provider    │
                        │ lat/lng   │   │ ip_address  │ │ sev    │  │ credentials │
                        │ address   │   │ status      │ │ msg    │  │ sync_status │
                        │ snmp_cfg  │   │ snmp_cfg    │ │ device │  │ (encrypted) │
                        │ parent_id │   │ mikrotik_cfg│ │ check  │  └─────────────┘
                        │ agent_tok │   │ site_id     │ └────────┘
                        └───────────┘   │ agent_assign│
                                        │ check_intv  │
                                        └─────────────┘

On-Call / Incident Management

Schedule ──has_many──> Layer ──has_many──> LayerMember ──belongs_to──> User
    │
    └──has_many──> Override (user, start_at, end_at)

EscalationPolicy ──has_many──> EscalationRule ──has_many──> EscalationTarget
    │                              │                            │
    │                              │ position, delay_minutes    ├── schedule_id
    │                              └────────────────────────    └── user_id
    │
Incident ──belongs_to──> Alert
    │                     EscalationPolicy
    │                     Organization
    └──has_many──> Notification ──belongs_to──> User

SNMP Data Model

Device (towerops) ──1:1──> Snmp.Device
    │
    ├── Snmp.Interface ──> InterfaceStat (time-series)
    ├── Snmp.Sensor ──> SensorReading (time-series)
    ├── Snmp.StateSensor
    ├── Snmp.Processor ──> ProcessorReading
    ├── Snmp.Storage ──> StorageReading
    ├── Snmp.WirelessClient ──> WirelessClientReading
    ├── Snmp.Neighbor (LLDP/CDP)
    ├── Snmp.ArpEntry
    ├── Snmp.Vlan
    └── Snmp.PhysicalEntity

Billing / CRM Schemas

Gaiia: Account, BillingSubscription, NetworkSite, InventoryItem, DeviceSubscriberLink
Preseem: AccessPoint, SubscriberMetric, FleetProfile, DeviceBaseline, Insight, SyncLog
Billing: StripeWebhookEvent

Other Schemas

Schema	Table	Purpose
`AgentToken`	`agent_tokens`	Remote polling agent credentials
`AgentAssignment`	`agent_assignments`	Device → agent mapping
`ApiToken`	`api_tokens`	REST/GraphQL API bearer tokens
`BrowserSession`	`browser_sessions`	Active login sessions
`LoginAttempt`	`login_attempts`	Auth attempt audit log
`UserConsent`	`user_consents`	Privacy/terms consent records
`PolicyVersion`	`policy_versions`	Privacy policy versions
`UserTotpDevice`	`user_totp_devices`	Multi-device TOTP
`UserRecoveryCode`	`user_recovery_codes`	MFA recovery codes
`MaintenanceWindow`	`maintenance_windows`	Scheduled maintenance
`ConfigChangeEvent`	`config_change_events`	MikroTik config diffs
`DeviceFirmwareHistory`	`device_firmware_histories`	Firmware change tracking
`FirmwareRelease`	`firmware_releases`	Known firmware versions
`BackupRequest`	`backup_requests`	MikroTik backup queue
`MikrotikBackup`	`mikrotik_backups`	Stored config backups
`Check`	`checks`	Monitoring check definitions
`CheckResult`	`check_results`	Check execution results
`MonitoringCheck`	`monitoring_checks`	Check configuration
`DeviceLink`	`device_links`	Discovered topology links
`DeviceLinkEvidence`	`device_link_evidence`	Link confidence scores
`DeviceNeighbor`	`device_neighbors`	Raw neighbor data
`ApplicationSetting`	`application_settings`	Global config KV store
`AuditLog`	`audit_logs`	Admin action audit trail
`IpBlock`	`ip_blocks`	Blocked IPs
`IpWhitelist`	`ip_whitelists`	Whitelisted IPs
`MobileSession`	`mobile_sessions`	Mobile app sessions
`QrLoginToken`	`qr_login_tokens`	QR-based mobile auth
`Weather.Observation`	`weather_observations`	Site weather data
`Weather.Alert`	`weather_alerts`	Active weather alerts
`GeoIp.Block` / `Location`	`geoip_*`	IP geolocation

3. Context Modules

`Towerops.Accounts`

User management, authentication, TOTP/MFA, password management, email changes, session tracking.

# User lookup
def get_user_by_email(email)
def get_user_by_email_and_password(email, password)
def get_user(id)
def get_user!(id)

# Registration
def register_user(attrs)
def register_user_with_organization(attrs)
def change_user_registration(user, attrs \\ %{})

# TOTP / MFA
def generate_totp_secret()
def generate_totp_uri(user, secret)
def generate_totp_qr_code(user, secret)
def verify_totp(secret, code)
def enable_totp(user, secret)
def totp_enabled?(user)
def verify_user_totp(user, code)
def list_user_totp_devices(user_id)
def create_totp_device(user_id, device_name)
def verify_user_totp_any_device(user, code)
def delete_totp_device(device_id, user_id)
def rename_totp_device(device_id, user_id, new_name)
def generate_recovery_codes(user_id)
def verify_recovery_code(user_id, code)
def count_unused_recovery_codes(user_id)
def verify_user_mfa(user, code)

# Profile & Settings
def change_user_profile(user, attrs \\ %{})
def update_user_profile(user, attrs)
def sudo_mode?(user, minutes \\ -10)
def grant_sudo_mode(user)

# Email & Password
def change_user_email(user, attrs \\ %{}, opts \\ [])
def update_user_email(user, token)
def change_user_password(user, attrs \\ %{}, opts \\ [])

# Sessions (+ more token functions not listed)

`Towerops.Organizations`

Multi-tenant org management, memberships, invitations, SNMP/agent propagation.

def list_user_organizations(user_id)
def user_has_access?(user_id, organization_id)
def get_organization!(id)
def get_organization_by_slug!(slug)
def create_organization(attrs, user_id, opts \\ [])
def update_organization(organization, attrs)
def delete_organization(organization)
def change_organization(organization, attrs \\ %{})

# Memberships
def get_membership(organization_id, user_id)
def list_organization_memberships(organization_id)
def list_organization_members(organization_id)
def remove_member(organization_id, user_id)
def update_member_role(organization_id, user_id, new_role)
def create_membership(attrs)
def get_default_membership(user_id)
def set_default_organization(user_id, organization_id)
def list_organization_notification_recipients(organization_id)

# Invitations
def list_pending_invitations(organization_id)
def create_invitation(attrs)
def get_invitation_by_token(token)
def accept_invitation(invitation, user_id)
def delete_invitation(invitation)

# Config propagation
def apply_snmp_config_to_all_equipment(organization_id)
def apply_agent_to_all_equipment(organization_id)
def clear_all_site_assignments(organization_id)

# Billing lookups
def get_organization_by_stripe_customer_id(stripe_customer_id)
def list_organizations_with_active_subscriptions()

`Towerops.Devices`

Device CRUD, SNMP/MikroTik config resolution (cascading: org → site → device), monitoring.

def list_site_devices(site_id)
def list_organization_devices(organization_id, filters \\ %{})
def count_organization_devices(organization_id)
def search_devices(organization_id, query)
def get_device_status_counts(organization_id)
def list_monitored_devices()
def list_snmp_enabled_devices()
def list_mikrotik_devices_with_api()
def get_device(id)
def get_device!(id)
def get_device_with_details(id)
def get_device_by_ip(ip_address, site_id, exclude_id \\ nil)
def get_snmp_config(device)          # Resolves cascading config
def get_mikrotik_config(device)      # Resolves cascading config
def get_snmpv3_config(device)        # Resolves cascading config
def resolve_agent_token_id(device)   # Device → site → org → global fallback
def propagate_site_snmpv3_change(site_id, attrs)
def propagate_organization_mikrotik_change(organization_id, attrs)
# ... CRUD, reordering, etc.

`Towerops.Sites`

Site CRUD, hierarchical tree structure, config propagation.

def list_organization_sites(organization_id)
def list_root_sites(organization_id)
def list_child_sites(parent_site_id)
def build_site_tree(organization_id)
def list_sites_with_coordinates()
def search_sites(organization_id, query)
def get_site!(id)
def create_site(attrs)
def update_site(site, attrs)
def delete_site(site)
def apply_snmp_config_to_all_equipment(site_id)
def apply_agent_to_all_equipment(site_id)
def reorder_site(site_id, new_position)

`Towerops.Alerts`

Alert lifecycle: creation, acknowledgement, resolution, Gaiia impact analysis.

def create_alert(attrs)
def list_devices_alerts(device_id, limit \\ 100)
def list_organization_active_alerts(organization_id)
def list_organization_alerts(organization_id, filters)
def count_active_alerts(organization_id)
def get_alert!(id)
def get_alert(id)
def acknowledge_alert(alert, user_id)
def resolve_alert(alert)
def resolve_alert_silent(alert)         # No PubSub broadcast
def acknowledge_alert_silent(alert)     # No PubSub broadcast
def has_active_alert?(device_id, alert_type)
def get_active_alert(device_id, alert_type)
def has_active_check_alert?(check_id)
def resolve_check_alerts(check_id)
def store_gaiia_impact(alert, impact_data)

`Towerops.Integrations`

Generic integration CRUD for all billing/CRM providers.

def list_integrations(organization_id)
def get_integration(organization_id, provider)
def create_integration(organization_id, attrs)
def update_integration(integration, attrs)
def update_sync_status(integration, status, message \\ nil)
def delete_integration(integration)
def get_billing_summary(organization_id)
def list_enabled_integrations(provider)
def update_billing_totals(integration, subscriber_count, total_mrr)

Valid providers: preseem, gaiia, pagerduty, netbox, sonar, splynx, visp

`Towerops.OnCall`

On-call schedules, escalation policies, incidents.

# Schedules
def list_schedules(organization_id)
def get_schedule!(id) / get_schedule(id)
def create_schedule(attrs) / update_schedule(schedule, attrs) / delete_schedule(schedule)
def create_layer(attrs) / update_layer(layer, attrs) / delete_layer(layer)
def add_layer_member(attrs) / remove_layer_member(member)
def create_override(attrs) / delete_override(override)
def who_is_on_call(schedule_id, datetime \\ DateTime.utc_now())

# Escalation Policies
def list_escalation_policies(organization_id)
def get_escalation_policy!(id) / get_escalation_policy(id)
def create_escalation_policy(attrs) / update_escalation_policy(policy, attrs) / delete_escalation_policy(policy)
def create_escalation_rule(attrs) / update_escalation_rule(rule, attrs) / delete_escalation_rule(rule)
def create_escalation_target(attrs) / delete_escalation_target(target)

`Towerops.Billing`

Stripe billing integration: checkout, portal, usage metering, subscription management.

def default_free_devices()
def default_price_per_device()
def create_checkout_session(organization, opts \\ [])
def create_portal_session(organization, return_url)
def billable_device_count(organization)
def effective_free_device_count(org)
def effective_price_per_device(org)
def estimated_monthly_cost(organization)
def sync_usage_to_stripe(organization)
def update_subscription_from_stripe(organization, stripe_subscription)
def update_payment_method_status(organization, status)
def migrate_all_subscriptions_to_price(new_price_id)

`Towerops.Agents`

Remote polling agent lifecycle: tokens, heartbeat, assignments, mass updates, debug mode.

def create_agent_token(organization_id, name)
def create_cloud_poller(name)
def list_organization_agent_tokens(organization_id)
def list_cloud_pollers()
def verify_agent_token(token)
def update_agent_token_heartbeat(agent_token_id, ip_address, metadata \\ nil)
def update_agent_token(agent_token, attrs)
def restart_agent(agent_token_id)
def update_agent(agent_token_id, url, checksum)
def toggle_agent_debug(agent_token, enabled, user)
def revoke_agent_token(id)
def delete_agent_token(id)
def assign_device_to_agent(agent_token_id, device_id)
def unassign_device(device_id)
def list_agent_devices(agent_token_id)
def list_agent_polling_targets(agent_token_id)
def get_effective_agent_token(device)
def broadcast_mass_update()

`Towerops.Monitoring`

Check definitions, execution, result tracking, state machine (soft/hard states).

def list_checks(organization_id, opts \\ [])
def get_check(id) / get_check!(id)
def create_check(attrs) / update_check(check, attrs) / delete_check(check)
def schedule_check(check)
def create_check_result(attrs)
def get_latest_check_result(check_id)
def get_check_results(check_id, opts \\ [])
def get_check_graph_data(check_id, from_time, to_time)
def update_check_state(check, status, output)     # Soft/hard state machine
def stop_device_checks(device_id)
def disable_device_checks(device_id)
def get_device_health_summary(device_ids)
def get_device_latest_response_times(device_ids)

`Towerops.Snmp`

SNMP device discovery, interface/sensor management, readings.

def test_connection(ip, community, version, port \\ 161)
def discover_device(device)
def discover_all_for_org(org_id)
def create_checks_from_discovery(device, snmp_device)
def get_device(device_id) / get_device_with_associations(device_id)
def list_interfaces(device_id) / list_monitored_interfaces(device_id)
def list_sensors(device_id) / list_sensors_by_type(device_id)
def get_sensor_readings(sensor_id, opts \\ [])
def get_interface_stats(interface_id, opts \\ [])
def set_manual_capacity(interface_id, capacity_bps)
# ... many more reading/stat functions

`Towerops.Gaiia`

Gaiia CRM/billing data: accounts, subscriptions, network sites, inventory, impact analysis.

def list_accounts(organization_id)
def upsert_account(organization_id, attrs)
def count_active_subscribers(organization_id)
def sum_active_mrr(organization_id)
def list_network_sites(organization_id)
def upsert_network_site(organization_id, attrs)
def list_inventory_items(organization_id)
def list_billing_subscriptions(organization_id)
def get_device_subscriber_counts(device_ids)
def suggest_site_matches(organization_id, network_site)
def suggest_device_matches(organization_id, inventory_item)

Sub-modules: Gaiia.Client, Gaiia.Webhooks, Gaiia.ImpactAnalysis, Gaiia.Reconciliation, Gaiia.SubscriberMatching, Gaiia.Actions

`Towerops.Preseem`

Preseem QoE integration: access points, subscriber metrics, fleet intelligence.

def get_access_point_for_device(device_id)
def list_access_points(organization_id)
def list_unmatched_access_points(organization_id)
def list_matched_access_points(organization_id)
def list_subscriber_metrics(access_point_id, opts \\ [])
def link_access_point(access_point_id, device_id)
def unlink_access_point(access_point_id)
def get_site_qoe_summary(site_id)

Sub-modules: Preseem.Client, Preseem.Sync, Preseem.Baseline, Preseem.DeviceMatcher, Preseem.FleetIntelligence, Preseem.Insights

Other Context Modules

Module	Purpose
`Towerops.Maintenance`	Maintenance window CRUD, device/site in-maintenance checks
`Towerops.Topology`	Network topology discovery (LLDP/CDP/ARP/MAC), link inference
`Towerops.Dashboard`	Dashboard summary, health scores, impact analysis
`Towerops.Search`	Cross-entity full-text search (devices, sites, accounts, alerts)
`Towerops.ActivityFeed`	Aggregated activity feed (config changes, alerts, syncs, events)
`Towerops.ApiTokens`	API token CRUD, hashed token verification
`Towerops.Capacity`	Interface throughput & utilization calculations
`Towerops.ConfigChanges`	MikroTik config change diff tracking
`Towerops.MobileSessions`	Mobile app sessions, QR login flow
`Towerops.Weather`	Weather observations & alerts per site
`Towerops.Admin`	Superuser admin: user/org listing, audit logs, billing overrides
`Towerops.Profiles`	SNMP device profile matching & OID management
`Towerops.Settings`	Global application settings (key-value store)
`Towerops.Geocoding`	Address → lat/lng geocoding
`Towerops.GeoIp`	IP → location lookup
`Towerops.Http`	Shared HTTP client helpers
`Towerops.Trace`	Subscriber trace functionality
`Towerops.RateLimit`	Rate limiting logic
`Towerops.Numeric`	Numeric formatting utilities

4. LiveView Pages

Main Application (requires auth + org)

Route	LiveView	Purpose
`/dashboard`	`DashboardLive`	Org dashboard with health score, alert summary, site overview
`/alerts`	`AlertLive.Index`	Alert list with filtering (active/ack/resolved), real-time updates
`/agents`	`AgentLive.Index`	Agent list with health status
`/agents/:id`	`AgentLive.Show`	Agent detail: assigned devices, heartbeat, metadata
`/agents/:id/edit`	`AgentLive.Edit`	Edit agent settings
`/devices`	`DeviceLive.Index`	Device list with status, filtering, real-time updates
`/devices/new`	`DeviceLive.Form`	Create device form (with SNMP credential test)
`/devices/:id`	`DeviceLive.Show`	Device detail: status, interfaces, sensors, alerts, backups
`/devices/:id/edit`	`DeviceLive.Form`	Edit device form
`/devices/:id/graph/:sensor_type`	`GraphLive.Show`	Time-series graphs for device metrics
`/devices/:device_id/backups/compare`	`MikrotikBackupLive.Compare`	Config backup diff viewer
`/devices/:device_id/config-timeline`	`ConfigTimelineLive`	Config change timeline
`/sites`	`SiteLive.Index`	Site list with device counts
`/sites/new`	`SiteLive.Form`	Create site
`/sites/:id`	`SiteLive.Show`	Site detail: devices, capacity, weather
`/sites/:id/edit`	`SiteLive.Form`	Edit site
`/insights`	`InsightsLive.Index`	AI-generated operational insights
`/trace`	`TraceLive.Index`	Subscriber trace tool
`/activity`	`ActivityFeedLive`	Activity feed (config changes, alerts, syncs)
`/schedules`	`ScheduleLive.Index`	On-call schedule list
`/schedules/new`	`ScheduleLive.Form`	Create schedule
`/schedules/:id`	`ScheduleLive.Show`	Schedule detail with calendar view
`/schedules/:id/edit`	`ScheduleLive.Form`	Edit schedule
`/schedules/escalation-policies`	`EscalationPolicyLive.Index`	Escalation policy list
`/schedules/escalation-policies/new`	`EscalationPolicyLive.Form`	Create escalation policy
`/schedules/escalation-policies/:id`	`EscalationPolicyLive.Show`	Policy detail with rules
`/schedules/escalation-policies/:id/edit`	`EscalationPolicyLive.Form`	Edit policy
`/maintenance`	`MaintenanceLive.Index`	Maintenance window list
`/maintenance/new`	`MaintenanceLive.Form`	Create maintenance window
`/maintenance/:id`	`MaintenanceLive.Show`	Window detail
`/maintenance/:id/edit`	`MaintenanceLive.Form`	Edit window
`/changelog`	`ChangelogLive`	Product changelog
`/network-map`	`NetworkMapLive`	Network topology visualization
`/sites-map`	`MapLive.Index`	Geographic site map

Organization Settings (requires auth + specific org slug)

Route	LiveView	Purpose
`/orgs/:org_slug/settings`	`Org.SettingsLive`	Org settings, SNMP defaults, billing
`/orgs/:org_slug/settings/integrations`	`Org.IntegrationsLive`	Integration management
`/orgs/:org_slug/settings/integrations/preseem/devices`	`Org.PreseemDevicesLive`	Preseem AP ↔ device matching
`/orgs/:org_slug/settings/integrations/preseem/insights`	`Org.PreseemInsightsLive`	Preseem fleet insights
`/orgs/:org_slug/settings/integrations/gaiia/mapping`	`Org.GaiiaMappingLive`	Gaiia site/device mapping
`/orgs/:org_slug/settings/integrations/gaiia/reconciliation`	`Org.GaiiaReconciliationLive`	Gaiia data reconciliation

Account / User Pages

Route	LiveView	Purpose
`/orgs`	`OrgLive.Index`	Organization picker
`/orgs/new`	`OrgLive.New`	Create organization
`/users/settings`	`UserSettingsLive`	Profile, email, password, API tokens, sessions, TOTP
`/users/my-data`	`AccountLive.MyData`	GDPR data export
`/account/activity`	`AccountLive.Activity`	User login history
`/account/totp-enrollment`	`AccountLive.TotpEnrollment`	TOTP setup wizard
`/mobile/qr-login`	`MobileQRLive`	QR code for mobile app login
`/users/register`	`UserRegistrationLive`	Registration form
`/users/reset-password/:token`	`UserResetPasswordLive`	Password reset
`/help`	`HelpLive.Index`	Help / documentation page

Admin Pages (superuser only)

Route	LiveView	Purpose
`/admin/`	`Admin.DashboardLive`	Admin overview
`/admin/users`	`Admin.UserLive.Index`	All users management
`/admin/organizations`	`Admin.OrgLive.Index`	All orgs management
`/admin/audit`	`Admin.AuditLive.Index`	Audit log viewer
`/admin/monitoring`	`Admin.MonitoringLive`	System monitoring
`/admin/agents`	`Admin.AgentLive.Index`	All agents health
`/admin/security`	`Admin.SecurityLive.Index`	IP blocks/whitelists

5. PubSub Topics

Alert Topics

Topic	Events	Subscribers
`alerts:org:{org_id}:new`	New alert created	`AlertLive.Index`, `DashboardLive`, `ActivityFeedLive`
`alerts:org:{org_id}:resolved`	Alert resolved	`AlertLive.Index`, `DashboardLive`, `ActivityFeedLive`
`organization:{org_id}:alerts`	Alert status updates	`UserAuth.subscribe_to_status_updates` (nav badge)

Device Topics

Topic	Events	Subscribers
`device:{device_id}`	Device status changes, poll results, sensor data	`DeviceLive.Show`, `GraphLive.Show`
`device:events`	All device events (cross-device)	`Devices.EventLogger`
`device_events:org:{org_id}`	Org-scoped device events	`ActivityFeedLive`
`devices:org:{org_id}`	Device created/updated/deleted	`DeviceLive.Index`, `ActivityFeedLive`
`wireless_clients:device:{device_id}`	Wireless client updates	`DeviceLive.Show`

Agent Topics

Topic	Events	Subscribers
`agents:health`	Agent heartbeat, stale detection	`AgentLive.Index`, `DeviceLive.Show`, `Admin.AgentLive`
`admin:agents`	Admin-level agent events	`Admin.AgentLive.Index`
`agent:{agent_id}:assignments`	Assignment changes	`AgentChannel`
`agent:{agent_id}:discovery`	Discovery commands	`AgentChannel`
`agent:{agent_id}:backup`	Backup commands	`AgentChannel`
`agent:{agent_id}:credential_test`	Credential test requests	`AgentChannel`
`agent:{agent_id}:live_poll`	Live poll requests	`AgentChannel`
`agent:{agent_id}:lifecycle`	Restart/update commands	`AgentChannel`
`agent:{agent_id}:latency_probe`	Latency probe requests	`AgentChannel`

Topic	Events	Subscribers
`config:org:{org_id}`	Config change events	`ActivityFeedLive`
`sync:org:{org_id}`	Integration sync events	`ActivityFeedLive`
`topology:{org_id}`	Topology link changes	`NetworkMapLive`
`credential_test:{test_id}`	SNMP credential test results	`DeviceLive.Form`
`security:blocks`	IP block changes	`Admin.SecurityLive`
`security:whitelist`	IP whitelist changes	`Admin.SecurityLive`
`job_monitoring`	Oban job health events	`Admin.MonitoringLive`

6. Workers (Oban)

Queue: `pollers`

Worker	Purpose
`DevicePollerWorker`	SNMP polling: interfaces, sensors, storage, processors, wireless, ARP, topology. The main data collection worker.

Queue: `monitors`

Worker	Purpose
`DeviceMonitorWorker`	Device up/down monitoring via ICMP ping, triggers alerts on status change

Queue: `discovery`

Worker	Purpose
`DiscoveryWorker`	SNMP device discovery: sysinfo, interfaces, sensors, profiles

Queue: `checks`

Worker	Purpose
`CheckWorker`	Schedules and dispatches monitoring check execution

Queue: `check_executors`

Worker	Purpose
`CheckExecutorWorker`	Executes individual monitoring checks (HTTP, ping, etc.)

Queue: `notifications`

Worker	Purpose
`AlertNotificationWorker`	Sends alert notifications (email, PagerDuty, on-call)
`WelcomeEmailWorker`	Sends welcome email after registration
`EscalationCheckWorker`	Checks for escalation timeouts and advances incidents

Queue: `weather`

Worker	Purpose
`WeatherSyncWorker`	Fetches weather data for sites with coordinates

Queue: `maintenance`

Worker	Purpose
`BillingSyncWorker`	Syncs device count usage to Stripe
`GaiiaSyncWorker`	Syncs data from Gaiia billing/CRM
`GaiiaInsightWorker`	Generates Gaiia-based insights
`PreseemSyncWorker`	Syncs data from Preseem
`PreseemBaselineWorker`	Calculates Preseem performance baselines
`NetboxSyncWorker`	Syncs data from NetBox DCIM
`SonarSyncWorker`	Syncs data from Sonar billing
`SplynxSyncWorker`	Syncs data from Splynx billing
`VispSyncWorker`	Syncs data from VISP billing
`MikrotikBackupWorker`	Fetches MikroTik config backups
`BackupSummaryWorker`	Generates backup comparison summaries
`BackupTimeoutWorker`	Cleans up timed-out backup requests
`FirmwareVersionFetcherWorker`	Fetches latest firmware versions from vendors
`StaleAgentWorker`	Detects agents that stopped heartbeating
`AgentLatencyEvaluator`	Evaluates agent-to-device latency data
`CloudLatencyProbeWorker`	Probes cloud poller latency
`SessionCleanupWorker`	Cleans up expired browser/mobile sessions
`LoginHistoryCleanupWorker`	Purges old login attempt records
`ExpiredBanCleanupWorker`	Removes expired IP bans
`StaleViolationCleanupWorker`	Cleans up stale rate limit violations
`CloudflareBanWorker`	Syncs IP bans to Cloudflare WAF
`JobHealthCheckWorker`	Monitors Oban job queue health
`DeviceHealthInsightWorker`	Generates device health insights
`CapacityInsightWorker`	Generates capacity utilization insights
`SystemInsightWorker`	Generates system-level insights
`WirelessInsightWorker`	Generates wireless performance insights

Additional Worker

Worker	Queue	Purpose
`Snmp.NeighborCleanupWorker`	(in snmp module)	Cleans up stale SNMP neighbor records

7. Integrations Architecture

Overview

Integrations are stored as Integration records with encrypted credentials (Cloak.Ecto). Each provider has:

A client module (Client) for API communication
A sync module (Sync) for data synchronization
An Oban worker for periodic sync

Billing/CRM Providers

Provider	Modules	Data Synced
Gaiia	`Gaiia.Client`, `Gaiia.Sync`, `Gaiia.Webhooks`, `Gaiia.ImpactAnalysis`, `Gaiia.Reconciliation`, `Gaiia.SubscriberMatching`, `Gaiia.Actions`	Accounts, billing subscriptions, network sites, inventory items, device-subscriber links
Preseem	`Preseem.Client`, `Preseem.Sync`, `Preseem.Baseline`, `Preseem.DeviceMatcher`, `Preseem.FleetIntelligence`, `Preseem.Insights`	Access points, subscriber metrics, fleet profiles, device baselines
Sonar	`Sonar.Client`, `Sonar.Sync`	Subscriber/billing data
Splynx	`Splynx.Client`, `Splynx.Sync`	Subscriber/billing data
VISP	`Visp.Client`, `Visp.Sync`	Subscriber/billing data

Infrastructure Providers

Provider	Modules	Purpose
NetBox	`Netbox.Client`, `Netbox.Sync`	DCIM inventory sync (devices, sites)
PagerDuty	`PagerDuty.Client`, `PagerDuty.Notifier`	Alert routing & incident escalation

Stripe Billing

Stripe integration is handled separately from the integration framework:

Towerops.Billing — checkout sessions, portal sessions, usage metering
BillingSyncWorker — periodic usage sync
StripeWebhookController — receives Stripe webhook events
StripeWebhookEvent schema — idempotent webhook processing
Usage-based billing: free device tier, then per-device pricing
Organization fields: stripe_customer_id, stripe_subscription_id, subscription_status

Gaiia Specifics

Gaiia has the deepest integration:

Webhook receiver at /api/v1/webhooks/gaiia/:organization_id
Impact analysis — when a device goes down, calculates affected subscribers & MRR
Reconciliation — UI for matching Gaiia network sites to TowerOps sites
Device-subscriber links — maps which subscribers are served by which device

Alert Routing

Organizations configure alert_routing:

"builtin" — uses on-call schedules + escalation policies
"pagerduty" — routes to PagerDuty
"both" — routes to both

8. Auth Flow

Authentication Stack

Request → Browser Pipeline → fetch_session → fetch_current_scope_for_user
                                                  │
                                                  ├─ Check session token
                                                  ├─ Check remember_me cookie
                                                  ├─ Build Scope (user + org + tz)
                                                  └─ Handle impersonation

`Towerops.Accounts.Scope`

The Scope struct flows through the entire request lifecycle:

%Scope{
  user: %User{},           # Current user (or impersonated user)
  superuser: %User{} | nil, # Actual superuser if impersonating
  impersonating?: boolean(),
  organization: %Organization{} | nil,
  timezone: "America/Chicago",
  time_format: "24h"
}

Session Lifecycle

Login — UserSessionController.create validates email/password
TOTP verification — If TOTP enabled, redirect to /users/log-in/totp
Session creation — Token stored in session + optional remember_me cookie
Organization loading — Default org set from membership, or from URL slug
Session renewal — Tokens reissued after configurable interval
Sudo mode — Sensitive operations require recent password re-entry (last_sudo_at)

LiveView `on_mount` Hooks

:load_cookie_consent           # Load GDPR consent state
:require_authenticated_user    # Redirect to login if no session
:require_totp_enrollment       # Redirect to TOTP setup if required
:load_current_organization     # Load org from URL slug param
:load_default_organization     # Load user's default org
:subscribe_to_status_updates   # Subscribe to org alert PubSub
:require_superuser             # Admin-only pages
:require_sudo_mode             # Require recent password entry
:handle_policy_consent         # Check privacy/terms acceptance
:redirect_if_user_is_authenticated  # Login page redirect
:require_organization_owner    # Owner-only operations

API Authentication

REST API (/api/v1/*) — Bearer token via ToweropsWeb.Plugs.ApiAuth, verified against hashed ApiToken records
GraphQL (/api/graphql) — Same bearer token auth + GraphQL.Context plug
Mobile API (/api/v1/mobile/*) — Session token via ToweropsWeb.Plugs.MobileAuth
Webhooks (/api/v1/webhooks/*) — Shared secret via ToweropsWeb.Plugs.WebhookAuth
Agent WebSocket — Token-based auth in AgentSocket/AgentChannel

Organization Switching

POST /orgs/switch — changes the current organization in session
User must have a membership in the target organization
Default org stored on Membership.is_default flag
URL-based: /orgs/:org_slug/settings/* uses slug from URL

Security Features

Rate limiting — Per-route rate limits (auth, API, admin) via RateLimit plug
Brute-force protection — Security.BruteForce with IP blocking + Cloudflare WAF sync
TOTP/MFA — Required enrollment, multi-device support, recovery codes
Sudo mode — Time-limited elevated access for sensitive operations
IP whitelisting/blocking — IpBlock, IpWhitelist schemas
Session tracking — BrowserSession records with device/location info
Impersonation — Superuser can impersonate users, tracked in Scope

9. Key Patterns

Cascading Configuration

SNMP and MikroTik credentials cascade: Organization → Site → Device. Each level can override or inherit:

# In Devices context
def resolve_agent_token_id(device) do
  get_direct_agent(device)       # Device-level assignment
  || get_site_agent(device)      # Site default agent
  || get_org_default_agent(device) # Organization default
  || get_global_default_agent()  # Global cloud poller fallback
end

AccessControl Helpers

ToweropsWeb.Live.Helpers.AccessControl provides organization-scoped access verification:

verify_device_access(device_id, organization_id)  # → {:ok, device} | {:error, :not_found | :unauthorized}
verify_site_access(site_id, organization_id)
verify_alert_access(alert_id, organization_id)

Membership Roles

Ordered by privilege: owner > admin > executive > technician > member > viewer

owner — Full access, can delete organization
admin — Full access except deleting org
executive — Read-only with financial visibility (MRR, revenue)
technician — Field ops: devices, sites, alerts, no financials
viewer — Read-only, no financials

Breadcrumbs

ToweropsWeb.Components.Breadcrumbs provides a DaisyUI-based breadcrumb component:

<.breadcrumb items={[
  %{label: "Sites", path: ~p"/sites"},
  %{label: @site.name, path: ~p"/sites/#{@site.id}"},
  %{label: "Edit"}
]} />

PubSub Pattern

Contexts broadcast changes; LiveViews subscribe on mount:

# In context (e.g., Alerts)
Phoenix.PubSub.broadcast(Towerops.PubSub, "alerts:org:#{org_id}:new", {:new_alert, alert})

# In LiveView mount
Phoenix.PubSub.subscribe(Towerops.PubSub, "alerts:org:#{organization.id}:new")

# In LiveView handle_info
def handle_info({:new_alert, alert}, socket), do: ...

Encrypted Fields

Sensitive data (SNMP passwords, API credentials, integration secrets) uses Cloak.Ecto encryption:

field :snmpv3_auth_password, Towerops.Encrypted.Binary
field :credentials, Towerops.Encrypted.Map

Multi-Tenancy

All queries are scoped by organization_id. The org context flows through:

Scope struct in conn/socket assigns
URL slug (/orgs/:org_slug/...)
Default organization from user membership

Form Pattern

LiveViews use Phoenix changesets with to_form/1:

# In mount/handle_params
changeset = Context.change_entity(entity, %{})
socket = assign(socket, form: to_form(changeset))

# In handle_event("save", ...)
case Context.update_entity(entity, params) do
  {:ok, entity} -> {:noreply, push_navigate(socket, to: ...)}
  {:error, changeset} -> {:noreply, assign(socket, form: to_form(changeset))}
end

Flash Messages

Standard Phoenix flash with custom styling:

socket |> put_flash(:info, "Device created successfully") |> push_navigate(to: ...)
socket |> put_flash(:error, "Failed to save device")

Agent Communication

Agents connect via WebSocket (AgentChannel):

Agent authenticates with token via AgentSocket
Channel subscribes to agent-specific PubSub topics
Server pushes commands: poll, discover, backup, restart, update
Agent sends results: poll data, discovery results, backup content
Protobuf used for efficient data serialization (lib/towerops/proto/)

GraphQL API

Absinthe-based GraphQL at /api/graphql:

Schema: ToweropsWeb.GraphQL.Schema
Types mirror core entities (devices, sites, alerts, agents, schedules, etc.)
Complexity analysis enabled (max 500 complexity, max 10 depth)
Subscriptions via ToweropsWeb.GraphQL.Subscriptions
API token authentication via GraphQL.Context plug

Health Checks

GET /health — Kubernetes probe endpoint (no auth)
DeviceMonitorWorker — ICMP ping monitoring
JobHealthCheckWorker — Oban queue health
AgentLatencyEvaluator — Agent responsiveness

Generated from source code analysis. Last updated: 2026-03-13.

42 KiB Raw Permalink Blame History