towerops/docs/ARCHITECTURE.md

TowerOps Architecture

Network infrastructure monitoring platform for ISPs/WISPs built with Phoenix LiveView, Oban, and TimescaleDB.

---

1. Project Structure

towerops/
├── assets/                    # Frontend: JS, CSS, Tailwind
├── config/                    # Runtime & env config (dev/test/prod/runtime)
├── c_src/                     # C NIFs (SNMP native interop)
├── docs/                      # Documentation
├── e2e/                       # End-to-end tests (Playwright)
├── flux/                      # FluxCD / GitOps manifests
├── k8s/                       # Kubernetes manifests
├── lib/
│   ├── mix/                   # Custom Mix tasks
│   ├── snmpkit/               # SNMP toolkit (native extension)
│   ├── towerops/              # Core business logic (contexts, schemas, workers)
│   └── towerops_web/          # Web layer (LiveView, controllers, channels, GraphQL)
├── mibs/                      # SNMP MIB files
├── priv/
│   ├── repo/migrations/       # Ecto migrations
│   ├── static/                # Static assets
│   └── gettext/               # i18n translations
├── rel/                       # Release configuration
├── scripts/                   # Operational scripts
├── test/                      # Test suite
├── towerops/                  # Agent-related files
└── vendor/                    # Vendored dependencies

Key lib/towerops/ Subdirectories

Directory	Purpose
accounts/	User, auth tokens, TOTP, recovery codes, sessions, consent
admin/	Superuser audit logs
agents/	Remote polling agent tokens & assignments
alerts/	Alert schema and queries
api_tokens/	API token management (hashed bearer tokens)
billing/	Stripe integration, usage metering
capacity/	Interface throughput & utilization calculations
config_changes/	MikroTik config change event tracking
contexts/	Shared context helpers
devices/	Device schema, backups, firmware, events
ecto_types/	Custom Ecto types (IP address, MAC address, SNMP OID)
gaiia/	Gaiia billing/CRM integration (accounts, subscriptions, impact analysis)
geoip/	GeoIP lookup (blocks, locations)
integrations/	Integration schema (multi-provider)
job_monitoring/	Oban job health events
maintenance/	Maintenance windows
mobile_sessions/	Mobile app sessions & QR login
monitoring/	Check definitions & check results
netbox/	NetBox DCIM sync
on_call/	Schedules, escalation policies, incidents, notifications
organizations/	Org, membership, invitations, policies
pagerduty/	PagerDuty client & notifier
preseem/	Preseem QoE integration (APs, baselines, fleet intelligence)
profiles/	SNMP device profiles (OID mappings)
proto/	Protobuf definitions (agent communication)
security/	Brute-force protection, IP blocks/whitelists
settings/	Global application settings (key-value store)
sites/	Site schema & tree structure
snmp/	SNMP device data (interfaces, sensors, readings, wireless, topology)
sonar/	Sonar billing integration
splynx/	Splynx billing integration
topology/	Network topology discovery (LLDP, CDP, ARP, MAC)
visp/	VISP billing integration
weather/	Weather observations & alerts per site
workers/	All Oban background workers

Key lib/towerops_web/ Subdirectories

Directory	Purpose
channels/	Phoenix Channels (agent WebSocket)
components/	Reusable UI components (breadcrumbs, tables, forms)
controllers/	Traditional controllers (auth, webhooks, health, API)
graphql/	Absinthe GraphQL schema, types, resolvers, subscriptions
helpers/	View helpers (time formatting, status display)
live/	All LiveView pages
plugs/	Plug middleware (auth, rate limiting, security headers)

---

2. Data Model

All primary keys are UUIDs (:binary_id). Multi-tenancy is organization-scoped.

Core Entities

┌──────────────┐     ┌──────────────────┐     ┌──────────────┐
│     User     │────<│   Membership     │>────│ Organization │
│              │     │ (role-based)     │     │              │
│ email        │     │ role: owner|     │     │ name, slug   │
│ hashed_pass  │     │  admin|executive │     │ subscription │
│ totp_secret  │     │  |technician|    │     │ snmp_config  │
│ is_superuser │     │  member|viewer   │     │ mikrotik_cfg │
│ timezone     │     │ is_default       │     │ stripe_*     │
│ default_org  │     └──────────────────┘     │ alert_routing│
└──────────────┘                               └──────┬───────┘
                                                      │
                              ┌────────────────┬──────┴──────┬───────────────┐
                              │                │             │               │
                        ┌─────┴─────┐   ┌──────┴──────┐ ┌───┴────┐  ┌──────┴──────┐
                        │   Site    │   │   Device    │ │ Alert  │  │ Integration │
                        │           │   │             │ │        │  │             │
                        │ name      │   │ name        │ │ type   │  │ provider    │
                        │ lat/lng   │   │ ip_address  │ │ sev    │  │ credentials │
                        │ address   │   │ status      │ │ msg    │  │ sync_status │
                        │ snmp_cfg  │   │ snmp_cfg    │ │ device │  │ (encrypted) │
                        │ parent_id │   │ mikrotik_cfg│ │ check  │  └─────────────┘
                        │ agent_tok │   │ site_id     │ └────────┘
                        └───────────┘   │ agent_assign│
                                        │ check_intv  │
                                        └─────────────┘

On-Call / Incident Management

Schedule ──has_many──> Layer ──has_many──> LayerMember ──belongs_to──> User
    │
    └──has_many──> Override (user, start_at, end_at)

EscalationPolicy ──has_many──> EscalationRule ──has_many──> EscalationTarget
    │                              │                            │
    │                              │ position, delay_minutes    ├── schedule_id
    │                              └────────────────────────    └── user_id
    │
Incident ──belongs_to──> Alert
    │                     EscalationPolicy
    │                     Organization
    └──has_many──> Notification ──belongs_to──> User

SNMP Data Model

Device (towerops) ──1:1──> Snmp.Device
    │
    ├── Snmp.Interface ──> InterfaceStat (time-series)
    ├── Snmp.Sensor ──> SensorReading (time-series)
    ├── Snmp.StateSensor
    ├── Snmp.Processor ──> ProcessorReading
    ├── Snmp.Storage ──> StorageReading
    ├── Snmp.WirelessClient ──> WirelessClientReading
    ├── Snmp.Neighbor (LLDP/CDP)
    ├── Snmp.ArpEntry
    ├── Snmp.Vlan
    └── Snmp.PhysicalEntity

Billing / CRM Schemas

• Gaiia: Account, BillingSubscription, NetworkSite, InventoryItem, DeviceSubscriberLink
• Preseem: AccessPoint, SubscriberMetric, FleetProfile, DeviceBaseline, Insight, SyncLog
• Billing: StripeWebhookEvent

Other Schemas

Schema	Table	Purpose
AgentToken	agent_tokens	Remote polling agent credentials
AgentAssignment	agent_assignments	Device → agent mapping
ApiToken	api_tokens	REST/GraphQL API bearer tokens
BrowserSession	browser_sessions	Active login sessions
LoginAttempt	login_attempts	Auth attempt audit log
UserConsent	user_consents	Privacy/terms consent records
PolicyVersion	policy_versions	Privacy policy versions
UserTotpDevice	user_totp_devices	Multi-device TOTP
UserRecoveryCode	user_recovery_codes	MFA recovery codes
MaintenanceWindow	maintenance_windows	Scheduled maintenance
ConfigChangeEvent	config_change_events	MikroTik config diffs
DeviceFirmwareHistory	device_firmware_histories	Firmware change tracking
FirmwareRelease	firmware_releases	Known firmware versions
BackupRequest	backup_requests	MikroTik backup queue
MikrotikBackup	mikrotik_backups	Stored config backups
Check	checks	Monitoring check definitions
CheckResult	check_results	Check execution results
MonitoringCheck	monitoring_checks	Check configuration
DeviceLink	device_links	Discovered topology links
DeviceLinkEvidence	device_link_evidence	Link confidence scores
DeviceNeighbor	device_neighbors	Raw neighbor data
ApplicationSetting	application_settings	Global config KV store
AuditLog	audit_logs	Admin action audit trail
IpBlock	ip_blocks	Blocked IPs
IpWhitelist	ip_whitelists	Whitelisted IPs
MobileSession	mobile_sessions	Mobile app sessions
QrLoginToken	qr_login_tokens	QR-based mobile auth
Weather.Observation	weather_observations	Site weather data
Weather.Alert	weather_alerts	Active weather alerts
GeoIp.Block / Location	geoip_*	IP geolocation

---

3. Context Modules

Towerops.Accounts

User management, authentication, TOTP/MFA, password management, email changes, session tracking.

# User lookup
def get_user_by_email(email)
def get_user_by_email_and_password(email, password)
def get_user(id)
def get_user!(id)

# Registration
def register_user(attrs)
def register_user_with_organization(attrs)
def change_user_registration(user, attrs \\ %{})

# TOTP / MFA
def generate_totp_secret()
def generate_totp_uri(user, secret)
def generate_totp_qr_code(user, secret)
def verify_totp(secret, code)
def enable_totp(user, secret)
def totp_enabled?(user)
def verify_user_totp(user, code)
def list_user_totp_devices(user_id)
def create_totp_device(user_id, device_name)
def verify_user_totp_any_device(user, code)
def delete_totp_device(device_id, user_id)
def rename_totp_device(device_id, user_id, new_name)
def generate_recovery_codes(user_id)
def verify_recovery_code(user_id, code)
def count_unused_recovery_codes(user_id)
def verify_user_mfa(user, code)

# Profile & Settings
def change_user_profile(user, attrs \\ %{})
def update_user_profile(user, attrs)
def sudo_mode?(user, minutes \\ -10)
def grant_sudo_mode(user)

# Email & Password
def change_user_email(user, attrs \\ %{}, opts \\ [])
def update_user_email(user, token)
def change_user_password(user, attrs \\ %{}, opts \\ [])

# Sessions (+ more token functions not listed)

Towerops.Organizations

Multi-tenant org management, memberships, invitations, SNMP/agent propagation.

def list_user_organizations(user_id)
def user_has_access?(user_id, organization_id)
def get_organization!(id)
def get_organization_by_slug!(slug)
def create_organization(attrs, user_id, opts \\ [])
def update_organization(organization, attrs)
def delete_organization(organization)
def change_organization(organization, attrs \\ %{})

# Memberships
def get_membership(organization_id, user_id)
def list_organization_memberships(organization_id)
def list_organization_members(organization_id)
def remove_member(organization_id, user_id)
def update_member_role(organization_id, user_id, new_role)
def create_membership(attrs)
def get_default_membership(user_id)
def set_default_organization(user_id, organization_id)
def list_organization_notification_recipients(organization_id)

# Invitations
def list_pending_invitations(organization_id)
def create_invitation(attrs)
def get_invitation_by_token(token)
def accept_invitation(invitation, user_id)
def delete_invitation(invitation)

# Config propagation
def apply_snmp_config_to_all_equipment(organization_id)
def apply_agent_to_all_equipment(organization_id)
def clear_all_site_assignments(organization_id)

# Billing lookups
def get_organization_by_stripe_customer_id(stripe_customer_id)
def list_organizations_with_active_subscriptions()

Towerops.Devices

Device CRUD, SNMP/MikroTik config resolution (cascading: org → site → device), monitoring.

def list_site_devices(site_id)
def list_organization_devices(organization_id, filters \\ %{})
def count_organization_devices(organization_id)
def search_devices(organization_id, query)
def get_device_status_counts(organization_id)
def list_monitored_devices()
def list_snmp_enabled_devices()
def list_mikrotik_devices_with_api()
def get_device(id)
def get_device!(id)
def get_device_with_details(id)
def get_device_by_ip(ip_address, site_id, exclude_id \\ nil)
def get_snmp_config(device)          # Resolves cascading config
def get_mikrotik_config(device)      # Resolves cascading config
def get_snmpv3_config(device)        # Resolves cascading config
def resolve_agent_token_id(device)   # Device → site → org → global fallback
def propagate_site_snmpv3_change(site_id, attrs)
def propagate_organization_mikrotik_change(organization_id, attrs)
# ... CRUD, reordering, etc.

Towerops.Sites

Site CRUD, hierarchical tree structure, config propagation.

def list_organization_sites(organization_id)
def list_root_sites(organization_id)
def list_child_sites(parent_site_id)
def build_site_tree(organization_id)
def list_sites_with_coordinates()
def search_sites(organization_id, query)
def get_site!(id)
def create_site(attrs)
def update_site(site, attrs)
def delete_site(site)
def apply_snmp_config_to_all_equipment(site_id)
def apply_agent_to_all_equipment(site_id)
def reorder_site(site_id, new_position)

Towerops.Alerts

Alert lifecycle: creation, acknowledgement, resolution, Gaiia impact analysis.

def create_alert(attrs)
def list_devices_alerts(device_id, limit \\ 100)
def list_organization_active_alerts(organization_id)
def list_organization_alerts(organization_id, filters)
def count_active_alerts(organization_id)
def get_alert!(id)
def get_alert(id)
def acknowledge_alert(alert, user_id)
def resolve_alert(alert)
def resolve_alert_silent(alert)         # No PubSub broadcast
def acknowledge_alert_silent(alert)     # No PubSub broadcast
def has_active_alert?(device_id, alert_type)
def get_active_alert(device_id, alert_type)
def has_active_check_alert?(check_id)
def resolve_check_alerts(check_id)
def store_gaiia_impact(alert, impact_data)

Towerops.Integrations

Generic integration CRUD for all billing/CRM providers.

def list_integrations(organization_id)
def get_integration(organization_id, provider)
def create_integration(organization_id, attrs)
def update_integration(integration, attrs)
def update_sync_status(integration, status, message \\ nil)
def delete_integration(integration)
def get_billing_summary(organization_id)
def list_enabled_integrations(provider)
def update_billing_totals(integration, subscriber_count, total_mrr)

Valid providers: preseem, gaiia, pagerduty, netbox, sonar, splynx, visp

Towerops.OnCall

On-call schedules, escalation policies, incidents.

# Schedules
def list_schedules(organization_id)
def get_schedule!(id) / get_schedule(id)
def create_schedule(attrs) / update_schedule(schedule, attrs) / delete_schedule(schedule)
def create_layer(attrs) / update_layer(layer, attrs) / delete_layer(layer)
def add_layer_member(attrs) / remove_layer_member(member)
def create_override(attrs) / delete_override(override)
def who_is_on_call(schedule_id, datetime \\ DateTime.utc_now())

# Escalation Policies
def list_escalation_policies(organization_id)
def get_escalation_policy!(id) / get_escalation_policy(id)
def create_escalation_policy(attrs) / update_escalation_policy(policy, attrs) / delete_escalation_policy(policy)
def create_escalation_rule(attrs) / update_escalation_rule(rule, attrs) / delete_escalation_rule(rule)
def create_escalation_target(attrs) / delete_escalation_target(target)

Towerops.Billing

Stripe billing integration: checkout, portal, usage metering, subscription management.

def default_free_devices()
def default_price_per_device()
def create_checkout_session(organization, opts \\ [])
def create_portal_session(organization, return_url)
def billable_device_count(organization)
def effective_free_device_count(org)
def effective_price_per_device(org)
def estimated_monthly_cost(organization)
def sync_usage_to_stripe(organization)
def update_subscription_from_stripe(organization, stripe_subscription)
def update_payment_method_status(organization, status)
def migrate_all_subscriptions_to_price(new_price_id)

Towerops.Agents

Remote polling agent lifecycle: tokens, heartbeat, assignments, mass updates, debug mode.

def create_agent_token(organization_id, name)
def create_cloud_poller(name)
def list_organization_agent_tokens(organization_id)
def list_cloud_pollers()
def verify_agent_token(token)
def update_agent_token_heartbeat(agent_token_id, ip_address, metadata \\ nil)
def update_agent_token(agent_token, attrs)
def restart_agent(agent_token_id)
def update_agent(agent_token_id, url, checksum)
def toggle_agent_debug(agent_token, enabled, user)
def revoke_agent_token(id)
def delete_agent_token(id)
def assign_device_to_agent(agent_token_id, device_id)
def unassign_device(device_id)
def list_agent_devices(agent_token_id)
def list_agent_polling_targets(agent_token_id)
def get_effective_agent_token(device)
def broadcast_mass_update()

Towerops.Monitoring

Check definitions, execution, result tracking, state machine (soft/hard states).

def list_checks(organization_id, opts \\ [])
def get_check(id) / get_check!(id)
def create_check(attrs) / update_check(check, attrs) / delete_check(check)
def schedule_check(check)
def create_check_result(attrs)
def get_latest_check_result(check_id)
def get_check_results(check_id, opts \\ [])
def get_check_graph_data(check_id, from_time, to_time)
def update_check_state(check, status, output)     # Soft/hard state machine
def stop_device_checks(device_id)
def disable_device_checks(device_id)
def get_device_health_summary(device_ids)
def get_device_latest_response_times(device_ids)

Towerops.Snmp

SNMP device discovery, interface/sensor management, readings.

def test_connection(ip, community, version, port \\ 161)
def discover_device(device)
def discover_all_for_org(org_id)
def create_checks_from_discovery(device, snmp_device)
def get_device(device_id) / get_device_with_associations(device_id)
def list_interfaces(device_id) / list_monitored_interfaces(device_id)
def list_sensors(device_id) / list_sensors_by_type(device_id)
def get_sensor_readings(sensor_id, opts \\ [])
def get_interface_stats(interface_id, opts \\ [])
def set_manual_capacity(interface_id, capacity_bps)
# ... many more reading/stat functions

Towerops.Gaiia

Gaiia CRM/billing data: accounts, subscriptions, network sites, inventory, impact analysis.

def list_accounts(organization_id)
def upsert_account(organization_id, attrs)
def count_active_subscribers(organization_id)
def sum_active_mrr(organization_id)
def list_network_sites(organization_id)
def upsert_network_site(organization_id, attrs)
def list_inventory_items(organization_id)
def list_billing_subscriptions(organization_id)
def get_device_subscriber_counts(device_ids)
def suggest_site_matches(organization_id, network_site)
def suggest_device_matches(organization_id, inventory_item)

Sub-modules: Gaiia.Client, Gaiia.Webhooks, Gaiia.ImpactAnalysis, Gaiia.Reconciliation, Gaiia.SubscriberMatching, Gaiia.Actions

Towerops.Preseem

Preseem QoE integration: access points, subscriber metrics, fleet intelligence.

def get_access_point_for_device(device_id)
def list_access_points(organization_id)
def list_unmatched_access_points(organization_id)
def list_matched_access_points(organization_id)
def list_subscriber_metrics(access_point_id, opts \\ [])
def link_access_point(access_point_id, device_id)
def unlink_access_point(access_point_id)
def get_site_qoe_summary(site_id)

Sub-modules: Preseem.Client, Preseem.Sync, Preseem.Baseline, Preseem.DeviceMatcher, Preseem.FleetIntelligence, Preseem.Insights

Other Context Modules

Module	Purpose
Towerops.Maintenance	Maintenance window CRUD, device/site in-maintenance checks
Towerops.Topology	Network topology discovery (LLDP/CDP/ARP/MAC), link inference
Towerops.Dashboard	Dashboard summary, health scores, impact analysis
Towerops.Search	Cross-entity full-text search (devices, sites, accounts, alerts)
Towerops.ActivityFeed	Aggregated activity feed (config changes, alerts, syncs, events)
Towerops.ApiTokens	API token CRUD, hashed token verification
Towerops.Capacity	Interface throughput & utilization calculations
Towerops.ConfigChanges	MikroTik config change diff tracking
Towerops.MobileSessions	Mobile app sessions, QR login flow
Towerops.Weather	Weather observations & alerts per site
Towerops.Admin	Superuser admin: user/org listing, audit logs, billing overrides
Towerops.Profiles	SNMP device profile matching & OID management
Towerops.Settings	Global application settings (key-value store)
Towerops.Geocoding	Address → lat/lng geocoding
Towerops.GeoIp	IP → location lookup
Towerops.Http	Shared HTTP client helpers
Towerops.Trace	Subscriber trace functionality
Towerops.RateLimit	Rate limiting logic
Towerops.Numeric	Numeric formatting utilities

---

4. LiveView Pages

Main Application (requires auth + org)

Route	LiveView	Purpose
/dashboard	DashboardLive	Org dashboard with health score, alert summary, site overview
/alerts	AlertLive.Index	Alert list with filtering (active/ack/resolved), real-time updates
/agents	AgentLive.Index	Agent list with health status
/agents/:id	AgentLive.Show	Agent detail: assigned devices, heartbeat, metadata
/agents/:id/edit	AgentLive.Edit	Edit agent settings
/devices	DeviceLive.Index	Device list with status, filtering, real-time updates
/devices/new	DeviceLive.Form	Create device form (with SNMP credential test)
/devices/:id	DeviceLive.Show	Device detail: status, interfaces, sensors, alerts, backups
/devices/:id/edit	DeviceLive.Form	Edit device form
/devices/:id/graph/:sensor_type	GraphLive.Show	Time-series graphs for device metrics
/devices/:device_id/backups/compare	MikrotikBackupLive.Compare	Config backup diff viewer
/devices/:device_id/config-timeline	ConfigTimelineLive	Config change timeline
/sites	SiteLive.Index	Site list with device counts
/sites/new	SiteLive.Form	Create site
/sites/:id	SiteLive.Show	Site detail: devices, capacity, weather
/sites/:id/edit	SiteLive.Form	Edit site
/insights	InsightsLive.Index	AI-generated operational insights
/trace	TraceLive.Index	Subscriber trace tool
/activity	ActivityFeedLive	Activity feed (config changes, alerts, syncs)
/schedules	ScheduleLive.Index	On-call schedule list
/schedules/new	ScheduleLive.Form	Create schedule
/schedules/:id	ScheduleLive.Show	Schedule detail with calendar view
/schedules/:id/edit	ScheduleLive.Form	Edit schedule
/schedules/escalation-policies	EscalationPolicyLive.Index	Escalation policy list
/schedules/escalation-policies/new	EscalationPolicyLive.Form	Create escalation policy
/schedules/escalation-policies/:id	EscalationPolicyLive.Show	Policy detail with rules
/schedules/escalation-policies/:id/edit	EscalationPolicyLive.Form	Edit policy
/maintenance	MaintenanceLive.Index	Maintenance window list
/maintenance/new	MaintenanceLive.Form	Create maintenance window
/maintenance/:id	MaintenanceLive.Show	Window detail
/maintenance/:id/edit	MaintenanceLive.Form	Edit window
/changelog	ChangelogLive	Product changelog
/network-map	NetworkMapLive	Network topology visualization
/sites-map	MapLive.Index	Geographic site map

Organization Settings (requires auth + specific org slug)

Route	LiveView	Purpose
/orgs/:org_slug/settings	Org.SettingsLive	Org settings, SNMP defaults, billing
/orgs/:org_slug/settings/integrations	Org.IntegrationsLive	Integration management
/orgs/:org_slug/settings/integrations/preseem/devices	Org.PreseemDevicesLive	Preseem AP ↔ device matching
/orgs/:org_slug/settings/integrations/preseem/insights	Org.PreseemInsightsLive	Preseem fleet insights
/orgs/:org_slug/settings/integrations/gaiia/mapping	Org.GaiiaMappingLive	Gaiia site/device mapping
/orgs/:org_slug/settings/integrations/gaiia/reconciliation	Org.GaiiaReconciliationLive	Gaiia data reconciliation

Account / User Pages

Route	LiveView	Purpose
/orgs	OrgLive.Index	Organization picker
/orgs/new	OrgLive.New	Create organization
/users/settings	UserSettingsLive	Profile, email, password, API tokens, sessions, TOTP
/users/my-data	AccountLive.MyData	GDPR data export
/account/activity	AccountLive.Activity	User login history
/account/totp-enrollment	AccountLive.TotpEnrollment	TOTP setup wizard
/mobile/qr-login	MobileQRLive	QR code for mobile app login
/users/register	UserRegistrationLive	Registration form
/users/reset-password/:token	UserResetPasswordLive	Password reset
/help	HelpLive.Index	Help / documentation page

Admin Pages (superuser only)

Route	LiveView	Purpose
/admin/	Admin.DashboardLive	Admin overview
/admin/users	Admin.UserLive.Index	All users management
/admin/organizations	Admin.OrgLive.Index	All orgs management
/admin/audit	Admin.AuditLive.Index	Audit log viewer
/admin/monitoring	Admin.MonitoringLive	System monitoring
/admin/agents	Admin.AgentLive.Index	All agents health
/admin/security	Admin.SecurityLive.Index	IP blocks/whitelists

---

5. PubSub Topics

Alert Topics

Topic	Events	Subscribers
alerts:org:{org_id}:new	New alert created	AlertLive.Index, DashboardLive, ActivityFeedLive
alerts:org:{org_id}:resolved	Alert resolved	AlertLive.Index, DashboardLive, ActivityFeedLive
organization:{org_id}:alerts	Alert status updates	UserAuth.subscribe_to_status_updates (nav badge)

Device Topics

Topic	Events	Subscribers
device:{device_id}	Device status changes, poll results, sensor data	DeviceLive.Show, GraphLive.Show
device:events	All device events (cross-device)	Devices.EventLogger
device_events:org:{org_id}	Org-scoped device events	ActivityFeedLive
devices:org:{org_id}	Device created/updated/deleted	DeviceLive.Index, ActivityFeedLive
wireless_clients:device:{device_id}	Wireless client updates	DeviceLive.Show

Agent Topics

Topic	Events	Subscribers
agents:health	Agent heartbeat, stale detection	AgentLive.Index, DeviceLive.Show, Admin.AgentLive
admin:agents	Admin-level agent events	Admin.AgentLive.Index
agent:{agent_id}:assignments	Assignment changes	AgentChannel
agent:{agent_id}:discovery	Discovery commands	AgentChannel
agent:{agent_id}:backup	Backup commands	AgentChannel
agent:{agent_id}:credential_test	Credential test requests	AgentChannel
agent:{agent_id}:live_poll	Live poll requests	AgentChannel
agent:{agent_id}:lifecycle	Restart/update commands	AgentChannel
agent:{agent_id}:latency_probe	Latency probe requests	AgentChannel

Other Topics

Topic	Events	Subscribers
config:org:{org_id}	Config change events	ActivityFeedLive
sync:org:{org_id}	Integration sync events	ActivityFeedLive
topology:{org_id}	Topology link changes	NetworkMapLive
credential_test:{test_id}	SNMP credential test results	DeviceLive.Form
security:blocks	IP block changes	Admin.SecurityLive
security:whitelist	IP whitelist changes	Admin.SecurityLive
job_monitoring	Oban job health events	Admin.MonitoringLive

---

6. Workers (Oban)

Queue: pollers

Worker	Purpose
DevicePollerWorker	SNMP polling: interfaces, sensors, storage, processors, wireless, ARP, topology. The main data collection worker.

Queue: monitors

Worker	Purpose
DeviceMonitorWorker	Device up/down monitoring via ICMP ping, triggers alerts on status change

Queue: discovery

Worker	Purpose
DiscoveryWorker	SNMP device discovery: sysinfo, interfaces, sensors, profiles

Queue: checks

Worker	Purpose
CheckWorker	Schedules and dispatches monitoring check execution

Queue: check_executors

Worker	Purpose
CheckExecutorWorker	Executes individual monitoring checks (HTTP, ping, etc.)

Queue: notifications

Worker	Purpose
AlertNotificationWorker	Sends alert notifications (email, PagerDuty, on-call)
WelcomeEmailWorker	Sends welcome email after registration
EscalationCheckWorker	Checks for escalation timeouts and advances incidents

Queue: weather

Worker	Purpose
WeatherSyncWorker	Fetches weather data for sites with coordinates

Queue: maintenance

Worker	Purpose
BillingSyncWorker	Syncs device count usage to Stripe
GaiiaSyncWorker	Syncs data from Gaiia billing/CRM
GaiiaInsightWorker	Generates Gaiia-based insights
PreseemSyncWorker	Syncs data from Preseem
PreseemBaselineWorker	Calculates Preseem performance baselines
NetboxSyncWorker	Syncs data from NetBox DCIM
SonarSyncWorker	Syncs data from Sonar billing
SplynxSyncWorker	Syncs data from Splynx billing
VispSyncWorker	Syncs data from VISP billing
MikrotikBackupWorker	Fetches MikroTik config backups
BackupSummaryWorker	Generates backup comparison summaries
BackupTimeoutWorker	Cleans up timed-out backup requests
FirmwareVersionFetcherWorker	Fetches latest firmware versions from vendors
StaleAgentWorker	Detects agents that stopped heartbeating
AgentLatencyEvaluator	Evaluates agent-to-device latency data
CloudLatencyProbeWorker	Probes cloud poller latency
SessionCleanupWorker	Cleans up expired browser/mobile sessions
LoginHistoryCleanupWorker	Purges old login attempt records
ExpiredBanCleanupWorker	Removes expired IP bans
StaleViolationCleanupWorker	Cleans up stale rate limit violations
CloudflareBanWorker	Syncs IP bans to Cloudflare WAF
JobHealthCheckWorker	Monitors Oban job queue health
DeviceHealthInsightWorker	Generates device health insights
CapacityInsightWorker	Generates capacity utilization insights
SystemInsightWorker	Generates system-level insights
WirelessInsightWorker	Generates wireless performance insights

Additional Worker

Worker	Queue	Purpose
Snmp.NeighborCleanupWorker	(in snmp module)	Cleans up stale SNMP neighbor records

---

7. Integrations Architecture

Overview

Integrations are stored as Integration records with encrypted credentials (Cloak.Ecto). Each provider has:

• A client module (Client) for API communication
• A sync module (Sync) for data synchronization
• An Oban worker for periodic sync

Billing/CRM Providers

Provider	Modules	Data Synced
Gaiia	Gaiia.Client, Gaiia.Sync, Gaiia.Webhooks, Gaiia.ImpactAnalysis, Gaiia.Reconciliation, Gaiia.SubscriberMatching, Gaiia.Actions	Accounts, billing subscriptions, network sites, inventory items, device-subscriber links
Preseem	Preseem.Client, Preseem.Sync, Preseem.Baseline, Preseem.DeviceMatcher, Preseem.FleetIntelligence, Preseem.Insights	Access points, subscriber metrics, fleet profiles, device baselines
Sonar	Sonar.Client, Sonar.Sync	Subscriber/billing data
Splynx	Splynx.Client, Splynx.Sync	Subscriber/billing data
VISP	Visp.Client, Visp.Sync	Subscriber/billing data

Infrastructure Providers

Provider	Modules	Purpose
NetBox	Netbox.Client, Netbox.Sync	DCIM inventory sync (devices, sites)
PagerDuty	PagerDuty.Client, PagerDuty.Notifier	Alert routing & incident escalation

Stripe Billing

Stripe integration is handled separately from the integration framework:

• Towerops.Billing — checkout sessions, portal sessions, usage metering
• BillingSyncWorker — periodic usage sync
• StripeWebhookController — receives Stripe webhook events
• StripeWebhookEvent schema — idempotent webhook processing
• Usage-based billing: free device tier, then per-device pricing
• Organization fields: stripe_customer_id, stripe_subscription_id, subscription_status

Gaiia Specifics

Gaiia has the deepest integration:

• Webhook receiver at /api/v1/webhooks/gaiia/:organization_id
• Impact analysis — when a device goes down, calculates affected subscribers & MRR
• Reconciliation — UI for matching Gaiia network sites to TowerOps sites
• Device-subscriber links — maps which subscribers are served by which device

Alert Routing

Organizations configure alert_routing:

• "builtin" — uses on-call schedules + escalation policies
• "pagerduty" — routes to PagerDuty
• "both" — routes to both

---

8. Auth Flow

Authentication Stack

Request → Browser Pipeline → fetch_session → fetch_current_scope_for_user
                                                  │
                                                  ├─ Check session token
                                                  ├─ Check remember_me cookie
                                                  ├─ Build Scope (user + org + tz)
                                                  └─ Handle impersonation

Towerops.Accounts.Scope

The Scope struct flows through the entire request lifecycle:

%Scope{
  user: %User{},           # Current user (or impersonated user)
  superuser: %User{} | nil, # Actual superuser if impersonating
  impersonating?: boolean(),
  organization: %Organization{} | nil,
  timezone: "America/Chicago",
  time_format: "24h"
}

Session Lifecycle

1. Login — UserSessionController.create validates email/password
2. TOTP verification — If TOTP enabled, redirect to /users/log-in/totp
3. Session creation — Token stored in session + optional remember_me cookie
4. Organization loading — Default org set from membership, or from URL slug
5. Session renewal — Tokens reissued after configurable interval
6. Sudo mode — Sensitive operations require recent password re-entry (last_sudo_at)

LiveView on_mount Hooks

:load_cookie_consent           # Load GDPR consent state
:require_authenticated_user    # Redirect to login if no session
:require_totp_enrollment       # Redirect to TOTP setup if required
:load_current_organization     # Load org from URL slug param
:load_default_organization     # Load user's default org
:subscribe_to_status_updates   # Subscribe to org alert PubSub
:require_superuser             # Admin-only pages
:require_sudo_mode             # Require recent password entry
:handle_policy_consent         # Check privacy/terms acceptance
:redirect_if_user_is_authenticated  # Login page redirect
:require_organization_owner    # Owner-only operations

API Authentication

• REST API (/api/v1/*) — Bearer token via ToweropsWeb.Plugs.ApiAuth, verified against hashed ApiToken records
• GraphQL (/api/graphql) — Same bearer token auth + GraphQL.Context plug
• Mobile API (/api/v1/mobile/*) — Session token via ToweropsWeb.Plugs.MobileAuth
• Webhooks (/api/v1/webhooks/*) — Shared secret via ToweropsWeb.Plugs.WebhookAuth
• Agent WebSocket — Token-based auth in AgentSocket/AgentChannel

Organization Switching

• POST /orgs/switch — changes the current organization in session
• User must have a membership in the target organization
• Default org stored on Membership.is_default flag
• URL-based: /orgs/:org_slug/settings/* uses slug from URL

Security Features

• Rate limiting — Per-route rate limits (auth, API, admin) via RateLimit plug
• Brute-force protection — Security.BruteForce with IP blocking + Cloudflare WAF sync
• TOTP/MFA — Required enrollment, multi-device support, recovery codes
• Sudo mode — Time-limited elevated access for sensitive operations
• IP whitelisting/blocking — IpBlock, IpWhitelist schemas
• Session tracking — BrowserSession records with device/location info
• Impersonation — Superuser can impersonate users, tracked in Scope

---

9. Key Patterns

Cascading Configuration

SNMP and MikroTik credentials cascade: Organization → Site → Device. Each level can override or inherit:

# In Devices context
def resolve_agent_token_id(device) do
  get_direct_agent(device)       # Device-level assignment
  || get_site_agent(device)      # Site default agent
  || get_org_default_agent(device) # Organization default
  || get_global_default_agent()  # Global cloud poller fallback
end

AccessControl Helpers

ToweropsWeb.Live.Helpers.AccessControl provides organization-scoped access verification:

verify_device_access(device_id, organization_id)  # → {:ok, device} | {:error, :not_found | :unauthorized}
verify_site_access(site_id, organization_id)
verify_alert_access(alert_id, organization_id)

Membership Roles

Ordered by privilege: owner > admin > executive > technician > member > viewer

• owner — Full access, can delete organization
• admin — Full access except deleting org
• executive — Read-only with financial visibility (MRR, revenue)
• technician — Field ops: devices, sites, alerts, no financials
• viewer — Read-only, no financials

Breadcrumbs

ToweropsWeb.Components.Breadcrumbs provides a DaisyUI-based breadcrumb component:

<.breadcrumb items={[
  %{label: "Sites", path: ~p"/sites"},
  %{label: @site.name, path: ~p"/sites/#{@site.id}"},
  %{label: "Edit"}
]} />

PubSub Pattern

Contexts broadcast changes; LiveViews subscribe on mount:

# In context (e.g., Alerts)
Phoenix.PubSub.broadcast(Towerops.PubSub, "alerts:org:#{org_id}:new", {:new_alert, alert})

# In LiveView mount
Phoenix.PubSub.subscribe(Towerops.PubSub, "alerts:org:#{organization.id}:new")

# In LiveView handle_info
def handle_info({:new_alert, alert}, socket), do: ...

Encrypted Fields

Sensitive data (SNMP passwords, API credentials, integration secrets) uses Cloak.Ecto encryption:

field :snmpv3_auth_password, Towerops.Encrypted.Binary
field :credentials, Towerops.Encrypted.Map

Multi-Tenancy

All queries are scoped by organization_id. The org context flows through:

1. Scope struct in conn/socket assigns
2. URL slug (/orgs/:org_slug/...)
3. Default organization from user membership

Form Pattern

LiveViews use Phoenix changesets with to_form/1:

# In mount/handle_params
changeset = Context.change_entity(entity, %{})
socket = assign(socket, form: to_form(changeset))

# In handle_event("save", ...)
case Context.update_entity(entity, params) do
  {:ok, entity} -> {:noreply, push_navigate(socket, to: ...)}
  {:error, changeset} -> {:noreply, assign(socket, form: to_form(changeset))}
end

Flash Messages

Standard Phoenix flash with custom styling:

socket |> put_flash(:info, "Device created successfully") |> push_navigate(to: ...)
socket |> put_flash(:error, "Failed to save device")

Agent Communication

Agents connect via WebSocket (AgentChannel):

1. Agent authenticates with token via AgentSocket
2. Channel subscribes to agent-specific PubSub topics
3. Server pushes commands: poll, discover, backup, restart, update
4. Agent sends results: poll data, discovery results, backup content
5. Protobuf used for efficient data serialization (lib/towerops/proto/)

GraphQL API

Absinthe-based GraphQL at /api/graphql:

• Schema: ToweropsWeb.GraphQL.Schema
• Types mirror core entities (devices, sites, alerts, agents, schedules, etc.)
• Complexity analysis enabled (max 500 complexity, max 10 depth)
• Subscriptions via ToweropsWeb.GraphQL.Subscriptions
• API token authentication via GraphQL.Context plug

Health Checks

• GET /health — Kubernetes probe endpoint (no auth)
• DeviceMonitorWorker — ICMP ping monitoring
• JobHealthCheckWorker — Oban queue health
• AgentLatencyEvaluator — Agent responsiveness

---

Generated from source code analysis. Last updated: 2026-03-13.