# TowerOps Architecture > Network infrastructure monitoring platform for ISPs/WISPs built with Phoenix LiveView, Oban, and TimescaleDB. --- ## 1. Project Structure ``` towerops/ ├── assets/ # Frontend: JS, CSS, Tailwind ├── config/ # Runtime & env config (dev/test/prod/runtime) ├── c_src/ # C NIFs (SNMP native interop) ├── docs/ # Documentation ├── e2e/ # End-to-end tests (Playwright) ├── k8s/ # Kubernetes manifests (reconciled by ArgoCD) ├── lib/ │ ├── mix/ # Custom Mix tasks │ ├── snmpkit/ # SNMP toolkit (native extension) │ ├── towerops/ # Core business logic (contexts, schemas, workers) │ └── towerops_web/ # Web layer (LiveView, controllers, channels, GraphQL) ├── mibs/ # SNMP MIB files ├── priv/ │ ├── repo/migrations/ # Ecto migrations │ ├── static/ # Static assets │ └── gettext/ # i18n translations ├── rel/ # Release configuration ├── scripts/ # Operational scripts ├── test/ # Test suite ├── towerops/ # Agent-related files └── vendor/ # Vendored dependencies ``` ### Key `lib/towerops/` Subdirectories | Directory | Purpose | |-----------|---------| | `accounts/` | User, auth tokens, TOTP, recovery codes, sessions, consent | | `admin/` | Superuser audit logs | | `agents/` | Remote polling agent tokens & assignments | | `alerts/` | Alert schema and queries | | `api_tokens/` | API token management (hashed bearer tokens) | | `billing/` | Stripe integration, usage metering | | `capacity/` | Interface throughput & utilization calculations | | `config_changes/` | MikroTik config change event tracking | | `contexts/` | Shared context helpers | | `devices/` | Device schema, backups, firmware, events | | `ecto_types/` | Custom Ecto types (IP address, MAC address, SNMP OID) | | `gaiia/` | Gaiia billing/CRM integration (accounts, subscriptions, impact analysis) | | `geoip/` | GeoIP lookup (blocks, locations) | | `integrations/` | Integration schema (multi-provider) | | `job_monitoring/` | Oban job health events | | `maintenance/` | Maintenance windows | | `mobile_sessions/` | Mobile app sessions & QR login | | `monitoring/` | Check definitions & check results | | `netbox/` | NetBox DCIM sync | | `on_call/` | Schedules, escalation policies, incidents, notifications | | `organizations/` | Org, membership, invitations, policies | | `pagerduty/` | PagerDuty client & notifier | | `preseem/` | Preseem QoE integration (APs, baselines, fleet intelligence) | | `profiles/` | SNMP device profiles (OID mappings) | | `proto/` | Protobuf definitions (agent communication) | | `security/` | Brute-force protection, IP blocks/whitelists | | `settings/` | Global application settings (key-value store) | | `sites/` | Site schema & tree structure | | `snmp/` | SNMP device data (interfaces, sensors, readings, wireless, topology) | | `sonar/` | Sonar billing integration | | `splynx/` | Splynx billing integration | | `topology/` | Network topology discovery (LLDP, CDP, ARP, MAC) | | `visp/` | VISP billing integration | | `weather/` | Weather observations & alerts per site | | `workers/` | All Oban background workers | ### Key `lib/towerops_web/` Subdirectories | Directory | Purpose | |-----------|---------| | `channels/` | Phoenix Channels (agent WebSocket) | | `components/` | Reusable UI components (breadcrumbs, tables, forms) | | `controllers/` | Traditional controllers (auth, webhooks, health, API) | | `graphql/` | Absinthe GraphQL schema, types, resolvers, subscriptions | | `helpers/` | View helpers (time formatting, status display) | | `live/` | All LiveView pages | | `plugs/` | Plug middleware (auth, rate limiting, security headers) | --- ## 2. Data Model All primary keys are UUIDs (`:binary_id`). Multi-tenancy is organization-scoped. ### Core Entities ``` ┌──────────────┐ ┌──────────────────┐ ┌──────────────┐ │ User │────<│ Membership │>────│ Organization │ │ │ │ (role-based) │ │ │ │ email │ │ role: owner| │ │ name, slug │ │ hashed_pass │ │ admin|executive │ │ subscription │ │ totp_secret │ │ |technician| │ │ snmp_config │ │ is_superuser │ │ member|viewer │ │ mikrotik_cfg │ │ timezone │ │ is_default │ │ stripe_* │ │ default_org │ └──────────────────┘ │ alert_routing│ └──────────────┘ └──────┬───────┘ │ ┌────────────────┬──────┴──────┬───────────────┐ │ │ │ │ ┌─────┴─────┐ ┌──────┴──────┐ ┌───┴────┐ ┌──────┴──────┐ │ Site │ │ Device │ │ Alert │ │ Integration │ │ │ │ │ │ │ │ │ │ name │ │ name │ │ type │ │ provider │ │ lat/lng │ │ ip_address │ │ sev │ │ credentials │ │ address │ │ status │ │ msg │ │ sync_status │ │ snmp_cfg │ │ snmp_cfg │ │ device │ │ (encrypted) │ │ parent_id │ │ mikrotik_cfg│ │ check │ └─────────────┘ │ agent_tok │ │ site_id │ └────────┘ └───────────┘ │ agent_assign│ │ check_intv │ └─────────────┘ ``` ### On-Call / Incident Management ``` Schedule ──has_many──> Layer ──has_many──> LayerMember ──belongs_to──> User │ └──has_many──> Override (user, start_at, end_at) EscalationPolicy ──has_many──> EscalationRule ──has_many──> EscalationTarget │ │ │ │ │ position, delay_minutes ├── schedule_id │ └──────────────────────── └── user_id │ Incident ──belongs_to──> Alert │ EscalationPolicy │ Organization └──has_many──> Notification ──belongs_to──> User ``` ### SNMP Data Model ``` Device (towerops) ──1:1──> Snmp.Device │ ├── Snmp.Interface ──> InterfaceStat (time-series) ├── Snmp.Sensor ──> SensorReading (time-series) ├── Snmp.StateSensor ├── Snmp.Processor ──> ProcessorReading ├── Snmp.Storage ──> StorageReading ├── Snmp.WirelessClient ──> WirelessClientReading ├── Snmp.Neighbor (LLDP/CDP) ├── Snmp.ArpEntry ├── Snmp.Vlan └── Snmp.PhysicalEntity ``` ### Billing / CRM Schemas - **Gaiia**: `Account`, `BillingSubscription`, `NetworkSite`, `InventoryItem`, `DeviceSubscriberLink` - **Preseem**: `AccessPoint`, `SubscriberMetric`, `FleetProfile`, `DeviceBaseline`, `Insight`, `SyncLog` - **Billing**: `StripeWebhookEvent` ### Other Schemas | Schema | Table | Purpose | |--------|-------|---------| | `AgentToken` | `agent_tokens` | Remote polling agent credentials | | `AgentAssignment` | `agent_assignments` | Device → agent mapping | | `ApiToken` | `api_tokens` | REST/GraphQL API bearer tokens | | `BrowserSession` | `browser_sessions` | Active login sessions | | `LoginAttempt` | `login_attempts` | Auth attempt audit log | | `UserConsent` | `user_consents` | Privacy/terms consent records | | `PolicyVersion` | `policy_versions` | Privacy policy versions | | `UserTotpDevice` | `user_totp_devices` | Multi-device TOTP | | `UserRecoveryCode` | `user_recovery_codes` | MFA recovery codes | | `MaintenanceWindow` | `maintenance_windows` | Scheduled maintenance | | `ConfigChangeEvent` | `config_change_events` | MikroTik config diffs | | `DeviceFirmwareHistory` | `device_firmware_histories` | Firmware change tracking | | `FirmwareRelease` | `firmware_releases` | Known firmware versions | | `BackupRequest` | `backup_requests` | MikroTik backup queue | | `MikrotikBackup` | `mikrotik_backups` | Stored config backups | | `Check` | `checks` | Monitoring check definitions | | `CheckResult` | `check_results` | Check execution results | | `MonitoringCheck` | `monitoring_checks` | Check configuration | | `DeviceLink` | `device_links` | Discovered topology links | | `DeviceLinkEvidence` | `device_link_evidence` | Link confidence scores | | `DeviceNeighbor` | `device_neighbors` | Raw neighbor data | | `ApplicationSetting` | `application_settings` | Global config KV store | | `AuditLog` | `audit_logs` | Admin action audit trail | | `IpBlock` | `ip_blocks` | Blocked IPs | | `IpWhitelist` | `ip_whitelists` | Whitelisted IPs | | `MobileSession` | `mobile_sessions` | Mobile app sessions | | `QrLoginToken` | `qr_login_tokens` | QR-based mobile auth | | `Weather.Observation` | `weather_observations` | Site weather data | | `Weather.Alert` | `weather_alerts` | Active weather alerts | | `GeoIp.Block` / `Location` | `geoip_*` | IP geolocation | --- ## 3. Context Modules ### `Towerops.Accounts` User management, authentication, TOTP/MFA, password management, email changes, session tracking. ```elixir # User lookup def get_user_by_email(email) def get_user_by_email_and_password(email, password) def get_user(id) def get_user!(id) # Registration def register_user(attrs) def register_user_with_organization(attrs) def change_user_registration(user, attrs \\ %{}) # TOTP / MFA def generate_totp_secret() def generate_totp_uri(user, secret) def generate_totp_qr_code(user, secret) def verify_totp(secret, code) def enable_totp(user, secret) def totp_enabled?(user) def verify_user_totp(user, code) def list_user_totp_devices(user_id) def create_totp_device(user_id, device_name) def verify_user_totp_any_device(user, code) def delete_totp_device(device_id, user_id) def rename_totp_device(device_id, user_id, new_name) def generate_recovery_codes(user_id) def verify_recovery_code(user_id, code) def count_unused_recovery_codes(user_id) def verify_user_mfa(user, code) # Profile & Settings def change_user_profile(user, attrs \\ %{}) def update_user_profile(user, attrs) def sudo_mode?(user, minutes \\ -10) def grant_sudo_mode(user) # Email & Password def change_user_email(user, attrs \\ %{}, opts \\ []) def update_user_email(user, token) def change_user_password(user, attrs \\ %{}, opts \\ []) # Sessions (+ more token functions not listed) ``` ### `Towerops.Organizations` Multi-tenant org management, memberships, invitations, SNMP/agent propagation. ```elixir def list_user_organizations(user_id) def user_has_access?(user_id, organization_id) def get_organization!(id) def get_organization_by_slug!(slug) def create_organization(attrs, user_id, opts \\ []) def update_organization(organization, attrs) def delete_organization(organization) def change_organization(organization, attrs \\ %{}) # Memberships def get_membership(organization_id, user_id) def list_organization_memberships(organization_id) def list_organization_members(organization_id) def remove_member(organization_id, user_id) def update_member_role(organization_id, user_id, new_role) def create_membership(attrs) def get_default_membership(user_id) def set_default_organization(user_id, organization_id) def list_organization_notification_recipients(organization_id) # Invitations def list_pending_invitations(organization_id) def create_invitation(attrs) def get_invitation_by_token(token) def accept_invitation(invitation, user_id) def delete_invitation(invitation) # Config propagation def apply_snmp_config_to_all_equipment(organization_id) def apply_agent_to_all_equipment(organization_id) def clear_all_site_assignments(organization_id) # Billing lookups def get_organization_by_stripe_customer_id(stripe_customer_id) def list_organizations_with_active_subscriptions() ``` ### `Towerops.Devices` Device CRUD, SNMP/MikroTik config resolution (cascading: org → site → device), monitoring. ```elixir def list_site_devices(site_id) def list_organization_devices(organization_id, filters \\ %{}) def count_organization_devices(organization_id) def search_devices(organization_id, query) def get_device_status_counts(organization_id) def list_monitored_devices() def list_snmp_enabled_devices() def list_mikrotik_devices_with_api() def get_device(id) def get_device!(id) def get_device_with_details(id) def get_device_by_ip(ip_address, site_id, exclude_id \\ nil) def get_snmp_config(device) # Resolves cascading config def get_mikrotik_config(device) # Resolves cascading config def get_snmpv3_config(device) # Resolves cascading config def resolve_agent_token_id(device) # Device → site → org → global fallback def propagate_site_snmpv3_change(site_id, attrs) def propagate_organization_mikrotik_change(organization_id, attrs) # ... CRUD, reordering, etc. ``` ### `Towerops.Sites` Site CRUD, hierarchical tree structure, config propagation. ```elixir def list_organization_sites(organization_id) def list_root_sites(organization_id) def list_child_sites(parent_site_id) def build_site_tree(organization_id) def list_sites_with_coordinates() def search_sites(organization_id, query) def get_site!(id) def create_site(attrs) def update_site(site, attrs) def delete_site(site) def apply_snmp_config_to_all_equipment(site_id) def apply_agent_to_all_equipment(site_id) def reorder_site(site_id, new_position) ``` ### `Towerops.Alerts` Alert lifecycle: creation, acknowledgement, resolution, Gaiia impact analysis. ```elixir def create_alert(attrs) def list_devices_alerts(device_id, limit \\ 100) def list_organization_active_alerts(organization_id) def list_organization_alerts(organization_id, filters) def count_active_alerts(organization_id) def get_alert!(id) def get_alert(id) def acknowledge_alert(alert, user_id) def resolve_alert(alert) def resolve_alert_silent(alert) # No PubSub broadcast def acknowledge_alert_silent(alert) # No PubSub broadcast def has_active_alert?(device_id, alert_type) def get_active_alert(device_id, alert_type) def has_active_check_alert?(check_id) def resolve_check_alerts(check_id) def store_gaiia_impact(alert, impact_data) ``` ### `Towerops.Integrations` Generic integration CRUD for all billing/CRM providers. ```elixir def list_integrations(organization_id) def get_integration(organization_id, provider) def create_integration(organization_id, attrs) def update_integration(integration, attrs) def update_sync_status(integration, status, message \\ nil) def delete_integration(integration) def get_billing_summary(organization_id) def list_enabled_integrations(provider) def update_billing_totals(integration, subscriber_count, total_mrr) ``` **Valid providers:** `preseem`, `gaiia`, `pagerduty`, `netbox`, `sonar`, `splynx`, `visp` ### `Towerops.OnCall` On-call schedules, escalation policies, incidents. ```elixir # Schedules def list_schedules(organization_id) def get_schedule!(id) / get_schedule(id) def create_schedule(attrs) / update_schedule(schedule, attrs) / delete_schedule(schedule) def create_layer(attrs) / update_layer(layer, attrs) / delete_layer(layer) def add_layer_member(attrs) / remove_layer_member(member) def create_override(attrs) / delete_override(override) def who_is_on_call(schedule_id, datetime \\ DateTime.utc_now()) # Escalation Policies def list_escalation_policies(organization_id) def get_escalation_policy!(id) / get_escalation_policy(id) def create_escalation_policy(attrs) / update_escalation_policy(policy, attrs) / delete_escalation_policy(policy) def create_escalation_rule(attrs) / update_escalation_rule(rule, attrs) / delete_escalation_rule(rule) def create_escalation_target(attrs) / delete_escalation_target(target) ``` ### `Towerops.Billing` Stripe billing integration: checkout, portal, usage metering, subscription management. ```elixir def default_free_devices() def default_price_per_device() def create_checkout_session(organization, opts \\ []) def create_portal_session(organization, return_url) def billable_device_count(organization) def effective_free_device_count(org) def effective_price_per_device(org) def estimated_monthly_cost(organization) def sync_usage_to_stripe(organization) def update_subscription_from_stripe(organization, stripe_subscription) def update_payment_method_status(organization, status) def migrate_all_subscriptions_to_price(new_price_id) ``` ### `Towerops.Agents` Remote polling agent lifecycle: tokens, heartbeat, assignments, mass updates, debug mode. ```elixir def create_agent_token(organization_id, name) def create_cloud_poller(name) def list_organization_agent_tokens(organization_id) def list_cloud_pollers() def verify_agent_token(token) def update_agent_token_heartbeat(agent_token_id, ip_address, metadata \\ nil) def update_agent_token(agent_token, attrs) def restart_agent(agent_token_id) def update_agent(agent_token_id, url, checksum) def toggle_agent_debug(agent_token, enabled, user) def revoke_agent_token(id) def delete_agent_token(id) def assign_device_to_agent(agent_token_id, device_id) def unassign_device(device_id) def list_agent_devices(agent_token_id) def list_agent_polling_targets(agent_token_id) def get_effective_agent_token(device) def broadcast_mass_update() ``` ### `Towerops.Monitoring` Check definitions, execution, result tracking, state machine (soft/hard states). ```elixir def list_checks(organization_id, opts \\ []) def get_check(id) / get_check!(id) def create_check(attrs) / update_check(check, attrs) / delete_check(check) def schedule_check(check) def create_check_result(attrs) def get_latest_check_result(check_id) def get_check_results(check_id, opts \\ []) def get_check_graph_data(check_id, from_time, to_time) def update_check_state(check, status, output) # Soft/hard state machine def stop_device_checks(device_id) def disable_device_checks(device_id) def get_device_health_summary(device_ids) def get_device_latest_response_times(device_ids) ``` ### `Towerops.Snmp` SNMP device discovery, interface/sensor management, readings. ```elixir def test_connection(ip, community, version, port \\ 161) def discover_device(device) def discover_all_for_org(org_id) def create_checks_from_discovery(device, snmp_device) def get_device(device_id) / get_device_with_associations(device_id) def list_interfaces(device_id) / list_monitored_interfaces(device_id) def list_sensors(device_id) / list_sensors_by_type(device_id) def get_sensor_readings(sensor_id, opts \\ []) def get_interface_stats(interface_id, opts \\ []) def set_manual_capacity(interface_id, capacity_bps) # ... many more reading/stat functions ``` ### `Towerops.Gaiia` Gaiia CRM/billing data: accounts, subscriptions, network sites, inventory, impact analysis. ```elixir def list_accounts(organization_id) def upsert_account(organization_id, attrs) def count_active_subscribers(organization_id) def sum_active_mrr(organization_id) def list_network_sites(organization_id) def upsert_network_site(organization_id, attrs) def list_inventory_items(organization_id) def list_billing_subscriptions(organization_id) def get_device_subscriber_counts(device_ids) def suggest_site_matches(organization_id, network_site) def suggest_device_matches(organization_id, inventory_item) ``` Sub-modules: `Gaiia.Client`, `Gaiia.Webhooks`, `Gaiia.ImpactAnalysis`, `Gaiia.Reconciliation`, `Gaiia.SubscriberMatching`, `Gaiia.Actions` ### `Towerops.Preseem` Preseem QoE integration: access points, subscriber metrics, fleet intelligence. ```elixir def get_access_point_for_device(device_id) def list_access_points(organization_id) def list_unmatched_access_points(organization_id) def list_matched_access_points(organization_id) def list_subscriber_metrics(access_point_id, opts \\ []) def link_access_point(access_point_id, device_id) def unlink_access_point(access_point_id) def get_site_qoe_summary(site_id) ``` Sub-modules: `Preseem.Client`, `Preseem.Sync`, `Preseem.Baseline`, `Preseem.DeviceMatcher`, `Preseem.FleetIntelligence`, `Preseem.Insights` ### Other Context Modules | Module | Purpose | |--------|---------| | `Towerops.Maintenance` | Maintenance window CRUD, device/site in-maintenance checks | | `Towerops.Topology` | Network topology discovery (LLDP/CDP/ARP/MAC), link inference | | `Towerops.Dashboard` | Dashboard summary, health scores, impact analysis | | `Towerops.Search` | Cross-entity full-text search (devices, sites, accounts, alerts) | | `Towerops.ActivityFeed` | Aggregated activity feed (config changes, alerts, syncs, events) | | `Towerops.ApiTokens` | API token CRUD, hashed token verification | | `Towerops.Capacity` | Interface throughput & utilization calculations | | `Towerops.ConfigChanges` | MikroTik config change diff tracking | | `Towerops.MobileSessions` | Mobile app sessions, QR login flow | | `Towerops.Weather` | Weather observations & alerts per site | | `Towerops.Admin` | Superuser admin: user/org listing, audit logs, billing overrides | | `Towerops.Profiles` | SNMP device profile matching & OID management | | `Towerops.Settings` | Global application settings (key-value store) | | `Towerops.Geocoding` | Address → lat/lng geocoding | | `Towerops.GeoIp` | IP → location lookup | | `Towerops.Http` | Shared HTTP client helpers | | `Towerops.Trace` | Subscriber trace functionality | | `Towerops.RateLimit` | Rate limiting logic | | `Towerops.Numeric` | Numeric formatting utilities | --- ## 4. LiveView Pages ### Main Application (requires auth + org) | Route | LiveView | Purpose | |-------|----------|---------| | `/dashboard` | `DashboardLive` | Org dashboard with health score, alert summary, site overview | | `/alerts` | `AlertLive.Index` | Alert list with filtering (active/ack/resolved), real-time updates | | `/agents` | `AgentLive.Index` | Agent list with health status | | `/agents/:id` | `AgentLive.Show` | Agent detail: assigned devices, heartbeat, metadata | | `/agents/:id/edit` | `AgentLive.Edit` | Edit agent settings | | `/devices` | `DeviceLive.Index` | Device list with status, filtering, real-time updates | | `/devices/new` | `DeviceLive.Form` | Create device form (with SNMP credential test) | | `/devices/:id` | `DeviceLive.Show` | Device detail: status, interfaces, sensors, alerts, backups | | `/devices/:id/edit` | `DeviceLive.Form` | Edit device form | | `/devices/:id/graph/:sensor_type` | `GraphLive.Show` | Time-series graphs for device metrics | | `/devices/:device_id/backups/compare` | `MikrotikBackupLive.Compare` | Config backup diff viewer | | `/devices/:device_id/config-timeline` | `ConfigTimelineLive` | Config change timeline | | `/sites` | `SiteLive.Index` | Site list with device counts | | `/sites/new` | `SiteLive.Form` | Create site | | `/sites/:id` | `SiteLive.Show` | Site detail: devices, capacity, weather | | `/sites/:id/edit` | `SiteLive.Form` | Edit site | | `/insights` | `InsightsLive.Index` | AI-generated operational insights | | `/trace` | `TraceLive.Index` | Subscriber trace tool | | `/activity` | `ActivityFeedLive` | Activity feed (config changes, alerts, syncs) | | `/schedules` | `ScheduleLive.Index` | On-call schedule list | | `/schedules/new` | `ScheduleLive.Form` | Create schedule | | `/schedules/:id` | `ScheduleLive.Show` | Schedule detail with calendar view | | `/schedules/:id/edit` | `ScheduleLive.Form` | Edit schedule | | `/schedules/escalation-policies` | `EscalationPolicyLive.Index` | Escalation policy list | | `/schedules/escalation-policies/new` | `EscalationPolicyLive.Form` | Create escalation policy | | `/schedules/escalation-policies/:id` | `EscalationPolicyLive.Show` | Policy detail with rules | | `/schedules/escalation-policies/:id/edit` | `EscalationPolicyLive.Form` | Edit policy | | `/maintenance` | `MaintenanceLive.Index` | Maintenance window list | | `/maintenance/new` | `MaintenanceLive.Form` | Create maintenance window | | `/maintenance/:id` | `MaintenanceLive.Show` | Window detail | | `/maintenance/:id/edit` | `MaintenanceLive.Form` | Edit window | | `/changelog` | `ChangelogLive` | Product changelog | | `/network-map` | `NetworkMapLive` | Network topology visualization | | `/sites-map` | `MapLive.Index` | Geographic site map | ### Organization Settings (requires auth + specific org slug) | Route | LiveView | Purpose | |-------|----------|---------| | `/orgs/:org_slug/settings` | `Org.SettingsLive` | Org settings, SNMP defaults, billing | | `/orgs/:org_slug/settings/integrations` | `Org.IntegrationsLive` | Integration management | | `/orgs/:org_slug/settings/integrations/preseem/devices` | `Org.PreseemDevicesLive` | Preseem AP ↔ device matching | | `/orgs/:org_slug/settings/integrations/preseem/insights` | `Org.PreseemInsightsLive` | Preseem fleet insights | | `/orgs/:org_slug/settings/integrations/gaiia/mapping` | `Org.GaiiaMappingLive` | Gaiia site/device mapping | | `/orgs/:org_slug/settings/integrations/gaiia/reconciliation` | `Org.GaiiaReconciliationLive` | Gaiia data reconciliation | ### Account / User Pages | Route | LiveView | Purpose | |-------|----------|---------| | `/orgs` | `OrgLive.Index` | Organization picker | | `/orgs/new` | `OrgLive.New` | Create organization | | `/users/settings` | `UserSettingsLive` | Profile, email, password, API tokens, sessions, TOTP | | `/users/my-data` | `AccountLive.MyData` | GDPR data export | | `/account/activity` | `AccountLive.Activity` | User login history | | `/account/totp-enrollment` | `AccountLive.TotpEnrollment` | TOTP setup wizard | | `/mobile/qr-login` | `MobileQRLive` | QR code for mobile app login | | `/users/register` | `UserRegistrationLive` | Registration form | | `/users/reset-password/:token` | `UserResetPasswordLive` | Password reset | | `/help` | `HelpLive.Index` | Help / documentation page | ### Admin Pages (superuser only) | Route | LiveView | Purpose | |-------|----------|---------| | `/admin/` | `Admin.DashboardLive` | Admin overview | | `/admin/users` | `Admin.UserLive.Index` | All users management | | `/admin/organizations` | `Admin.OrgLive.Index` | All orgs management | | `/admin/audit` | `Admin.AuditLive.Index` | Audit log viewer | | `/admin/monitoring` | `Admin.MonitoringLive` | System monitoring | | `/admin/agents` | `Admin.AgentLive.Index` | All agents health | | `/admin/security` | `Admin.SecurityLive.Index` | IP blocks/whitelists | --- ## 5. PubSub Topics ### Alert Topics | Topic | Events | Subscribers | |-------|--------|-------------| | `alerts:org:{org_id}:new` | New alert created | `AlertLive.Index`, `DashboardLive`, `ActivityFeedLive` | | `alerts:org:{org_id}:resolved` | Alert resolved | `AlertLive.Index`, `DashboardLive`, `ActivityFeedLive` | | `organization:{org_id}:alerts` | Alert status updates | `UserAuth.subscribe_to_status_updates` (nav badge) | ### Device Topics | Topic | Events | Subscribers | |-------|--------|-------------| | `device:{device_id}` | Device status changes, poll results, sensor data | `DeviceLive.Show`, `GraphLive.Show` | | `device:events` | All device events (cross-device) | `Devices.EventLogger` | | `device_events:org:{org_id}` | Org-scoped device events | `ActivityFeedLive` | | `devices:org:{org_id}` | Device created/updated/deleted | `DeviceLive.Index`, `ActivityFeedLive` | | `wireless_clients:device:{device_id}` | Wireless client updates | `DeviceLive.Show` | ### Agent Topics | Topic | Events | Subscribers | |-------|--------|-------------| | `agents:health` | Agent heartbeat, stale detection | `AgentLive.Index`, `DeviceLive.Show`, `Admin.AgentLive` | | `admin:agents` | Admin-level agent events | `Admin.AgentLive.Index` | | `agent:{agent_id}:assignments` | Assignment changes | `AgentChannel` | | `agent:{agent_id}:discovery` | Discovery commands | `AgentChannel` | | `agent:{agent_id}:backup` | Backup commands | `AgentChannel` | | `agent:{agent_id}:credential_test` | Credential test requests | `AgentChannel` | | `agent:{agent_id}:live_poll` | Live poll requests | `AgentChannel` | | `agent:{agent_id}:lifecycle` | Restart/update commands | `AgentChannel` | | `agent:{agent_id}:latency_probe` | Latency probe requests | `AgentChannel` | ### Other Topics | Topic | Events | Subscribers | |-------|--------|-------------| | `config:org:{org_id}` | Config change events | `ActivityFeedLive` | | `sync:org:{org_id}` | Integration sync events | `ActivityFeedLive` | | `topology:{org_id}` | Topology link changes | `NetworkMapLive` | | `credential_test:{test_id}` | SNMP credential test results | `DeviceLive.Form` | | `security:blocks` | IP block changes | `Admin.SecurityLive` | | `security:whitelist` | IP whitelist changes | `Admin.SecurityLive` | | `job_monitoring` | Oban job health events | `Admin.MonitoringLive` | --- ## 6. Workers (Oban) ### Queue: `pollers` | Worker | Purpose | |--------|---------| | `DevicePollerWorker` | SNMP polling: interfaces, sensors, storage, processors, wireless, ARP, topology. The main data collection worker. | ### Queue: `monitors` | Worker | Purpose | |--------|---------| | `DeviceMonitorWorker` | Device up/down monitoring via ICMP ping, triggers alerts on status change | ### Queue: `discovery` | Worker | Purpose | |--------|---------| | `DiscoveryWorker` | SNMP device discovery: sysinfo, interfaces, sensors, profiles | ### Queue: `checks` | Worker | Purpose | |--------|---------| | `CheckWorker` | Schedules and dispatches monitoring check execution | ### Queue: `check_executors` | Worker | Purpose | |--------|---------| | `CheckExecutorWorker` | Executes individual monitoring checks (HTTP, ping, etc.) | ### Queue: `notifications` | Worker | Purpose | |--------|---------| | `AlertNotificationWorker` | Sends alert notifications (email, PagerDuty, on-call) | | `WelcomeEmailWorker` | Sends welcome email after registration | | `EscalationCheckWorker` | Checks for escalation timeouts and advances incidents | ### Queue: `weather` | Worker | Purpose | |--------|---------| | `WeatherSyncWorker` | Fetches weather data for sites with coordinates | ### Queue: `maintenance` | Worker | Purpose | |--------|---------| | `BillingSyncWorker` | Syncs device count usage to Stripe | | `GaiiaSyncWorker` | Syncs data from Gaiia billing/CRM | | `GaiiaInsightWorker` | Generates Gaiia-based insights | | `PreseemSyncWorker` | Syncs data from Preseem | | `PreseemBaselineWorker` | Calculates Preseem performance baselines | | `NetboxSyncWorker` | Syncs data from NetBox DCIM | | `SonarSyncWorker` | Syncs data from Sonar billing | | `SplynxSyncWorker` | Syncs data from Splynx billing | | `VispSyncWorker` | Syncs data from VISP billing | | `MikrotikBackupWorker` | Fetches MikroTik config backups | | `BackupSummaryWorker` | Generates backup comparison summaries | | `BackupTimeoutWorker` | Cleans up timed-out backup requests | | `FirmwareVersionFetcherWorker` | Fetches latest firmware versions from vendors | | `StaleAgentWorker` | Detects agents that stopped heartbeating | | `AgentLatencyEvaluator` | Evaluates agent-to-device latency data | | `CloudLatencyProbeWorker` | Probes cloud poller latency | | `SessionCleanupWorker` | Cleans up expired browser/mobile sessions | | `LoginHistoryCleanupWorker` | Purges old login attempt records | | `ExpiredBanCleanupWorker` | Removes expired IP bans | | `StaleViolationCleanupWorker` | Cleans up stale rate limit violations | | `CloudflareBanWorker` | Syncs IP bans to Cloudflare WAF | | `JobHealthCheckWorker` | Monitors Oban job queue health | | `DeviceHealthInsightWorker` | Generates device health insights | | `CapacityInsightWorker` | Generates capacity utilization insights | | `SystemInsightWorker` | Generates system-level insights | | `WirelessInsightWorker` | Generates wireless performance insights | ### Additional Worker | Worker | Queue | Purpose | |--------|-------|---------| | `Snmp.NeighborCleanupWorker` | (in snmp module) | Cleans up stale SNMP neighbor records | --- ## 7. Integrations Architecture ### Overview Integrations are stored as `Integration` records with encrypted credentials (`Cloak.Ecto`). Each provider has: - A **client module** (`Client`) for API communication - A **sync module** (`Sync`) for data synchronization - An **Oban worker** for periodic sync ### Billing/CRM Providers | Provider | Modules | Data Synced | |----------|---------|-------------| | **Gaiia** | `Gaiia.Client`, `Gaiia.Sync`, `Gaiia.Webhooks`, `Gaiia.ImpactAnalysis`, `Gaiia.Reconciliation`, `Gaiia.SubscriberMatching`, `Gaiia.Actions` | Accounts, billing subscriptions, network sites, inventory items, device-subscriber links | | **Preseem** | `Preseem.Client`, `Preseem.Sync`, `Preseem.Baseline`, `Preseem.DeviceMatcher`, `Preseem.FleetIntelligence`, `Preseem.Insights` | Access points, subscriber metrics, fleet profiles, device baselines | | **Sonar** | `Sonar.Client`, `Sonar.Sync` | Subscriber/billing data | | **Splynx** | `Splynx.Client`, `Splynx.Sync` | Subscriber/billing data | | **VISP** | `Visp.Client`, `Visp.Sync` | Subscriber/billing data | ### Infrastructure Providers | Provider | Modules | Purpose | |----------|---------|---------| | **NetBox** | `Netbox.Client`, `Netbox.Sync` | DCIM inventory sync (devices, sites) | | **PagerDuty** | `PagerDuty.Client`, `PagerDuty.Notifier` | Alert routing & incident escalation | ### Stripe Billing Stripe integration is handled separately from the integration framework: - `Towerops.Billing` — checkout sessions, portal sessions, usage metering - `BillingSyncWorker` — periodic usage sync - `StripeWebhookController` — receives Stripe webhook events - `StripeWebhookEvent` schema — idempotent webhook processing - Usage-based billing: free device tier, then per-device pricing - Organization fields: `stripe_customer_id`, `stripe_subscription_id`, `subscription_status` ### Gaiia Specifics Gaiia has the deepest integration: - **Webhook receiver** at `/api/v1/webhooks/gaiia/:organization_id` - **Impact analysis** — when a device goes down, calculates affected subscribers & MRR - **Reconciliation** — UI for matching Gaiia network sites to TowerOps sites - **Device-subscriber links** — maps which subscribers are served by which device ### Alert Routing Organizations configure `alert_routing`: - `"builtin"` — uses on-call schedules + escalation policies - `"pagerduty"` — routes to PagerDuty - `"both"` — routes to both --- ## 8. Auth Flow ### Authentication Stack ``` Request → Browser Pipeline → fetch_session → fetch_current_scope_for_user │ ├─ Check session token ├─ Check remember_me cookie ├─ Build Scope (user + org + tz) └─ Handle impersonation ``` ### `Towerops.Accounts.Scope` The `Scope` struct flows through the entire request lifecycle: ```elixir %Scope{ user: %User{}, # Current user (or impersonated user) superuser: %User{} | nil, # Actual superuser if impersonating impersonating?: boolean(), organization: %Organization{} | nil, timezone: "America/Chicago", time_format: "24h" } ``` ### Session Lifecycle 1. **Login** — `UserSessionController.create` validates email/password 2. **TOTP verification** — If TOTP enabled, redirect to `/users/log-in/totp` 3. **Session creation** — Token stored in session + optional remember_me cookie 4. **Organization loading** — Default org set from membership, or from URL slug 5. **Session renewal** — Tokens reissued after configurable interval 6. **Sudo mode** — Sensitive operations require recent password re-entry (`last_sudo_at`) ### LiveView `on_mount` Hooks ```elixir :load_cookie_consent # Load GDPR consent state :require_authenticated_user # Redirect to login if no session :require_totp_enrollment # Redirect to TOTP setup if required :load_current_organization # Load org from URL slug param :load_default_organization # Load user's default org :subscribe_to_status_updates # Subscribe to org alert PubSub :require_superuser # Admin-only pages :require_sudo_mode # Require recent password entry :handle_policy_consent # Check privacy/terms acceptance :redirect_if_user_is_authenticated # Login page redirect :require_organization_owner # Owner-only operations ``` ### API Authentication - **REST API** (`/api/v1/*`) — Bearer token via `ToweropsWeb.Plugs.ApiAuth`, verified against hashed `ApiToken` records - **GraphQL** (`/api/graphql`) — Same bearer token auth + `GraphQL.Context` plug - **Mobile API** (`/api/v1/mobile/*`) — Session token via `ToweropsWeb.Plugs.MobileAuth` - **Webhooks** (`/api/v1/webhooks/*`) — Shared secret via `ToweropsWeb.Plugs.WebhookAuth` - **Agent WebSocket** — Token-based auth in `AgentSocket`/`AgentChannel` ### Organization Switching - `POST /orgs/switch` — changes the current organization in session - User must have a membership in the target organization - Default org stored on `Membership.is_default` flag - URL-based: `/orgs/:org_slug/settings/*` uses slug from URL ### Security Features - **Rate limiting** — Per-route rate limits (auth, API, admin) via `RateLimit` plug - **Brute-force protection** — `Security.BruteForce` with IP blocking + Cloudflare WAF sync - **TOTP/MFA** — Required enrollment, multi-device support, recovery codes - **Sudo mode** — Time-limited elevated access for sensitive operations - **IP whitelisting/blocking** — `IpBlock`, `IpWhitelist` schemas - **Session tracking** — `BrowserSession` records with device/location info - **Impersonation** — Superuser can impersonate users, tracked in `Scope` --- ## 9. Key Patterns ### Cascading Configuration SNMP and MikroTik credentials cascade: **Organization → Site → Device**. Each level can override or inherit: ```elixir # In Devices context def resolve_agent_token_id(device) do get_direct_agent(device) # Device-level assignment || get_site_agent(device) # Site default agent || get_org_default_agent(device) # Organization default || get_global_default_agent() # Global cloud poller fallback end ``` ### AccessControl Helpers `ToweropsWeb.Live.Helpers.AccessControl` provides organization-scoped access verification: ```elixir verify_device_access(device_id, organization_id) # → {:ok, device} | {:error, :not_found | :unauthorized} verify_site_access(site_id, organization_id) verify_alert_access(alert_id, organization_id) ``` ### Membership Roles Ordered by privilege: `owner > admin > executive > technician > member > viewer` - **owner** — Full access, can delete organization - **admin** — Full access except deleting org - **executive** — Read-only with financial visibility (MRR, revenue) - **technician** — Field ops: devices, sites, alerts, no financials - **viewer** — Read-only, no financials ### Breadcrumbs `ToweropsWeb.Components.Breadcrumbs` provides a DaisyUI-based breadcrumb component: ```heex <.breadcrumb items={[ %{label: "Sites", path: ~p"/sites"}, %{label: @site.name, path: ~p"/sites/#{@site.id}"}, %{label: "Edit"} ]} /> ``` ### PubSub Pattern Contexts broadcast changes; LiveViews subscribe on mount: ```elixir # In context (e.g., Alerts) Phoenix.PubSub.broadcast(Towerops.PubSub, "alerts:org:#{org_id}:new", {:new_alert, alert}) # In LiveView mount Phoenix.PubSub.subscribe(Towerops.PubSub, "alerts:org:#{organization.id}:new") # In LiveView handle_info def handle_info({:new_alert, alert}, socket), do: ... ``` ### Encrypted Fields Sensitive data (SNMP passwords, API credentials, integration secrets) uses `Cloak.Ecto` encryption: ```elixir field :snmpv3_auth_password, Towerops.Encrypted.Binary field :credentials, Towerops.Encrypted.Map ``` ### Multi-Tenancy All queries are scoped by `organization_id`. The org context flows through: 1. `Scope` struct in conn/socket assigns 2. URL slug (`/orgs/:org_slug/...`) 3. Default organization from user membership ### Form Pattern LiveViews use Phoenix changesets with `to_form/1`: ```elixir # In mount/handle_params changeset = Context.change_entity(entity, %{}) socket = assign(socket, form: to_form(changeset)) # In handle_event("save", ...) case Context.update_entity(entity, params) do {:ok, entity} -> {:noreply, push_navigate(socket, to: ...)} {:error, changeset} -> {:noreply, assign(socket, form: to_form(changeset))} end ``` ### Flash Messages Standard Phoenix flash with custom styling: ```elixir socket |> put_flash(:info, "Device created successfully") |> push_navigate(to: ...) socket |> put_flash(:error, "Failed to save device") ``` ### Agent Communication Agents connect via WebSocket (`AgentChannel`): 1. Agent authenticates with token via `AgentSocket` 2. Channel subscribes to agent-specific PubSub topics 3. Server pushes commands: poll, discover, backup, restart, update 4. Agent sends results: poll data, discovery results, backup content 5. Protobuf used for efficient data serialization (`lib/towerops/proto/`) ### GraphQL API Absinthe-based GraphQL at `/api/graphql`: - Schema: `ToweropsWeb.GraphQL.Schema` - Types mirror core entities (devices, sites, alerts, agents, schedules, etc.) - Complexity analysis enabled (max 500 complexity, max 10 depth) - Subscriptions via `ToweropsWeb.GraphQL.Subscriptions` - API token authentication via `GraphQL.Context` plug ### Health Checks - `GET /health` — Kubernetes probe endpoint (no auth) - `DeviceMonitorWorker` — ICMP ping monitoring - `JobHealthCheckWorker` — Oban queue health - `AgentLatencyEvaluator` — Agent responsiveness --- *Generated from source code analysis. Last updated: 2026-03-13.*