graham/towerops
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
towerops/IMPLEMENTATION_PLAN.md
Graham McIntire 93e0b869be
more features
2026-01-02 14:06:44 -06:00

21 KiB
Raw Blame History

TowerOps - Network Monitoring & Alerting Platform

Implementation Plan

Overview

Multi-tenant network monitoring and alerting application for tracking network equipment health via ping monitoring.

Core Requirements

  1. User accounts (email as username)
  2. Organizations (multi-tenant)
  3. Organization membership with permission levels
  4. Site hierarchy
  5. Equipment management (IP-based)
  6. Automated ping monitoring (5-minute intervals)
  7. Alerting system

Data Model Design

Users

  • id (binary_id, PK)
  • email (string, unique, username)
  • hashed_password (string)
  • confirmed_at (utc_datetime, nullable)
  • inserted_at / updated_at (utc_datetime)

Organizations

  • id (binary_id, PK)
  • name (string)
  • slug (string, unique) - for URLs
  • inserted_at / updated_at (utc_datetime)

OrganizationMemberships

  • id (binary_id, PK)
  • organization_id (binary_id, FK -> organizations)
  • user_id (binary_id, FK -> users)
  • role (enum: owner, admin, member, viewer)
  • inserted_at / updated_at (utc_datetime)
  • Unique constraint on (organization_id, user_id)

Permission Levels:

  • owner - Full control, can delete org, manage all settings
  • admin - Can manage users, sites, equipment, view all
  • member - Can add/edit equipment, view sites
  • viewer - Read-only access

Sites

  • id (binary_id, PK)
  • organization_id (binary_id, FK -> organizations)
  • parent_site_id (binary_id, FK -> sites, nullable) - for hierarchy
  • name (string)
  • description (text, nullable)
  • location (string, nullable)
  • inserted_at / updated_at (utc_datetime)

Equipment

  • id (binary_id, PK)
  • site_id (binary_id, FK -> sites)
  • name (string)
  • ip_address (string)
  • description (text, nullable)
  • status (enum: up, down, unknown)
  • last_checked_at (utc_datetime, nullable)
  • last_status_change_at (utc_datetime, nullable)
  • monitoring_enabled (boolean, default: true)
  • check_interval_seconds (integer, default: 300) - 5 minutes
  • inserted_at / updated_at (utc_datetime)

MonitoringChecks (historical log)

  • id (binary_id, PK)
  • equipment_id (binary_id, FK -> equipment)
  • status (enum: success, failure)
  • response_time_ms (integer, nullable)
  • checked_at (utc_datetime)
  • Index on (equipment_id, checked_at)

Alerts

  • id (binary_id, PK)
  • equipment_id (binary_id, FK -> equipment)
  • alert_type (enum: equipment_down, equipment_up)
  • triggered_at (utc_datetime)
  • acknowledged_at (utc_datetime, nullable)
  • acknowledged_by_id (binary_id, FK -> users, nullable)
  • resolved_at (utc_datetime, nullable)
  • email_sent_at (utc_datetime, nullable)
  • inserted_at / updated_at (utc_datetime)

OrganizationInvitations

  • id (binary_id, PK)
  • organization_id (binary_id, FK -> organizations)
  • email (string)
  • role (enum: admin, member, viewer) - cannot invite as owner
  • token (string, unique) - secure random token for invite link
  • invited_by_id (binary_id, FK -> users)
  • accepted_at (utc_datetime, nullable)
  • accepted_by_id (binary_id, FK -> users, nullable)
  • expires_at (utc_datetime) - invites expire after 7 days
  • inserted_at / updated_at (utc_datetime)

Architecture Components

1. Authentication & Authorization

Use Phoenix.LiveView built-in patterns:

  • Email/password authentication
  • Session-based auth with LiveView
  • Password reset via email

Authorization Strategy:

  • Context-based permissions (organization-scoped)
  • Plugs for organization membership verification
  • LiveView mount hooks for permission checks
  • Helper functions: can?(user, :action, resource)

2. Multi-Tenancy

Organization Scoping:

  • All queries scoped by current_organization
  • LiveView assigns: @current_user, @current_organization, @current_membership
  • Router organization switcher for users in multiple orgs
  • URL structure: /orgs/:org_slug/sites, /orgs/:org_slug/equipment

3. Monitoring System

Background Job Architecture:

  • Use GenServer or DynamicSupervisor for monitoring workers
  • One worker per equipment (or batched by site)
  • Quantum or similar for scheduling (or custom OTP solution)
  • Use :gen_icmp or System.cmd("ping") for ping checks

Monitoring Flow:

  1. Worker wakes up every N seconds (configurable per equipment)
  2. Pings equipment IP address
  3. Records result in monitoring_checks table
  4. Updates equipment status if changed
  5. Creates alert if status transitions (up->down or down->up)
  6. Broadcasts status change via PubSub for real-time UI updates

4. Real-Time Updates

Phoenix PubSub Topics:

  • organization:#{org_id}:equipment:#{equipment_id} - Equipment status
  • organization:#{org_id}:alerts - New alerts
  • organization:#{org_id}:sites - Site changes

LiveView Integration:

  • Subscribe to relevant topics on mount
  • Handle PubSub messages to update assigns
  • Stream-based updates for lists

5. UI Structure

LiveView Pages:

  • /login, /register - Authentication
  • /orgs - Organization list/switcher
  • /orgs/:slug/dashboard - Overview, active alerts, recent changes
  • /orgs/:slug/sites - Site hierarchy tree view
  • /orgs/:slug/sites/:id - Site detail with equipment list
  • /orgs/:slug/equipment - All equipment list
  • /orgs/:slug/equipment/:id - Equipment detail with check history
  • /orgs/:slug/alerts - Alert history
  • /orgs/:slug/settings - Org settings, members

Components:

  • Organization switcher (header)
  • Site tree navigator
  • Equipment status badge
  • Alert list/feed
  • Permission-based action buttons

Implementation Stages

Stage 1: Foundation & Authentication

Goal: User authentication and basic org structure Success Criteria:

  • Users can register/login with email
  • Users can create organizations
  • Users can switch between organizations
  • Basic navigation structure
  • Tests passing

Detailed Tasks:

1.1 User Authentication

  • Run mix phx.gen.auth Accounts User users to scaffold auth system
  • Review and customize generated code (email as username)
  • Update user registration to create first organization
  • Add tests for auth flows

Files Created:

  • lib/towerops/accounts.ex - User context
  • lib/towerops/accounts/user.ex - User schema
  • lib/towerops/accounts/user_token.ex - Session tokens
  • lib/towerops_web/user_auth.ex - Auth plugs
  • lib/towerops_web/controllers/user_session_controller.ex
  • lib/towerops_web/controllers/user_registration_controller.ex
  • lib/towerops_web/controllers/user_reset_password_controller.ex
  • lib/towerops_web/controllers/user_settings_controller.ex
  • Migrations for users and user_tokens tables

1.2 Organizations & Memberships

  • Create migration: mix ecto.gen.migration create_organizations
  • Create migration: mix ecto.gen.migration create_organization_memberships
  • Create migration: mix ecto.gen.migration create_organization_invitations
  • Create lib/towerops/organizations.ex context
  • Create lib/towerops/organizations/organization.ex schema
  • Create lib/towerops/organizations/membership.ex schema
  • Create lib/towerops/organizations/invitation.ex schema
  • Add slug generation for organizations (use Ecto changeset)
  • Add tests for organizations context

Migration Details:

# organizations table
create table(:organizations, primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :name, :string, null: false
  add :slug, :string, null: false
  timestamps(type: :utc_datetime)
end
create unique_index(:organizations, [:slug])

# organization_memberships table
create table(:organization_memberships, primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :organization_id, references(:organizations, type: :binary_id, on_delete: :delete_all)
  add :user_id, references(:users, type: :binary_id, on_delete: :delete_all)
  add :role, :string, null: false
  timestamps(type: :utc_datetime)
end
create unique_index(:organization_memberships, [:organization_id, :user_id])
create index(:organization_memberships, [:user_id])

1.3 Multi-Org Navigation

  • Create organization switcher LiveView component
  • Add current_organization plug to router
  • Create /orgs LiveView (list user's organizations)
  • Create /orgs/new LiveView (create organization)
  • Update router with organization-scoped routes
  • Add breadcrumb navigation component
  • Add tests for organization switching

Router Structure:

scope "/", ToweropsWeb do
  pipe_through [:browser, :require_authenticated_user]

  live "/orgs", OrgLive.Index
  live "/orgs/new", OrgLive.New
end

scope "/orgs/:org_slug", ToweropsWeb do
  pipe_through [:browser, :require_authenticated_user, :load_current_organization]

  live "/", DashboardLive
  # Future: sites, equipment, etc.
end

1.4 Authorization System

  • Create lib/towerops/organizations/policy.ex for permission checks
  • Add can?/3 helper function
  • Create :load_current_organization plug
  • Add permission checks to LiveView mount callbacks
  • Add tests for authorization

Permission Matrix:

Action Owner Admin Member Viewer
View org ✓ ✓ ✓ ✓
Edit org settings ✓ ✓ ✗ ✗
Delete org ✓ ✗ ✗ ✗
Manage members ✓ ✓ ✗ ✗
Add/edit equipment ✓ ✓ ✓ ✗
View equipment ✓ ✓ ✓ ✓

1.5 Basic UI & Layouts

  • Update layouts.ex to include org switcher in header
  • Create organization badge component
  • Style navigation with Tailwind
  • Add flash message styling
  • Ensure responsive design

Stage 2: Sites & Equipment Management

Goal: CRUD for sites and equipment Success Criteria:

  • Users can create/edit/delete sites
  • Sites can have parent sites (hierarchy)
  • Users can add equipment to sites
  • Equipment has IP address and basic info

Tasks:

  1. Generate sites schema and LiveViews
  2. Build site hierarchy tree component
  3. Generate equipment schema and LiveViews
  4. Add IP address validation
  5. Permission checks on all actions

Stage 3: Monitoring System ✓ COMPLETE

Goal: Automated ping monitoring with TimescaleDB Success Criteria:

  • ✓ Equipment is pinged at configurable intervals
  • ✓ Status updates in real-time via PubSub
  • ✓ Monitoring checks are logged
  • ✓ TimescaleDB hypertable for efficient time-series storage
  • ✓ Automatic data retention and compression
  • ✓ Continuous aggregates for dashboard metrics

Tasks:

  1. ✓ Design monitoring worker architecture (GenServer + DynamicSupervisor)
  2. ✓ Implement ping functionality (System.cmd with OS-specific args)
  3. ✓ Create monitoring_checks schema with TimescaleDB hypertable
  4. ✓ Build GenServer workers for monitoring
  5. ✓ Add PubSub broadcasting
  6. ✓ Update LiveView to receive real-time updates
  7. ✓ Configure TimescaleDB retention policies
  8. ✓ Create continuous aggregates for metrics

Completed: 2025-12-21 Files Created:

  • lib/towerops/monitoring/check.ex - MonitoringCheck schema
  • lib/towerops/monitoring.ex - Monitoring context
  • lib/towerops/monitoring/ping.ex - Ping functionality
  • lib/towerops/monitoring/equipment_monitor.ex - GenServer for monitoring individual equipment
  • lib/towerops/monitoring/supervisor.ex - Supervisor for managing monitor workers
  • priv/repo/migrations/*_create_monitoring_checks.exs - Database migration with TimescaleDB hypertable

TimescaleDB Integration:

  • monitoring_checks converted to hypertable partitioned by checked_at
  • Retention policy: Keep raw data for 90 days, then automatically delete
  • Compression policy: Compress chunks older than 7 days
  • Continuous aggregates: Hourly and daily rollups for dashboard performance

Stage 4: Alerting ✓ COMPLETE

Goal: Alert generation and management Success Criteria:

  • ✓ Alerts created on status changes
  • ✓ Users can view alert history
  • ✓ Users can acknowledge alerts
  • ✓ Real-time alert notifications via PubSub

Tasks:

  1. ✓ Create alerts schema
  2. ✓ Build alert creation logic in monitoring workers
  3. ✓ Create alert LiveView pages
  4. ✓ Add alert acknowledgment
  5. ✓ Alert notifications in UI
  6. ✓ Dashboard integration with active alerts

Completed: 2025-12-21 Files Created:

  • priv/repo/migrations/*_create_alerts.exs - Alerts table migration
  • lib/towerops/alerts/alert.ex - Alert schema
  • lib/towerops/alerts.ex - Alerts context
  • lib/towerops_web/live/alert_live/index.ex - Alerts listing page
  • lib/towerops_web/live/alert_live/index.html.heex - Alerts UI

Files Modified:

  • lib/towerops/monitoring/equipment_monitor.ex - Alert creation on status changes
  • lib/towerops_web/live/dashboard_live.ex - Active alerts display
  • lib/towerops_web/live/dashboard_live.html.heex - Dashboard UI updates
  • lib/towerops_web/router.ex - Alert routes

Stage 5: Polish & Production ✓ COMPLETE

Goal: Production-ready application Success Criteria:

  • ✓ All tests passing (154 tests, 60.83% coverage)
  • ✓ Polished UI with Tailwind (removed daisyUI, custom components)
  • ✓ Email notifications configured
  • ✓ Production deployment ready

Tasks:

  1. ✓ Comprehensive test coverage (100% on core business logic)
  2. ✓ UI/UX improvements (Dashboard redesigned with custom Tailwind)
  3. ✓ Email alert notifications (SMTP-based, sent to owners/admins)
  4. ✓ Performance optimization (Added monitoring_enabled index)
  5. ✓ Documentation (Complete DEPLOYMENT.md guide)

Completed: 2025-12-24 Files Created:

  • lib/towerops/alerts/alert_notifier.ex - Email notification service
  • DEPLOYMENT.md - Comprehensive production deployment guide
  • priv/repo/migrations/*_add_email_sent_at_to_alerts.exs - Email tracking
  • priv/repo/migrations/*_add_monitoring_enabled_index.exs - Performance optimization

Files Modified:

  • lib/towerops/alerts/alert.ex - Added email_sent_at field
  • lib/towerops/alerts.ex - Added send_alert_notification/1 function
  • lib/towerops/organizations.ex - Added list_organization_notification_recipients/1
  • lib/towerops/monitoring/equipment_monitor.ex - Integrated email sending
  • lib/towerops_web/live/dashboard_live.html.heex - Modern Tailwind UI

Email Notification System:

  • Sends emails to organization owners and admins
  • Equipment down alerts with details
  • Equipment recovery notifications
  • Disabled in test environment to avoid connection issues
  • Tracks email_sent_at timestamp

UI Improvements:

  • Removed all daisyUI classes
  • Custom Tailwind components with dark mode support
  • Modern card-based dashboard layout
  • Proper semantic color scheme (zinc, blue, red, green, amber)
  • Improved accessibility and responsiveness

Performance Optimizations:

  • Added index on equipment.monitoring_enabled for fast filtering
  • All existing queries already optimized with proper indexes
  • TimescaleDB compression and retention in production

Production Readiness:

  • Complete deployment documentation
  • Environment-specific configurations
  • SSL/TLS setup guide
  • Docker and systemd examples
  • Database backup procedures
  • Troubleshooting guide

Stage 6: Distributed Monitoring Agents (Future)

Goal: Deploy-able Rust-based monitoring agents for internal network monitoring Status: PLANNED - Not yet started

Overview: Customer-deployable Rust binary that runs inside customer networks to monitor equipment from within their own infrastructure. Agents authenticate using deployment keys and report monitoring results back to the TowerOps platform.

Success Criteria:

  • Rust agent binary can be deployed on customer networks (Linux, Windows, macOS)
  • Agent authenticates with deployment key tied to organization
  • Agent pings equipment reachable on local network
  • Results reported back to TowerOps API
  • Equipment can be configured to use agent-based or platform-based monitoring
  • Agent status visible in TowerOps dashboard

Architecture:

  • Rust Agent: Lightweight binary for customer deployment

    • Ping functionality using OS-native ICMP
    • Configurable check intervals
    • Local caching/queue for offline resilience
    • Secure API communication over HTTPS
    • Auto-update capability

  • Deployment Keys: Scoped API credentials

    • Per-organization deployment keys
    • Limited scope (can only submit monitoring results)
    • Revocable from TowerOps dashboard
    • Multiple keys per organization for different sites/networks

  • API Endpoints: Backend support for agent communication

    • POST /api/v1/monitoring/checks - Submit monitoring results
    • GET /api/v1/monitoring/config - Fetch equipment list for agent
    • Authentication via deployment key header

Tasks:

  1. Design deployment key schema and API
  2. Create agent registration and key management in TowerOps
  3. Build Rust monitoring agent
    • ICMP ping implementation
    • API client for result submission
    • Configuration management
    • Error handling and retry logic
  4. Add equipment assignment to agents
  5. Agent status monitoring in dashboard
  6. Documentation for agent deployment

Database Changes Needed:

# deployment_keys table
- id (binary_id)
- organization_id (FK)
- name (string) - Human-readable name for the key
- key_hash (string) - Hashed version of the key
- last_used_at (utc_datetime)
- created_by_id (FK -> users)
- revoked_at (utc_datetime, nullable)

# monitoring_agents table
- id (binary_id)
- organization_id (FK)
- deployment_key_id (FK)
- name (string)
- version (string) - Agent version
- last_seen_at (utc_datetime)
- status (enum: active, inactive, error)

# equipment table additions
- monitoring_agent_id (FK -> monitoring_agents, nullable)
- monitoring_source (enum: platform, agent) - Where monitoring happens

Notes:

  • This is a significant feature and will be implemented much later
  • Current platform-based monitoring (Stage 3) remains as fallback
  • Allows monitoring of internal/private networks not accessible from internet
  • Agent runs continuously, not triggered from platform

Technical Decisions

Database

  • PostgreSQL with Ecto
  • Use binary_id (UUID) for all primary keys (already configured)
  • Indexes on foreign keys and frequently queried fields

TimescaleDB for Time-Series Data:

  • TimescaleDB extension enabled on PostgreSQL
  • monitoring_checks table converted to hypertable
  • Automatic partitioning by time (checked_at column)
  • Retention policy: 90 days for raw data
  • Compression policy: Compress chunks older than 7 days
  • Continuous aggregates for dashboard performance:
    • monitoring_checks_hourly - Hourly rollups (avg response time, success rate)
    • monitoring_checks_daily - Daily rollups for long-term trends

Why TimescaleDB:

  • Built as PostgreSQL extension (not separate database)
  • Works seamlessly with Ecto
  • Optimized for time-series queries
  • Automatic data lifecycle management
  • Fast aggregations for dashboards
  • Scales to millions of monitoring checks

Background Jobs

Options:

  1. Custom OTP solution with GenServer + Process.send_after
  2. Quantum scheduler
  3. Oban (more heavyweight but robust)

Recommendation: Start with custom OTP, migrate to Oban if needed

Real-time Communication

  • Phoenix PubSub for server-side messaging
  • LiveView for UI updates
  • No WebSocket client code needed

Ping Implementation

Options:

  1. :gen_icmp library (requires raw sockets, might need permissions)
  2. System.cmd("ping") - simpler, cross-platform
  3. HTTP health checks (future enhancement)

Recommendation: System.cmd("ping") for MVP, abstract for future protocols

Decisions Made

  1. Email notifications: ✓ Yes - Users receive email alerts on status changes
  2. Multi-org users: ✓ Yes - Users can belong to multiple organizations
  3. Invite system: ✓ Email invitation system (secure token-based invites)
  4. Check intervals: ✓ Customizable per equipment (stored in equipment.check_interval_seconds)

Additional Schema Needed

Based on decisions:

  • OrganizationInvitations table for email invite workflow
  • email_sent_at field in Alerts for tracking email delivery

Open Questions

  1. Alert escalation: Any escalation policies (e.g., page admin if down > X minutes)?
  2. Retention: How long to keep monitoring_checks history?
  3. Equipment types: Just ping for now, or plan for SNMP, HTTP, etc.?
  4. Site hierarchy depth: Any limit on site nesting levels?

Status: Stage 5 Complete - Production Ready! 🚀

Completed Stages:

  • ✓ Stage 1: Foundation & Authentication (2025-12-21)
  • ✓ Stage 2: Sites & Equipment Management (2025-12-21)
  • ✓ Stage 3: Monitoring System with TimescaleDB (2025-12-21)
  • ✓ Stage 4: Alerting (2025-12-21)
  • ✓ Stage 5: Polish & Production (2025-12-24)

Current Status: All tests passing (154/154), production-ready

Features Implemented:

  • Multi-tenant organizations with role-based access
  • Hierarchical site management
  • Equipment monitoring with ping checks
  • TimescaleDB time-series optimization (production-ready)
  • Automatic alerting on status changes
  • Email notifications to owners/admins
  • Real-time dashboard with LiveView
  • Alert acknowledgment system
  • Modern Tailwind UI (no daisyUI)
  • Comprehensive deployment documentation
  • Performance optimizations (database indexes)

Next Steps:

  • Stage 6: Distributed Monitoring Agents (Future - see above for details)
  • Deploy to production following DEPLOYMENT.md guide

Last updated: 2025-12-24