towerops/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

CRITICAL: Read AGENTS.md First

Before starting any work in this repository, you MUST read AGENTS.md in the project root.

• AGENTS.md contains comprehensive Phoenix/LiveView/Elixir guidelines that are mandatory for this project
• These guidelines take precedence over general development practices when there's a conflict
• Always read AGENTS.md at the start of a new conversation or when resuming work
• Follow the project-specific patterns, conventions, and constraints documented there

Project Overview

Towerops is a Phoenix 1.8 web application built with Elixir, using Ecto for database operations (PostgreSQL), LiveView for real-time interactions, and Tailwind CSS v4 for styling.

Data Model Relationships

┌─────────────┐
│    User     │
│ (Accounts)  │
└──────┬──────┘
       │ owns/member of (many-to-many)
       │
       ↓
┌──────────────────────────────────────────────────────┐
│                   Organization                        │
│                  (Organizations)                      │
└────┬────────────┬─────────────┬──────────────────────┘
     │            │             │
     │ has many   │ has many    │ has many
     │            │             │
     ↓            ↓             ↓
┌─────────┐  ┌──────────┐  ┌────────────────┐
│  Site   │  │Equipment │  │  AgentToken    │
│ (Sites) │  │(Equipment│  │   (Agents)     │
└────┬────┘  └────┬─────┘  └────────┬───────┘
     │            │                  │
     │ has many   │ has one          │ assigned via
     │            │                  │
     ↓            ↓                  ↓
┌──────────┐ ┌──────────────┐  ┌──────────────────┐
│Equipment │ │  SNMPDevice  │  │ AgentAssignment  │◀──┐
│(Equipment│ │(Snmp.Devices)│  │    (Agents)      │   │
└────┬─────┘ └──────────────┘  └──────────────────┘   │
     │                                  │               │
     │ has many                         └───────────────┘
     ├────────────────┬─────────────────┘
     │                │               assigns equipment
     ↓                ↓                  to agent
┌──────────────┐ ┌─────────┐
│MonitoringCheck│ │ Alert   │
│ (Monitoring) │ │ (Alerts)│
└──────────────┘ └─────────┘

Key Relationships:
- User can own/belong to multiple Organizations
- Organization has default_agent_token_id (optional)
- Equipment belongs to both Site and Organization (denormalized)
- Equipment can be assigned to one AgentToken via AgentAssignment
- Equipment has one SNMPDevice with Sensors and Interfaces
- Equipment has many MonitoringChecks (polling results)
- Equipment has many Alerts (equipment_down, equipment_up)
- AgentToken authenticates remote agents for local SNMP polling

Note to AI assistants: When you make changes to the data model (migrations, schema additions, new contexts), update this diagram to reflect the current state.

Essential Commands

Setup and Development

• mix setup - Install dependencies, create/migrate database, setup and build assets
• mix phx.server - Start the Phoenix server (accessible at http://localhost:4000)
• iex -S mix phx.server - Start server with interactive Elixir shell

Testing and Quality

• mix test - Run all tests (automatically creates and migrates test database)
• mix test test/path/to/specific_test.exs - Run a specific test file
• mix test --failed - Re-run only previously failed tests
• mix test --cover - Run tests with coverage report (view: open cover/modules_*.html)
• mix precommit - Run before committing: compiles with warnings as errors, unlocks unused deps, formats code, and runs tests
• mix dialyzer - Run static type analysis

Coverage Notes:

• Target: 90% minimum
• Protobuf-generated modules (Towerops.Agent.*) show 0% but don't need tests
• TimescaleDB aggregate tests should be tagged @tag :skip (test DB lacks continuous aggregates)

Database

• mix ecto.create - Create the database
• mix ecto.migrate - Run pending migrations
• mix ecto.reset - Drop, create, and migrate database
• mix ecto.gen.migration migration_name_using_underscores - Generate a new migration file

Assets

• mix assets.build - Build CSS and JS assets (Tailwind + esbuild)
• mix assets.deploy - Build minified assets for production

Architecture

Application Structure

Standard Phoenix conventions with separation between business logic (lib/towerops/) and web interface (lib/towerops_web/).

Key Configuration:

• Binary IDs: Generators use binary (UUID) primary keys by default (binary_id: true)
• Timestamps: Use :utc_datetime for all timestamps
• Web server: Bandit adapter (not Cowboy)
• HTTP client: :req library (Req module) - this is the only approved HTTP client
• Ecto repos: [Towerops.Repo]

Web Layer Patterns

All LiveViews, LiveComponents, and HTML modules automatically get these imports/aliases via html_helpers/0:

• ToweropsWeb.CoreComponents - Core UI components (<.button>, <.input>, <.form>, etc.)
• ToweropsWeb.Layouts - Layout components
• Phoenix.LiveView.JS - Client-side JS commands
• Gettext for translations
• Verified routes with ~p sigil

Asset Pipeline

• Tailwind CSS v4: Uses new @import "tailwindcss" syntax in assets/css/app.css, no tailwind.config.js needed
• esbuild: Bundles JavaScript from assets/js/app.js
• All vendor scripts/styles must be imported into app.js/app.css - cannot reference external src/href in layouts
• No inline <script> tags in templates - use colocated LiveView hooks instead
• Assets are rebuilt on save automatically

Development Environment

• Dev routes: LiveDashboard at /dev/dashboard, Swoosh mailbox at /dev/mailbox
• Phoenix LiveReload watches for file changes
• Code reloader enabled

Background Job Architecture (Oban)

Uses PostgreSQL-backed Oban for all background processing with cluster-wide coordination.

Queues:

• default (10 workers) - General background tasks
• discovery (10 workers) - SNMP discovery operations
• pollers (50 workers) - SNMP polling jobs (one per device)
• monitors (50 workers) - Health check jobs (one per device)
• maintenance (5 workers) - Periodic cleanup and health check workers

Key Workers:

1. Self-Scheduling Workers (per-device recurring jobs):

   • DeviceMonitorWorker - Health check polling (60s default)
   • DevicePollerWorker - SNMP data collection (60s default)
   • Schedule next run before completing, one job per enabled device
   • Automatically created/cancelled when device settings change

2. Oban Cron Workers (cluster-wide periodic maintenance):

   • NeighborCleanupWorker - Hourly cleanup of stale neighbors/ARP/MAC entries
   • StaleAgentWorker - Detects agents that haven't checked in for 10+ minutes (every minute)
   • AgentLatencyEvaluator - Latency-based agent reassignment (every 5 minutes)
   • JobHealthCheckWorker - Safety net to recover missing monitor/poller jobs (every 10 minutes)

Resilience:

• Oban Cron jobs run cluster-wide via PostgreSQL-based locking
• Self-scheduling workers recovered by JobHealthCheckWorker every 10 minutes
• All jobs visible in Oban dashboard at /dev/dashboard → Oban tab
• Failed jobs automatically retried with exponential backoff

Configuration:

• Dev: config/dev.exs - Oban config with cron plugin
• Prod: config/runtime.exs - Same cron schedule as dev
• Application: lib/towerops/application.ex - Oban supervision

Oban Web Dashboard:

• Location: /admin/oban (superuser access required)
• Vendored dependencies in vendor/: oban_pro, oban_web, oban_met
• See vendor/README.md for update instructions

LiveDashboard and Telemetry

Available at /admin/dashboard (or /dev/dashboard in development).

Metrics: Phoenix requests, DB queries, VM stats, Oban queue sizes, Redis/Valkey stats

• Collected every 10 seconds via telemetry_poller
• Access: Navigate to dashboard → "Metrics" tab

SNMP Polling Architecture

Two separate SNMP collection mechanisms:

1. Discovery (Towerops.Snmp.Discovery) - One-time or manual

• Triggered manually or via DiscoveryWorker Oban job
• Collects: system info, device identification, interfaces, sensors, neighbors
• Creates/updates device record and associated data

2. Polling (DevicePollerWorker) - Continuous time-series

• Runs automatically every 60 seconds (configurable per equipment)
• Collects: sensor readings, interface stats, neighbor topology
• Updates time-series data, keeps neighbor data fresh
• Direct Oban worker (no GenServer layer)

Load Distribution - Staggered Polling:

• Uses deterministic hash-based offsets via PollingOffset.calculate_offset/2
• Formula: offset = :erlang.phash2(device_id) |> rem(interval_seconds)
• Distributes devices evenly across full interval (prevents thundering herd)
• Same device always gets same offset (stable across restarts)
• Applied to both DevicePollerWorker and DeviceMonitorWorker
• Implementation: lib/towerops/workers/polling_offset.ex

MIB Name Resolution (Rust NIF)

Uses Rust NIF via Rustler for fast SNMP MIB name resolution (replaces unreliable Erlang SNMP modules).

Architecture:

• NIF Module: lib/towerops_native.ex
• Rust Implementation: native/towerops_native/src/mib.rs (uses snmptranslate command)
• Build Integration: mix.exs - Rustler compiler configured via rustler_crates

How It Works:

1. Application startup calls ToweropsNative.load_mib_directory/1 with priv/mibs path
2. Client calls resolve_mib_name/1 before SNMP operations
3. Tries ToweropsNative.resolve_oid/1 first, falls back to SnmpKit.resolve/1
4. Uses snmptranslate -On command-line tool with auto-detected system MIB directories

System Dependencies:

• Development (macOS): brew install net-snmp (keg-only, requires PKG_CONFIG_PATH)
• Production (Docker): apt-get install libsnmp-dev snmp-mibs-downloader
• Runtime: snmptranslate command must be in PATH

MIB Files:

• Located in priv/mibs/ directory (570+ vendor and standard MIB files)
• Standard MIBs: SNMPv2-MIB, IF-MIB, etc.
• Vendor MIBs: Cisco, Ubiquiti, IEEE, etc.

Important:

• SnmpKit still used for SNMP protocol operations (get, walk, get_next, get_bulk)
• NIF only handles MIB name resolution
• Rust compilation happens automatically via Mix
• Clean Rust builds: cd native/towerops_native && cargo clean

Admin Features

User Impersonation

Location: /admin/users (superuser only)

Files: lib/towerops_web/live/admin/user_live/index.ex, lib/towerops_web/user_auth.ex

Notes:

• Superusers CAN impersonate other superusers (security feature, not bug)
• Backend checks prevent self-impersonation
• All events logged to audit_logs table
• Stop impersonation link appears in user menu when active

GeoIP Database Management

Purpose: Import MaxMind GeoLite2-City database for IP-based country/city detection (GDPR cookie consent).

Files: lib/towerops_web/controllers/api/v1/geoip_controller.ex, lib/mix/tasks/geoip.import.ex

Import Methods:

# Local Import (Development)
make geoip-import DIR=~/Downloads/GeoLite2-City-CSV_20260127/

# Production Import via API (requires TOWEROPS_KEY env var)
make geoip-import-prod DIR=~/Downloads/GeoLite2-City-CSV_20260127/

Implementation:

• Local mode: Direct database import via Ecto
• Production mode: Processes CSVs locally, sends data to API in 5,000 record batches
• API endpoint: POST /admin/api/geoip/import (superuser only)
• Download from: https://dev.maxmind.com/geoip/geolite2-free-geolocation-data

Project-Specific Constraints

Key constraints from AGENTS.md (see that file for complete details):

• Use mix precommit before committing changes
• Use :req (Req) for all HTTP requests - never use :httpoison, :tesla, or :httpc
• Never use daisyUI - write custom Tailwind components
• LiveView templates must start with <Layouts.app flash={@flash}>
• Use <.icon name="hero-x-mark"> component for icons, never Heroicons modules
• Use LiveView streams for collections to avoid memory issues
• Forms must use to_form/2 in LiveView and <.form for={@form}> in templates
• Never access changesets directly in templates - always use the form assign
• Tailwind v4 uses new import syntax - never use @apply in CSS
• When adding/updating hex modules, only use https://hex.pm
• Always run mix format after modifying Elixir files (don't specify file, format everything)

Database Schema Critical Notes

Binary UUID Primary Keys: All tables MUST use :binary_id for primary keys, not bigint.

create table(:table_name, primary_key: false) do
  add :id, :binary_id, primary_key: true
  # ...
end

Issue History: The snmp_neighbors table was initially created with bigint instead of :binary_id, causing DBConnection.EncodeError. Always verify actual schema matches migration:

psql towerops_dev -c "\d table_name"  # Check actual schema

SNMP Binary Data Handling: All SNMP values must be converted to printable strings before saving. Non-printable binaries should be converted to colon-separated hex format. See lib/towerops/snmp/neighbor_discovery.ex:sanitize_string_field/1.

API Documentation

Available at /docs/api (adapted from Tailwind UI Protocol template).

Files: lib/towerops_web/controllers/api_docs_controller.ex, lib/towerops_web/controllers/api_docs_html/index.html.heex

When updating API endpoints:

1. Update controller documentation (@doc comments)
2. Update /docs/api template
3. Wrap code examples in <%= raw(~S"""...""") %> to prevent HEEx parsing curly braces

Current Documented Endpoints: Sites API, Devices API (GET/POST/PATCH/DELETE)

Kubernetes Deployment

Prerequisites: cert-manager, Traefik, MetalLB, FluxCD, GitLab Agent, NFS Provisioner Secrets: All stored in 1Password, created manually in towerops namespace

• gitlab-registry - Docker registry credentials
• towerops-secrets - RELEASE_COOKIE, SECRET_KEY_BASE
• towerops-db - PostgreSQL connection details
• towerops-aws - AWS credentials

Deployment: FluxCD monitors Git repository, automatically applies manifests in k8s/ directory Manual: kubectl apply -k k8s/

Troubleshooting:

• FluxCD: kubectl get kustomization -n flux-system
• Certificates: kubectl describe certificate towerops-net-cert -n towerops
• Pods: kubectl describe pod -n towerops -l app=towerops

Testing Patterns

SNMP Mocking with Mox

SNMP Adapter Mock Format:

• snmp_adapter().get/3 returns {:ok, value} or {:error, reason}
• snmp_adapter().walk/3 returns {:ok, [%{oid: "...", value: ...}]} (list of maps, NOT a map)
• Values from get/3 are already extracted (no type wrapper needed)

Common Pitfalls:

• ❌ Returning {:ok, %{}} from walk - should be {:ok, []}
• ❌ Returning type wrappers like {:integer, 123} - values are already extracted
• ❌ Not matching OIDs in get expectations - use pattern matching on OID string

Test Organization

• Use DataCase for database tests, ConnCase for controller/LiveView tests
• Use async: true for tests that can run in parallel
• Use async: false for tests with shared state
• Never open test coverage HTML files
• Never try to use npm to install JS things, use esbuild built in to Phoenix
• When working in Rust, always run cargo fmt before committing

Dialyzer: Static Type Analysis

Commands:

• mix dialyzer - Run analysis (builds PLT on first run)
• mix dialyzer --format dialyzer - Show detailed error locations
• mix dialyzer --format ignore_file - Generate .dialyzer_ignore.exs for gradual adoption

Configuration (mix.exs):

defp dialyzer do
  [
    plt_add_deps: :apps_direct,  # Faster incremental analysis
    plt_file: {:no_warn, "priv/plts/dialyzer.plt"},
    flags: [:error_handling, :underspecs]
  ]
end

Common Warning Types:

1. Invalid Contracts - Return type doesn't match all code paths
2. Function Application Arguments - Type alignment issues
3. Opaque Type Mismatches - External module accessing opaque type internals
4. Range Errors - Incomplete specs (e.g., not accounting for empty list case)

Workflow:

• Start with broad types, narrow gradually
• Use ignore file for gradual adoption in legacy code
• Combine with runtime type checking for complete safety
• PLT files in priv/plts/ - don't commit
• Never suppress valid warnings - fix the root cause