towerops/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

CRITICAL: Read AGENTS.md First

Before starting any work in this repository, you MUST read AGENTS.md in the project root.

• AGENTS.md contains comprehensive Phoenix/LiveView/Elixir guidelines that are mandatory for this project
• These guidelines take precedence over general development practices when there's a conflict
• Always read AGENTS.md at the start of a new conversation or when resuming work
• Follow the project-specific patterns, conventions, and constraints documented there

Project Overview

Towerops is a Phoenix 1.8 web application built with Elixir, using Ecto for database operations (PostgreSQL), LiveView for real-time interactions, and Tailwind CSS v4 for styling.

Data Model Relationships

┌─────────────┐
│    User     │
│ (Accounts)  │
└──────┬──────┘
       │ owns/member of (many-to-many)
       │
       ↓
┌──────────────────────────────────────────────────────┐
│                   Organization                        │
│                  (Organizations)                      │
└────┬────────────┬─────────────┬──────────────────────┘
     │            │             │
     │ has many   │ has many    │ has many
     │            │             │
     ↓            ↓             ↓
┌─────────┐  ┌──────────┐  ┌────────────────┐
│  Site   │  │Equipment │  │  AgentToken    │
│ (Sites) │  │(Equipment│  │   (Agents)     │
└────┬────┘  └────┬─────┘  └────────┬───────┘
     │            │                  │
     │ has many   │ has one          │ assigned via
     │            │                  │
     ↓            ↓                  ↓
┌──────────┐ ┌──────────────┐  ┌──────────────────┐
│Equipment │ │  SNMPDevice  │  │ AgentAssignment  │◀──┐
│(Equipment│ │(Snmp.Devices)│  │    (Agents)      │   │
└────┬─────┘ └──────────────┘  └──────────────────┘   │
     │                                  │               │
     │ has many                         └───────────────┘
     ├────────────────┬─────────────────┘
     │                │               assigns equipment
     ↓                ↓                  to agent
┌──────────────┐ ┌─────────┐
│MonitoringCheck│ │ Alert   │
│ (Monitoring) │ │ (Alerts)│
└──────────────┘ └─────────┘

Key Relationships:
- User can own/belong to multiple Organizations
- Organization has default_agent_token_id (optional)
- Equipment belongs to both Site and Organization (denormalized)
- Equipment can be assigned to one AgentToken via AgentAssignment
- Equipment has one SNMPDevice with Sensors and Interfaces
- Equipment has many MonitoringChecks (polling results)
- Equipment has many Alerts (equipment_down, equipment_up)
- AgentToken authenticates remote agents for local SNMP polling

Note to AI assistants: When you make changes to the data model (migrations, schema additions, new contexts), update this diagram to reflect the current state.

Essential Commands

Setup and Development

• mix setup - Install dependencies, create/migrate database, setup and build assets
• mix phx.server - Start the Phoenix server (accessible at http://localhost:4000)
• iex -S mix phx.server - Start server with interactive Elixir shell

Testing and Quality

• mix test - Run all tests (automatically creates and migrates test database)
• mix test test/path/to/specific_test.exs - Run a specific test file
• mix test --failed - Re-run only previously failed tests
• mix test --cover - Run tests with coverage report (generates HTML report in cover/ directory)
• mix precommit - Run before committing: compiles with warnings as errors, unlocks unused deps, formats code, and runs tests

Coverage Target: 90% minimum (currently configured threshold)

• Coverage reports show line-by-line coverage in cover/ directory
• View detailed HTML report: open cover/modules_*.html
• Current status: 60.40% overall (as of Jan 12, 2026, 6:35 PM)
• Recent improvements (Session 1):
  • Towerops.Snmp: 10.87% → 97.83% (+87%)
  • Towerops.Snmp.Profiles.NetSnmp: 56.79% → 92.59% (+35.8%)
  • Towerops.Monitoring.Supervisor: 21.21% → 63.64% (+42%)
  • Towerops.Snmp.Profiles.Cisco: 20.22% → 57.30% (+37%)
  • ToweropsWeb.EquipmentLive.Show: 35.51% → 46.38% (+10.87%)

Coverage Notes:

• Protobuf-generated modules (Towerops.Agent.*) show 0% but don't need tests
• Focus areas for improvement (in priority order):
  • Towerops.Monitoring context (33.33%)
  • SNMP profiles: NetSnmp (56.79%), Base (69.52%), Mikrotik (77.97%)
  • LiveView modules: EquipmentLive.Show (35.51%), EquipmentLive.Form (46.73%)
  • Accounts modules: UserCredentialController (33.93%), Accounts context (61.82%)
  • 0% coverage modules (Admin, WebAuthn, etc.) - may be intentionally untested

Database

• mix ecto.create - Create the database
• mix ecto.migrate - Run pending migrations
• mix ecto.reset - Drop, create, and migrate database
• mix ecto.gen.migration migration_name_using_underscores - Generate a new migration file

Assets

• mix assets.build - Build CSS and JS assets (Tailwind + esbuild)
• mix assets.deploy - Build minified assets for production

Architecture

Application Structure

The application follows standard Phoenix conventions with clear separation between business logic (lib/towerops/) and web interface (lib/towerops_web/):

• Towerops.Application - OTP application that supervises:

  • Telemetry for metrics
  • Repo (Ecto) for database
  • DNSCluster for service discovery
  • PubSub for pub/sub messaging
  • Endpoint (web server)

• ToweropsWeb - Main web module that provides use macros for:

  • :router - Route definitions
  • :controller - Traditional request/response controllers
  • :live_view - LiveView modules
  • :live_component - LiveView component modules
  • :html - Phoenix.Component modules

Key Configuration Details

• Binary IDs: Generators use binary (UUID) primary keys by default (binary_id: true)
• Timestamps: Use :utc_datetime for all timestamps
• Web server: Uses Bandit adapter (not Cowboy)
• HTTP client: Uses :req library (Req module) - this is the only approved HTTP client
• Ecto repos: [Towerops.Repo]

Web Layer Patterns

All LiveViews, LiveComponents, and HTML modules automatically get these imports/aliases via html_helpers/0:

• ToweropsWeb.CoreComponents - Core UI components (<.button>, <.input>, <.form>, etc.)
• ToweropsWeb.Layouts - Layout components (aliased, no need to re-alias)
• Phoenix.LiveView.JS - Client-side JS commands
• Gettext for translations
• Verified routes with ~p sigil

Asset Pipeline

• Tailwind CSS v4: Uses new @import "tailwindcss" syntax in assets/css/app.css, no tailwind.config.js needed
• esbuild: Bundles JavaScript from assets/js/app.js
• All vendor scripts/styles must be imported into app.js/app.css - cannot reference external src/href in layouts
• No inline <script> tags in templates - use colocated LiveView hooks instead

Development Environment

• Dev routes enabled for:
  • LiveDashboard at /dev/dashboard
  • Swoosh mailbox preview at /dev/mailbox
• Phoenix LiveReload watches for file changes
• Code reloader enabled via listeners: [Phoenix.CodeReloader]

Project-Specific Constraints

Key constraints from AGENTS.md (see that file for complete details):

• Use mix precommit before committing changes
• Use :req (Req) for all HTTP requests - never use :httpoison, :tesla, or :httpc
• Never use daisyUI - write custom Tailwind components for world-class design
• LiveView templates must start with <Layouts.app flash={@flash}>
• Use <.icon name="hero-x-mark"> component for icons, never Heroicons modules
• Use LiveView streams for collections to avoid memory issues
• Forms must use to_form/2 in LiveView and <.form for={@form}> in templates
• Never access changesets directly in templates - always use the form assign
• Tailwind v4 uses new import syntax - never use @apply in CSS
• when adding/updating hex modules, only use https://hex.pm
• assets are rebuilt on save and don't need to be built with mix assets.build
• be sure to always run mix format after you modify an elixir file
• when you run mix format, don't specify a file, let it format everything

Database Schema Critical Notes

Binary UUID Primary Keys: All tables MUST use :binary_id for primary keys, not bigint.

When creating new migrations with tables that have UUID primary keys:

create table(:table_name, primary_key: false) do
  add :id, :binary_id, primary_key: true
  # ...
end

Issue History: The snmp_neighbors table was initially created with a bigint primary key instead of :binary_id, causing DBConnection.EncodeError when trying to insert records. This happened because the migration file was correct but the table was created before the fix. Always verify the actual database schema matches the migration:

psql towerops_dev -c "\d table_name"  # Check actual schema

If a table has the wrong primary key type, create a fix migration that drops and recreates the table.

SNMP Binary Data Handling: All SNMP values (chassis IDs, port IDs, system names) must be converted to printable strings before saving to the database. Non-printable binaries (like MAC addresses in binary format) should be converted to colon-separated hex format using String.printable?/1 checks. See lib/towerops/snmp/neighbor_discovery.ex:sanitize_string_field/1 for the pattern.

SNMP Polling Architecture

The application has two separate SNMP collection mechanisms:

1. Discovery (Towerops.Snmp.Discovery) - One-time or manual collection

• Triggered manually by user or on a schedule
• Collects: system info, device identification, interfaces, sensors, neighbors
• Creates/updates the device record and all associated data
• Typically runs when equipment is first added or when user clicks "Rediscover"

2. Polling (Towerops.Snmp.PollerWorker) - Continuous time-series collection

• Runs automatically every 60 seconds (configurable per equipment)
• Collects: sensor readings, interface statistics, neighbor topology
• Updates time-series data without recreating device records
• Keeps neighbor data fresh by polling LLDP/CDP every minute

Neighbor Data Lifecycle:

• Discovery saves initial neighbors with 5-minute stale threshold
• Poller continuously updates neighbors every 60 seconds with 5-minute stale threshold
• NeighborCleanupWorker deletes neighbors older than 24 hours (safety net)
• This ensures neighbors stay current and topology changes are detected quickly

Kubernetes Deployment

Prerequisites for Talos Kubernetes Cluster

The application requires the following infrastructure components to be installed in the cluster before deployment:

1. Core Infrastructure

cert-manager - Manages SSL/TLS certificates

• Required for automatic Let's Encrypt certificate provisioning
• Must have ClusterIssuers configured: letsencrypt-prod and letsencrypt-staging

Traefik - Ingress controller

• Handles HTTP/HTTPS traffic routing
• Must be configured with web (HTTP) and websecure (HTTPS) entrypoints
• The application uses IngressRoute CRDs (not standard Ingress resources)

MetalLB - Load balancer for bare metal clusters

• Provides LoadBalancer service type support in on-premises environments
• Required for Traefik to get an external IP

FluxCD - GitOps continuous delivery

• Automatically syncs Kubernetes resources from Git repository
• Required components: source-controller, kustomize-controller, helm-controller, notification-controller
• Must have GitRepository resource pointing to this repository
• Must have Kustomization resource for the k8s/ directory

2. Storage

NFS Provisioner - Dynamic persistent volume provisioning

• Provides StorageClass for persistent volume claims
• Required for stateful workloads (if needed in the future)

3. CI/CD Integration

GitLab Agent - Connects cluster to GitLab for CI/CD

• Enables GitLab CI/CD pipelines to deploy to the cluster
• Required for automated deployments from GitLab CI
• Agent configuration managed in .gitlab/agents/ directory
• Agent must be registered in GitLab project settings

4. Network Services

Newt - Pangolin tunnel forwarding

• Enables external access to internal services
• Required for Pangolin protocol forwarding
• Configuration specific to network topology

CoreDNS - Cluster DNS

• Standard Kubernetes DNS service (included in Talos by default)

Tailscale Operator - VPN mesh networking (optional)

• Provides secure access to cluster services
• Useful for development and administrative access

Secrets Management

All secrets are stored in 1Password and must be created manually in the cluster before deployment.

Required Secrets in towerops Namespace

1. gitlab-registry - Docker registry credentials

   • Type: kubernetes.io/dockerconfigjson
   • Used to pull images from GitLab Container Registry
   • Retrieve from 1Password or GitLab project settings

2. towerops-secrets - Application secrets

   • Type: Opaque
   • Fields:
     • RELEASE_COOKIE - Erlang distributed cookie for node clustering
     • SECRET_KEY_BASE - Phoenix secret key base for session encryption
   • Stored in 1Password: "Towerops K8s Secrets"

3. towerops-db - Database connection

   • Type: Opaque
   • Fields:
     • POSTGRES_HOST - Database hostname
     • POSTGRES_PORT - Database port
     • POSTGRES_DB - Database name
     • POSTGRES_USER - Database username
     • POSTGRES_PASSWORD - Database password
     • DATABASE_URL - Full connection URL
   • Stored in 1Password: "Towerops K8s Database"

4. towerops-aws - AWS credentials

   • Type: Opaque
   • Fields:
     • AWS_ACCESS_KEY_ID - AWS access key
     • AWS_SECRET_ACCESS_KEY - AWS secret key
     • AWS_REGION - AWS region (us-east-1)
   • Stored in 1Password: "Towerops K8s AWS"

Creating Secrets from 1Password

# Set 1Password account
export OP_ACCOUNT=YOOATCZZSVGH7AD6VABUVPORLI

# Create towerops-secrets
kubectl create secret generic towerops-secrets -n towerops \
  --from-literal=RELEASE_COOKIE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=RELEASE_COOKIE)" \
  --from-literal=SECRET_KEY_BASE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=SECRET_KEY_BASE)"

# Create towerops-db
kubectl create secret generic towerops-db -n towerops \
  --from-literal=POSTGRES_HOST="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=server)" \
  --from-literal=POSTGRES_PORT="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=port)" \
  --from-literal=POSTGRES_DB="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=database)" \
  --from-literal=POSTGRES_USER="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=username)" \
  --from-literal=POSTGRES_PASSWORD="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=password)" \
  --from-literal=DATABASE_URL="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=DATABASE_URL)"

# Create towerops-aws
kubectl create secret generic towerops-aws -n towerops \
  --from-literal=AWS_ACCESS_KEY_ID="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_ACCESS_KEY_ID)" \
  --from-literal=AWS_SECRET_ACCESS_KEY="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_SECRET_ACCESS_KEY)" \
  --from-literal=AWS_REGION="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_REGION)"

Deployment Process

FluxCD Automatic Deployment

The application is automatically deployed via FluxCD GitOps:

1. FluxCD monitors the Git repository for changes
2. When changes are detected, it applies the manifests in k8s/ directory
3. Kustomize builds the manifests and applies them to the cluster

GitRepository Configuration:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: towerops
  namespace: flux-system
spec:
  interval: 1m0s
  url: ssh://git@gitlab.com/graham/towerops.git
  ref:
    branch: main
  secretRef:
    name: towerops-git

Kustomization Configuration:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: towerops-app
  namespace: flux-system
spec:
  interval: 1m0s
  path: ./k8s
  prune: true
  sourceRef:
    kind: GitRepository
    name: towerops
  targetNamespace: towerops

Manual Deployment

If FluxCD is not available or for testing:

# Apply all resources using kustomize
kubectl apply -k k8s/

# Or apply individually
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/service-headless.yaml
kubectl apply -f k8s/certificate.yaml
kubectl apply -f k8s/ingressroute.yaml

Cluster Setup Checklist

When recreating a Talos cluster from scratch:

• Install Talos on nodes and bootstrap cluster
• Install MetalLB and configure IP address pool
• Install Traefik ingress controller
• Install cert-manager and configure ClusterIssuers
• Install FluxCD and configure GitRepository source
• Install GitLab Agent and connect to GitLab project
• Install Newt for Pangolin forwarding
• Install NFS provisioner (if using persistent storage)
• Create towerops namespace
• Create all required secrets from 1Password
• Create FluxCD Kustomization resource for k8s/ directory
• Verify deployment with kubectl get pods -n towerops
• Check certificate provisioning with kubectl get certificate -n towerops
• Test ingress with curl https://towerops.net/health

Troubleshooting

FluxCD reconciliation errors:

• Check kustomization status: kubectl get kustomization -n flux-system
• View detailed status: kubectl describe kustomization towerops-app -n flux-system
• Force reconciliation: kubectl annotate kustomization towerops-app -n flux-system reconcile.fluxcd.io/requestedAt="$(date +%Y-%m-%dT%H:%M:%S%z)" --overwrite

Certificate issues:

• Check certificate status: kubectl describe certificate towerops-net-cert -n towerops
• View cert-manager logs: kubectl logs -n cert-manager deployment/cert-manager
• Verify ClusterIssuer: kubectl get clusterissuer letsencrypt-prod -o yaml

Pod not starting:

• Check pod status: kubectl describe pod -n towerops -l app=towerops
• View logs: kubectl logs -n towerops deployment/towerops
• Verify secrets exist: kubectl get secrets -n towerops

Testing Patterns

SNMP Mocking with Mox

The application uses Mox for SNMP client mocking in tests. Key patterns:

SNMP Adapter Mock Format:

• snmp_adapter().get/3 returns {:ok, value} or {:error, reason}
• snmp_adapter().walk/3 returns {:ok, [%{oid: "...", value: ...}]} (list of maps, NOT a map)
• Values from get/3 are already extracted (no type wrapper needed in mocks)

Mock Expectations:

# Mock get_multiple - it calls get/3 once per OID
expect(SnmpMock, :get, 6, fn _target, oid, _opts ->
  case oid do
    "1.3.6.1.2.1.1.1.0" -> {:ok, "Cisco IOS Software"}  # sysDescr
    "1.3.6.1.2.1.1.2.0" -> {:ok, [1, 3, 6, 1, 4, 1, 9]}  # sysObjectID
    "1.3.6.1.2.1.1.3.0" -> {:ok, 12_345}  # sysUpTime (integer)
    "1.3.6.1.2.1.1.4.0" -> {:ok, "admin@example.com"}  # sysContact
    "1.3.6.1.2.1.1.5.0" -> {:ok, "test-device"}  # sysName
    "1.3.6.1.2.1.1.6.0" -> {:ok, "Test Location"}  # sysLocation
  end
end)

# Mock walk - returns list of OID/value maps
expect(SnmpMock, :walk, fn _target, _oid, _opts ->
  {:ok, [
    %{oid: "1.3.6.1.4.1.9.9.91.1.1.1.1.1.1000", value: 8}
  ]}
end)

# Empty walk result (no data found)
expect(SnmpMock, :walk, fn _, _, _ ->
  {:ok, []}  # NOT {:ok, %{}}
end)

Common Pitfalls:

• ❌ Returning {:ok, %{}} from walk - should be {:ok, []}
• ❌ Returning type wrappers like {:integer, 123} - values are already extracted
• ❌ Not matching OIDs in get expectations - use pattern matching on OID string
• ❌ Expecting wrong number of calls - get_multiple/2 calls get/3 once per OID

TimescaleDB Tests:

• Tests that query continuous aggregates should be tagged with @tag :skip
• Reason: Test database doesn't have TimescaleDB continuous aggregates configured
• Example: get_hourly_stats/3, get_daily_stats/3, get_uptime_percentage/1

Test Organization

• Use DataCase for tests that need database access
• Use ConnCase for controller/LiveView tests
• Use async: true for tests that can run in parallel (most unit tests)
• Use async: false for tests with shared state (supervisor tests, integration tests)
• never open the test coverage html files
• remember when working in rust to always run cargo fmt before committing