towerops/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

CRITICAL: Read AGENTS.md First

Before starting any work in this repository, you MUST read AGENTS.md in the project root.

• AGENTS.md contains mandatory Phoenix/LiveView/Elixir coding guidelines AND project-specific patterns for this codebase
• Covers: Elixir/OTP conventions, Phoenix/LiveView best practices, browser navigation URL state, JS hook memory management, form handling patterns, test guidelines, quality/security checks
• These guidelines take precedence over general development practices when there's a conflict
• Always read AGENTS.md at the start of a new conversation or when resuming work

Project Overview

Towerops is a Phoenix 1.8 web application built with Elixir, using Ecto for database operations (PostgreSQL), LiveView for real-time interactions, and Tailwind CSS v4 for styling.

Data Model Relationships

User (Accounts) → owns/member of → Organization (Organizations)
                                    ├─ Site (Sites) → Equipment
                                    ├─ Equipment → SNMPDevice, MonitoringCheck, Alert
                                    └─ AgentToken → AgentAssignment → Equipment

Key Relationships:
- User can own/belong to multiple Organizations
- Organization has default_agent_token_id (optional)
- Equipment belongs to both Site and Organization (denormalized)
- Equipment can be assigned to one AgentToken via AgentAssignment
- Equipment has one SNMPDevice with Sensors and Interfaces
- Equipment has many MonitoringChecks (polling results) and Alerts
- AgentToken authenticates remote agents for local SNMP polling

Note: Update this diagram when making data model changes.

Essential Commands

Setup and Development

• mix setup - Install dependencies, create/migrate database, build assets
• mix phx.server - Start Phoenix server (http://localhost:4000)
• iex -S mix phx.server - Start server with IEx shell

Testing and Quality

• mix test - Run all tests
• mix test --failed - Re-run failed tests
• mix test --cover - Run with coverage (target: 90% minimum)
• mix precommit - Run before committing: compiles with warnings as errors, formats, runs tests
• mix dialyzer - Static type analysis
• cd e2e && npm test - Run end-to-end tests (Playwright)

E2E Testing: When adding or modifying user-facing features, ALWAYS add corresponding e2e tests in e2e/tests/. E2E tests ensure the full user experience works across browsers (chromium, firefox, webkit). Tests should be defensive (use if (await element.isVisible()) checks) and handle edge cases like missing data or sudo verification redirects.

Database

• mix ecto.create/migrate/reset - Database operations
• mix ecto.gen.migration name_using_underscores - Generate migration

Assets

• mix assets.build/deploy - Build CSS/JS assets (auto-rebuilds on save in dev)

Architecture

Application Structure

Standard Phoenix conventions: business logic in lib/towerops/, web interface in lib/towerops_web/.

Key Configuration:

• Binary IDs: UUID primary keys by default (binary_id: true)
• Timestamps: :utc_datetime for all timestamps
• Web server: Bandit adapter
• HTTP client: :req library (Req module) - ONLY approved client
• Ecto repos: [Towerops.Repo]

Web Layer

All LiveViews get these imports via html_helpers/0:

• ToweropsWeb.CoreComponents - Core UI (<.button>, <.input>, <.form>)
• ToweropsWeb.Layouts, Phoenix.LiveView.JS, Gettext, verified routes (~p)

Rate Limiting

Uses Hammer (ETS-backed):

• Auth endpoints: 10 req/min per IP (/users/log-in, /users/register, TOTP)
• API v1: 1000 req/min per IP (/api/v1/*)
• Admin API: Not rate limited (superuser only)
• Returns 429 Too Many Requests with Retry-After header
• Disabled in test: config :towerops, :rate_limiting_enabled, false

Asset Pipeline

• Tailwind CSS v4: Uses @import "tailwindcss" in assets/css/app.css (no config file)
• esbuild: Bundles assets/js/app.js
• Import all vendor assets into app.js/app.css - no external src/href in layouts
• No inline <script> tags - use LiveView hooks

Custom Ecto Types

Avoid "primitive obsession" by using custom types:

Available Types:

1. Towerops.EctoTypes.IpAddress - IPv4/IPv6 validation, struct with version/tuple
2. Towerops.EctoTypes.EmailAddress - Normalized emails (Planned)
3. Towerops.EctoTypes.MacAddress - MAC address normalization (Planned)

Pattern: Implement Ecto.Type with type/0, cast/1, load/1, dump/1.

When to use:

• ✅ Domain concepts with validation rules (emails, IPs, phone numbers)
• ✅ Values needing normalization (case-insensitive emails, MAC addresses)
• ✅ Multiple representations (IP strings vs tuples)
• ❌ Simple strings, primitives, one-off validations

Migration: No database changes needed - column stays primitive, Ecto handles conversion.

Background Jobs (Oban)

PostgreSQL-backed with cluster-wide coordination.

Queues:

• default (10) - General tasks
• discovery (10) - SNMP discovery
• pollers (50) - SNMP polling (per-device)
• monitors (50) - Health checks (per-device)
• maintenance (5) - Periodic cleanup

Key Workers:

1. Self-Scheduling (per-device):

   • DeviceMonitorWorker - Health checks (60s default)
   • DevicePollerWorker - SNMP data collection (60s default)
   • Auto-created/cancelled when device settings change

2. Oban Cron (cluster-wide):

   • NeighborCleanupWorker - Hourly stale data cleanup
   • StaleAgentWorker - Detect agents offline 10+ minutes (every minute)
   • AgentLatencyEvaluator - Latency-based reassignment (every 5 minutes)
   • JobHealthCheckWorker - Recover missing jobs (every 10 minutes)

Resilience: Oban Cron uses PostgreSQL locking, self-scheduling recovered every 10 minutes. Dashboard: /admin/oban (superuser), /dev/dashboard → Oban tab (dev)

SNMP Polling

Two mechanisms:

1. Discovery (Towerops.Snmp.Discovery) - One-time/manual, collects full device info
2. Polling (DevicePollerWorker) - Continuous (60s default), time-series data

Load Distribution: Staggered polling using hash-based offsets:

• offset = :erlang.phash2(device_id) |> rem(interval_seconds)
• Prevents thundering herd, stable across restarts
• Implementation: lib/towerops/workers/polling_offset.ex

MikroTik API Integration

RouterOS API access alongside SNMP.

Architecture:

• 3-tier credential cascade: Organization → Site → Device (like SNMP)
• Encryption: Passwords encrypted at rest (Cloak AES-256-GCM)
• Detection: Auto-detected from SNMP manufacturer field
• Transport: API-SSL (8729, default) or plain API (8728, insecure)
• Security: Plain API blocked for cloud pollers, passwords require CLOAK_KEY env var

Key Files:

• lib/towerops/vault.ex - Cloak vault
• lib/towerops/ecto_types/encrypted_binary.ex - Encrypted field type
• lib/towerops/devices.ex - Config resolution, propagation
• priv/proto/agent.proto - MikrotikDevice protobuf messages

Credential Resolution: Each field resolves independently up the hierarchy.

MIB Name Resolution (Rust NIF)

Uses Rust NIF via Rustler for fast MIB resolution (replaces unreliable Erlang SNMP).

• NIF Module: lib/towerops_native.ex
• Rust: native/towerops_native/src/mib.rs (uses snmptranslate command)
• MIB Files: priv/mibs/ (570+ vendor and standard MIBs)
• Dependencies: brew install net-snmp (macOS), apt-get install libsnmp-dev snmp-mibs-downloader (Docker)
• Usage: Try ToweropsNative.resolve_oid/1 first, fallback to SnmpKit.resolve/1

Note: SnmpKit still used for SNMP protocol operations (get, walk, etc.).

Admin Features

User Impersonation

• Location: /admin/users (superuser only)
• Superusers CAN impersonate other superusers (feature, not bug)
• All events logged to audit_logs
• Stop link appears in user menu when active

GeoIP Database

• Import MaxMind GeoLite2-City for IP-based country/city detection (GDPR cookies)
• Local: make geoip-import DIR=~/Downloads/GeoLite2-City-CSV_20260127/
• Production: make geoip-import-prod DIR=... (requires TOWEROPS_KEY)
• API: POST /admin/api/geoip/import (superuser only)

Project-Specific Constraints

See AGENTS.md for the full set of coding constraints. Key reminders:

• Use mix precommit before committing (formats, compiles with warnings-as-errors, runs tests)
• Use :req (Req) for HTTP — never :httpoison, :tesla, :httpc
• Never use daisyUI — custom Tailwind components only
• Never put Ecto queries directly in LiveViews — use context modules
• Always run mix format after Elixir changes

API Documentation

• Available at /docs/api (Tailwind UI Protocol template)
• Update controller @doc comments and /docs/api template when changing endpoints
• Wrap examples in <%= raw(~S"""...""") %> to prevent HEEx parsing

Kubernetes Deployment

Prerequisites: cert-manager, Traefik, MetalLB, FluxCD, GitLab Agent, NFS Provisioner

Secrets (1Password, towerops namespace):

• gitlab-registry - Docker credentials
• towerops-secrets - RELEASE_COOKIE, SECRET_KEY_BASE, CLOAK_KEY
• towerops-db - PostgreSQL connection
• towerops-aws - AWS credentials

Create CLOAK_KEY:

# Generate and store in 1Password first
CLOAK_KEY=$(openssl rand -base64 32)

# Create secret
kubectl create secret generic towerops-secrets \
  --from-literal=RELEASE_COOKIE=$(openssl rand -base64 32) \
  --from-literal=SECRET_KEY_BASE=$(mix phx.gen.secret) \
  --from-literal=CLOAK_KEY="$CLOAK_KEY" \
  -n towerops

# Or patch existing
kubectl patch secret towerops-secrets -n towerops \
  --type='json' \
  -p="[{'op': 'add', 'path': '/data/CLOAK_KEY', 'value': '$(echo -n "$CLOAK_KEY" | base64)'}]"

# Restart pods
kubectl rollout restart deployment/towerops -n towerops

Important: Losing CLOAK_KEY makes encrypted data unrecoverable.

Deployment: FluxCD auto-applies k8s/ manifests. Manual: kubectl apply -k k8s/

Deployment

Branch-Based Deployment Strategy

Towerops uses branch-based deployment with Forgejo CI/CD:

Staging (Dokku):

• Push to main branch triggers automatic deploy to Dokku staging server
• Direct git push to Dokku (no Docker build)
• Buildpack-based deployment (detects Elixir/Phoenix)
• URL: staging.towerops.app (or configured Dokku domain)

Production (Kubernetes):

• Push to production branch triggers CI/CD pipeline
• Test Gates (must pass):
  • All ExUnit tests (mix test)
  • All e2e tests (Playwright)
• Only after tests pass:
  • Builds Docker image from k8s/Dockerfile
  • Pushes to container registry
  • Updates k8s/deployment.yaml with new image tag
  • FluxCD detects change and applies to cluster

Deploying Changes

CRITICAL GIT WORKFLOW RULE:

• ALWAYS push to main after commits (default behavior)
• NEVER push to production unless explicitly instructed by the user
• Production deployments must be manually authorized each time
• When in doubt, only push to main

To staging:

git push origin main
# CI automatically deploys to Dokku
# Watch: https://git.mcintire.me/graham/towerops-web/actions

To production (ONLY when explicitly requested):

# Merge main → production
git checkout production
git merge main
git push origin production

# Or fast-forward if no conflicts
git push origin main:production

# CI/CD Pipeline:
# 1. Runs all ExUnit tests (must pass)
# 2. Runs all e2e tests (must pass)
# 3. Builds Docker image
# 4. Pushes to container registry
# 5. Updates k8s/deployment.yaml with new image tag
# 6. FluxCD auto-applies changes (~5 minutes)
#
# Watch progress: https://git.mcintire.me/graham/towerops-web/actions

Direct Dokku deploy (bypass CI):

# Add Dokku remote (one-time)
git remote add dokku dokku@204.110.191.231:towerops

# Force push to deploy
git push dokku main:main --force

Manual production deploy:

# Apply k8s manifests directly
kubectl apply -k k8s/

# Or trigger rollout restart
kubectl rollout restart deployment/towerops -n towerops

Deployment Verification

Staging:

# Check Dokku logs
ssh dokku@204.110.191.231 logs towerops --tail 100

# Check app status
ssh dokku@204.110.191.231 ps:report towerops

Production:

# Check pod status
kubectl get pods -n towerops

# Check logs
kubectl logs -n towerops -l app=towerops --tail=100 -f

# Check deployment status
kubectl rollout status deployment/towerops -n towerops

Rollback

Staging (Dokku):

# Revert git commit, push to main
git revert <bad-commit>
git push origin main

# Or rebuild previous release
ssh dokku@204.110.191.231 ps:rebuild towerops <release-id>

Production (Kubernetes):

# Rollback deployment
kubectl rollout undo deployment/towerops -n towerops

# Or revert git commit on production branch
git checkout production
git revert <bad-commit>
git push origin production

Common Patterns

Pagination in LiveView

Use <.pagination> component from CoreComponents:

def handle_params(params, _url, socket) do
  page = params |> Map.get("page", "1") |> String.to_integer()
  per_page = 20

  all_items = MyContext.list_items(org_id)
  total_count = length(all_items)
  total_pages = ceil(total_count / per_page)
  page = max(1, min(page, max(1, total_pages)))

  offset = (page - 1) * per_page
  items = Enum.slice(all_items, offset, per_page)

  {:noreply, socket
   |> assign(:items, items)
   |> assign(:pagination, %{page: page, per_page: per_page,
                             total_count: total_count, total_pages: total_pages})}
end

Template:

<.pagination meta={@pagination} path={~p"/devices"} params={%{"tab" => @tab}} />

For large datasets (>10K), use database-level limit/offset instead of in-memory slicing.

Testing Notes

• Never use npm — esbuild is built into Phoenix
• Run cargo fmt before committing Rust changes
• Never open test coverage HTML files — read results in terminal
• See AGENTS.md for full test guidelines, SNMP mocking patterns, and LiveView test helpers

Dialyzer

• mix dialyzer - Run analysis (builds PLT first time)
• mix dialyzer --format dialyzer - Detailed error locations
• Never suppress valid warnings — fix root cause
• PLT files in priv/plts/ — don't commit

Changelog

Two changelog files to maintain:

CHANGELOG.txt (Technical/Internal)

After every code change, append to CHANGELOG.txt:

• Date (YYYY-MM-DD)
• Short description (e.g. "fix: update last_snmp_poll_at for agent-polled devices")
• Files changed and brief explanation
• Technical details about implementation

Keep reverse chronological (newest at top). Never remove entries.

priv/static/changelog.txt (User-Facing)

After significant changes, update priv/static/changelog.txt:

• Group changes by date (YYYY-MM-DD)
• Brief bullet points without code specifics
• Generic descriptions (no function names, modules, file paths, or language details)
• Focus on what changed for the user, not how it was implemented
• Follow existing pattern: "* Feature description" or "* Bug fix: brief description"

Example conversions:

• ❌ "fix: update last_snmp_poll_at in agent_channel.ex"

• ✅ "Poll time tracking improvements"

• ❌ "feat: add discover_wireless_sensors/1 to MikroTik vendor module"

• ✅ "Enhanced wireless monitoring for MikroTik devices"

When to update user-facing changelog:

• User-visible features or improvements
• Bug fixes affecting user experience
• New vendor/device support
• Performance improvements
• Security improvements
• Skip: Test-only changes, internal refactoring, documentation updates