towerops/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

CRITICAL: Read AGENTS.md First

Before starting any work in this repository, you MUST read AGENTS.md in the project root.

• AGENTS.md contains mandatory Phoenix/LiveView/Elixir coding guidelines AND project-specific patterns for this codebase
• Covers: Elixir/OTP conventions, Phoenix/LiveView best practices, browser navigation URL state, JS hook memory management, form handling patterns, test guidelines, quality/security checks
• These guidelines take precedence over general development practices when there's a conflict
• Always read AGENTS.md at the start of a new conversation or when resuming work

Project Overview

Towerops is a Phoenix 1.8 web application built with Elixir, using Ecto for database operations (PostgreSQL), LiveView for real-time interactions, and Tailwind CSS v4 for styling.

Data Model Relationships

User (Accounts) → owns/member of → Organization (Organizations)
                                    ├─ Site (Sites) → Device
                                    ├─ Device → Snmp.Device, MonitoringCheck, Alert
                                    └─ AgentToken → AgentAssignment → Device

Key Relationships:
- User can own/belong to multiple Organizations
- Organization has default_agent_token_id (optional)
- Device belongs to both Site and Organization (denormalized)
- Device can be assigned to one AgentToken via AgentAssignment
- Device has one Snmp.Device with Sensors and Interfaces
- Device has many MonitoringChecks (polling results) and Alerts
- AgentToken authenticates remote agents for local SNMP polling

Note: the equipment→device rename happened in migration 20260117190134. Older docs and comments may still say "equipment".

Note: Update this diagram when making data model changes.

Essential Commands

Setup and Development

• mix setup - Install dependencies, create/migrate database, build assets
• mix phx.server - Start Phoenix server (http://localhost:4000)
• iex -S mix phx.server - Start server with IEx shell

Testing and Quality

• mix test - Run all tests
• mix test --failed - Re-run failed tests
• mix test --cover - Run with coverage (target: 90% minimum)
• mix precommit - Run before committing: compiles with warnings as errors, formats, runs tests
• mix dialyzer - Static type analysis
• cd e2e && npm test - Run end-to-end tests (Playwright)

E2E Testing: When adding or modifying user-facing features, ALWAYS add corresponding e2e tests in e2e/tests/. E2E tests ensure the full user experience works across browsers (chromium, firefox, webkit). Tests should be defensive (use if (await element.isVisible()) checks) and handle edge cases like missing data or sudo verification redirects.

Database

• mix ecto.create/migrate/reset - Database operations
• mix ecto.gen.migration name_using_underscores - Generate migration

Assets

• mix assets.build/deploy - Build CSS/JS assets (auto-rebuilds on save in dev)

Architecture

Application Structure

Standard Phoenix conventions: business logic in lib/towerops/, web interface in lib/towerops_web/.

Key Configuration:

• Binary IDs: UUID primary keys by default (binary_id: true)
• Timestamps: :utc_datetime for all timestamps
• Web server: Bandit adapter
• HTTP client: :req library (Req module) - ONLY approved client
• Ecto repos: [Towerops.Repo]

Web Layer

All LiveViews get these imports via html_helpers/0:

• ToweropsWeb.CoreComponents - Core UI (<.button>, <.input>, <.form>)
• ToweropsWeb.Layouts, Phoenix.LiveView.JS, Gettext, verified routes (~p)

Rate Limiting

In-tree Towerops.RateLimit GenServer (ETS-backed fixed-window limiter, replaces former Hammer dep):

• Auth endpoints: 10 req/min per IP (/users/log-in, /users/register, TOTP)
• API v1: 1000 req/min per IP (/api/v1/*)
• Admin API: Not rate limited (superuser only)
• Returns 429 Too Many Requests with Retry-After header
• Disabled in test: config :towerops, :rate_limiting_enabled, false

Asset Pipeline

• Tailwind CSS v4: Uses @import "tailwindcss" in assets/css/app.css (no config file)
• esbuild: Bundles assets/js/app.js
• Import all vendor assets into app.js/app.css - no external src/href in layouts
• No inline <script> tags - use LiveView hooks

Custom Ecto Types

Avoid "primitive obsession" by using custom types in lib/towerops/ecto_types/:

1. IpAddress - IPv4/IPv6 validation, struct with version/tuple
2. MacAddress - MAC address normalization
3. EncryptedBinary / EncryptedMap - Cloak-backed encrypted fields
4. SnmpOid - OID normalization
5. JsonAny - permissive JSON column

Pattern: Implement Ecto.Type with type/0, cast/1, load/1, dump/1.

When to use:

• ✅ Domain concepts with validation rules (emails, IPs, phone numbers)
• ✅ Values needing normalization (case-insensitive emails, MAC addresses)
• ✅ Multiple representations (IP strings vs tuples)
• ❌ Simple strings, primitives, one-off validations

Migration: No database changes needed - column stays primitive, Ecto handles conversion.

Background Jobs (Oban)

PostgreSQL-backed with cluster-wide coordination. Oban Pro 1.7.0 is vendored at vendor/oban_pro/ (not fetched from hex) — see vendor/README.md for the update procedure.

Queues (concurrency scaled at runtime by OBAN_SCALE env var):

• default (10) — general tasks
• discovery (10) — SNMP discovery
• pollers (50, override via POLLER_CONCURRENCY) — SNMP polling (per-device)
• monitors (50) — health checks (per-device)
• checks (50) / check_executors (50) — monitoring checks pipeline
• notifications (25) — email/push fan-out
• weather (2) — HRRR / forecast fetchers
• maintenance (5) — periodic cleanup

Key Workers:

1. Self-Scheduling (per-device):

   • DeviceMonitorWorker — health checks (60s default)
   • DevicePollerWorker — SNMP data collection (60s default)
   • Auto-created/cancelled when device settings change

2. Oban Cron (cluster-wide): ~28 cron jobs covering insights, billing sync, vendor integrations (Preseem, Gaiia, NetBox, Sonar, Splynx, VISP, UISP, CnMaestro), agent latency probes, and cleanup. See the :crontab block in config/runtime.exs for the authoritative list and schedules.

Resilience: Oban Cron uses PostgreSQL locking; self-scheduling jobs are recovered every 10 minutes by JobHealthCheckWorker. Dashboard: /admin/oban (superuser), /dev/dashboard → Oban tab (dev)

SNMP Polling

Two mechanisms:

1. Discovery (Towerops.Snmp.Discovery) - One-time/manual, collects full device info
2. Polling (DevicePollerWorker) - Continuous (60s default), time-series data

Load Distribution: Staggered polling using hash-based offsets:

• offset = :erlang.phash2(device_id) |> rem(interval_seconds)
• Prevents thundering herd, stable across restarts
• Implementation: lib/towerops/workers/polling_offset.ex

MikroTik API Integration

RouterOS API access alongside SNMP.

Architecture:

• 3-tier credential cascade: Organization → Site → Device (like SNMP)
• Encryption: Passwords encrypted at rest (Cloak AES-256-GCM)
• Detection: Auto-detected from SNMP manufacturer field
• Transport: API-SSL (8729, default) or plain API (8728, insecure)
• Security: Plain API blocked for cloud pollers, passwords require CLOAK_KEY env var

Key Files:

• lib/towerops/vault.ex - Cloak vault
• lib/towerops/ecto_types/encrypted_binary.ex - Encrypted field type
• lib/towerops/devices.ex - Config resolution, propagation
• priv/proto/agent.proto - MikrotikDevice protobuf messages

Credential Resolution: Each field resolves independently up the hierarchy.

MIB Name Resolution (C NIF)

Pure C NIF calling libnetsnmp directly for fast in-process MIB resolution (replaces unreliable Erlang SNMP and an earlier Rust attempt).

• NIF source: c_src/towerops_nif.c (built via c_src/Makefile)
• Wrapper: lib/towerops_native.ex (loads priv/towerops_nif)
• MIB Files: priv/mibs/ (~560 vendor and standard MIBs)
• Dependencies: brew install net-snmp (macOS), apt-get install libsnmp-dev snmp-mibs-downloader (Docker)
• Usage: Try ToweropsNative.resolve_oid/1 first, fallback to SnmpKit.resolve/1
• Performance: ~1–20µs per OID (no shell-out)

Note: SnmpKit still used for SNMP protocol operations (get, walk, etc.).

LIDAR Elevation Catalog

Catalog-only system for accessing LIDAR-derived DEMs covering Texas. We do not mirror raster data — the DB stores tile metadata (URL + footprint geometry) and elevation values are streamed on-demand from public USGS 3DEP / TNRIS COGs via GDAL /vsicurl/ HTTP byte-range reads.

• Catalog: lib/towerops/lidar/catalog.ex, schemas in lib/towerops/lidar/
• Sources: lib/towerops/lidar/sources/three_dep.ex (primary), lib/towerops/lidar/sources/tnris.ex (fallback for areas not in 3DEP)
• Reader: lib/towerops/lidar/reader.ex shells to gdallocationinfo / gdal_translate against /vsicurl/<url> — only fetches the bytes needed.
• Sync: Towerops.Workers.LidarCatalogSyncWorker runs monthly via Oban cron
• Dependencies: brew install gdal postgis (macOS), apt-get install gdal-bin postgis (Docker — gdal-bin is in k8s/Dockerfile)
• PostGIS: enabled by migration 20260504123340_enable_postgis, used for spatial indexes on tile footprints (GiST) and ST_Contains dedup checks
• Geometry types: registered via lib/towerops/postgrex_types.ex, wired into the Repo via config :towerops, Towerops.Repo, types: ...

Admin Features

User Impersonation

• Location: /admin/users (superuser only)
• Superusers CAN impersonate other superusers (feature, not bug)
• All events logged to audit_logs
• Stop link appears in user menu when active

GeoIP Database

• Import MaxMind GeoLite2-City for IP-based country/city detection (GDPR cookies)
• Local: make geoip-import DIR=~/Downloads/GeoLite2-City-CSV_20260127/
• Production: make geoip-import-prod DIR=... (requires TOWEROPS_KEY)
• API: POST /admin/api/geoip/import (superuser only)

Project-Specific Constraints

See AGENTS.md for the full set of coding constraints. Key reminders:

• Use mix precommit before committing (formats, compiles with warnings-as-errors, runs tests)
• Use :req (Req) for HTTP — never :httpoison, :tesla, :httpc
• Never use daisyUI — custom Tailwind components only
• Never put Ecto queries directly in LiveViews — use context modules
• Always run mix format after Elixir changes

API Documentation

• Available at /docs/api (Tailwind UI Protocol template)
• Update controller @doc comments and /docs/api template when changing endpoints
• Wrap examples in <%= raw(~S"""...""") %> to prevent HEEx parsing

Kubernetes Deployment

Prerequisites: cert-manager, Traefik, MetalLB, ArgoCD (with argocd-image-updater), NFS Provisioner

Secrets (1Password, towerops namespace):

• forgejo-registry - Docker registry credentials (image pull)
• towerops-secrets - RELEASE_COOKIE, SECRET_KEY_BASE, CLOAK_KEY
• towerops-db - PostgreSQL connection
• towerops-aws - AWS credentials
• towerops-billing - Stripe keys (STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET, STRIPE_PRICE_ID, STRIPE_METER_ID)
• towerops-redis - Redis connection (envFrom)

Create CLOAK_KEY:

# Generate and store in 1Password first
CLOAK_KEY=$(openssl rand -base64 32)

# Create secret
kubectl create secret generic towerops-secrets \
  --from-literal=RELEASE_COOKIE=$(openssl rand -base64 32) \
  --from-literal=SECRET_KEY_BASE=$(mix phx.gen.secret) \
  --from-literal=CLOAK_KEY="$CLOAK_KEY" \
  -n towerops

# Or patch existing
kubectl patch secret towerops-secrets -n towerops \
  --type='json' \
  -p="[{'op': 'add', 'path': '/data/CLOAK_KEY', 'value': '$(echo -n "$CLOAK_KEY" | base64)'}]"

# Restart pods
kubectl rollout restart deployment/towerops -n towerops

Important: Losing CLOAK_KEY makes encrypted data unrecoverable.

Deployment: ArgoCD reconciles k8s/ manifests automatically. Manual: kubectl apply -k k8s/

Deployment

Deployment Strategy

Towerops uses Forgejo CI/CD with two trigger paths (see .forgejo/workflows/):

Staging (Dokku) — pull requests:

• Opening / updating a PR triggers staging.yaml
• The PR branch is force-pushed to Dokku via SSH (no Docker build, buildpack-based)
• URL: staging.towerops.app (or configured Dokku domain)

Production (Kubernetes) — push to main:

• Push to main triggers production.yaml
• Test gate: ExUnit suite must pass before any image is built
• After tests pass:
  • Builds Docker image from k8s/Dockerfile — FROM codeberg.org/gmcintire/towerops-base:latest
  • Pushes to codeberg.org/gmcintire/towerops (tagged main-<ts>-<sha> and production)
  • argocd-image-updater picks up the new tag (~2 min), writes it to the ArgoCD Application's spec.source.kustomize.images, and ArgoCD rolls the Deployment. No commit to k8s/deployment.yaml.

There is no production branch — main IS production. Pushing to main will ship to prod once tests pass.

Base image (k8s/Dockerfile.base)

The slow runtime apt installs (gdal-bin, snmp, libsnmp40, locales, BEAM runtime libs) live in a prebuilt base image — codeberg.org/gmcintire/towerops-base:latest — so day-to-day app builds skip ~500 MB of GIS dep installation.

• Source: k8s/Dockerfile.base
• Built by: .forgejo/workflows/build-base.yaml
• Triggers: changes to k8s/Dockerfile.base or the workflow itself, weekly cron (Sundays 06:00 UTC for Debian security updates), or workflow_dispatch.
• Cache busting: weekly cron passes CACHE_BUST=<ISO year+week> so apt layers re-execute on a week boundary; same-week reruns reuse the layer cache.

Bumping the Debian release: edit ARG DEBIAN_VERSION= in k8s/Dockerfile.base and push. The base workflow rebuilds and re-tags :latest; the next app push picks it up automatically.

If the registry GCs base layers and the next app build fails with "could not fetch content descriptor … not found", trigger Build base image via workflow_dispatch (or push any change to Dockerfile.base) to re-upload the layers.

Deploying Changes

To deploy:

git push origin main
# CI runs ExUnit → builds image → pushes → argocd-image-updater rolls out
# Watch: https://git.mcintire.me/graham/towerops-web/actions

Manual production deploy (rare, e.g. config-only change to k8s/ that image-updater can't pick up):

kubectl apply -k k8s/
# Or trigger rollout restart
kubectl rollout restart deployment/towerops -n towerops

Direct Dokku deploy (bypass CI, e.g. testing without a PR):

# Add Dokku remote (one-time)
git remote add dokku dokku@204.110.191.231:towerops

# Force push to deploy
git push dokku main:main --force

Deployment Verification

Staging:

# Check Dokku logs
ssh dokku@204.110.191.231 logs towerops --tail 100

# Check app status
ssh dokku@204.110.191.231 ps:report towerops

Production:

# Check pod status
kubectl get pods -n towerops

# Check logs
kubectl logs -n towerops -l app=towerops --tail=100 -f

# Check deployment status
kubectl rollout status deployment/towerops -n towerops

Rollback

Staging (Dokku):

# Revert git commit, push to main
git revert <bad-commit>
git push origin main

# Or rebuild previous release
ssh dokku@204.110.191.231 ps:rebuild towerops <release-id>

Production (Kubernetes):

# Rollback deployment
kubectl rollout undo deployment/towerops -n towerops

# Or revert git commit on main (will trigger a new production build)
git revert <bad-commit>
git push origin main

Common Patterns

Pagination in LiveView

Use <.pagination> component from CoreComponents:

def handle_params(params, _url, socket) do
  page = params |> Map.get("page", "1") |> String.to_integer()
  per_page = 20

  all_items = MyContext.list_items(org_id)
  total_count = length(all_items)
  total_pages = ceil(total_count / per_page)
  page = max(1, min(page, max(1, total_pages)))

  offset = (page - 1) * per_page
  items = Enum.slice(all_items, offset, per_page)

  {:noreply, socket
   |> assign(:items, items)
   |> assign(:pagination, %{page: page, per_page: per_page,
                             total_count: total_count, total_pages: total_pages})}
end

Template:

<.pagination meta={@pagination} path={~p"/devices"} params={%{"tab" => @tab}} />

For large datasets (>10K), use database-level limit/offset instead of in-memory slicing.

Testing Notes

• Never use npm — esbuild is built into Phoenix
• Run cargo fmt before committing Rust changes
• Never open test coverage HTML files — read results in terminal
• See AGENTS.md for full test guidelines, SNMP mocking patterns, and LiveView test helpers

Dialyzer

• mix dialyzer - Run analysis (builds PLT first time)
• mix dialyzer --format dialyzer - Detailed error locations
• Never suppress valid warnings — fix root cause
• PLT files in priv/plts/ — don't commit

Changelog

Two changelog files to maintain:

CHANGELOG.txt (Technical/Internal)

After every code change, append to CHANGELOG.txt:

• Date (YYYY-MM-DD)
• Short description (e.g. "fix: update last_snmp_poll_at for agent-polled devices")
• Files changed and brief explanation
• Technical details about implementation

Keep reverse chronological (newest at top). Never remove entries.

priv/static/changelog.txt (User-Facing)

After significant changes, update priv/static/changelog.txt:

• Group changes by date (YYYY-MM-DD)
• Brief bullet points without code specifics
• Generic descriptions (no function names, modules, file paths, or language details)
• Focus on what changed for the user, not how it was implemented
• Follow existing pattern: "* Feature description" or "* Bug fix: brief description"

Example conversions:

• ❌ "fix: update last_snmp_poll_at in agent_channel.ex"

• ✅ "Poll time tracking improvements"

• ❌ "feat: add discover_wireless_sensors/1 to MikroTik vendor module"

• ✅ "Enhanced wireless monitoring for MikroTik devices"

When to update user-facing changelog:

• User-visible features or improvements
• Bug fixes affecting user experience
• New vendor/device support
• Performance improvements
• Security improvements
• Skip: Test-only changes, internal refactoring, documentation updates