towerops/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## CRITICAL: Read AGENTS.md First

**Before starting any work in this repository**, you MUST read `AGENTS.md` in the project root.

- `AGENTS.md` contains mandatory Phoenix/LiveView/Elixir coding guidelines AND project-specific patterns for this codebase
- Covers: Elixir/OTP conventions, Phoenix/LiveView best practices, browser navigation URL state, JS hook memory management, form handling patterns, test guidelines, quality/security checks
- These guidelines take precedence over general development practices when there's a conflict
- Always read `AGENTS.md` at the start of a new conversation or when resuming work

## Project Overview

Towerops is a Phoenix 1.8 web application built with Elixir, using Ecto for database operations (PostgreSQL), LiveView for real-time interactions, and Tailwind CSS v4 for styling.

### Data Model Relationships

```
User (Accounts) → owns/member of → Organization (Organizations)
                                    ├─ Site (Sites) → Device
                                    ├─ Device → Snmp.Device, MonitoringCheck, Alert
                                    └─ AgentToken → AgentAssignment → Device

Key Relationships:
- User can own/belong to multiple Organizations
- Organization has default_agent_token_id (optional)
- Device belongs to both Site and Organization (denormalized)
- Device can be assigned to one AgentToken via AgentAssignment
- Device has one Snmp.Device with Sensors and Interfaces
- Device has many MonitoringChecks (polling results) and Alerts
- AgentToken authenticates remote agents for local SNMP polling
```

Note: the equipment→device rename happened in migration `20260117190134`. Older
docs and comments may still say "equipment".

**Note**: Update this diagram when making data model changes.

## Essential Commands

### Setup and Development
- `mix setup` - Install dependencies, create/migrate database, build assets
- `mix phx.server` - Start Phoenix server (http://localhost:4000)
- `iex -S mix phx.server` - Start server with IEx shell

### Testing and Quality
- `mix test` - Run all tests
- `mix test --failed` - Re-run failed tests
- `mix test --cover` - Run with coverage (target: 90% minimum)
- `mix precommit` - **Run before committing**: compiles with warnings as errors, formats, runs tests
- `mix dialyzer` - Static type analysis
- `cd e2e && npm test` - Run end-to-end tests (Playwright)

**E2E Testing**: When adding or modifying user-facing features, ALWAYS add corresponding e2e tests in `e2e/tests/`. E2E tests ensure the full user experience works across browsers (chromium, firefox, webkit). Tests should be defensive (use `if (await element.isVisible())` checks) and handle edge cases like missing data or sudo verification redirects.

### Database
- `mix ecto.create/migrate/reset` - Database operations
- `mix ecto.gen.migration name_using_underscores` - Generate migration

### Assets
- `mix assets.build/deploy` - Build CSS/JS assets (auto-rebuilds on save in dev)

## Architecture

### Application Structure

Standard Phoenix conventions: business logic in `lib/towerops/`, web interface in `lib/towerops_web/`.

**Key Configuration**:
- **Binary IDs**: UUID primary keys by default (`binary_id: true`)
- **Timestamps**: `:utc_datetime` for all timestamps
- **Web server**: Bandit adapter
- **HTTP client**: `:req` library (Req module) - ONLY approved client
- **Ecto repos**: `[Towerops.Repo]`

### Web Layer

All LiveViews get these imports via `html_helpers/0`:
- `ToweropsWeb.CoreComponents` - Core UI (`<.button>`, `<.input>`, `<.form>`)
- `ToweropsWeb.Layouts`, `Phoenix.LiveView.JS`, Gettext, verified routes (`~p`)

### Rate Limiting

In-tree `Towerops.RateLimit` GenServer (ETS-backed fixed-window limiter, replaces former Hammer dep):
- **Auth endpoints**: 10 req/min per IP (`/users/log-in`, `/users/register`, TOTP)
- **API v1**: 1000 req/min per IP (`/api/v1/*`)
- **Admin API**: Not rate limited (superuser only)
- Returns `429 Too Many Requests` with `Retry-After` header
- Disabled in test: `config :towerops, :rate_limiting_enabled, false`

### Asset Pipeline

- **Tailwind CSS v4**: Uses `@import "tailwindcss"` in `assets/css/app.css` (no config file)
- **esbuild**: Bundles `assets/js/app.js`
- Import all vendor assets into app.js/app.css - no external src/href in layouts
- No inline `<script>` tags - use LiveView hooks

### Custom Ecto Types

Avoid "primitive obsession" by using custom types in `lib/towerops/ecto_types/`:

1. **`IpAddress`** - IPv4/IPv6 validation, struct with version/tuple
2. **`MacAddress`** - MAC address normalization
3. **`EncryptedBinary`** / **`EncryptedMap`** - Cloak-backed encrypted fields
4. **`SnmpOid`** - OID normalization
5. **`JsonAny`** - permissive JSON column

**Pattern**: Implement `Ecto.Type` with `type/0`, `cast/1`, `load/1`, `dump/1`.

**When to use**:
- ✅ Domain concepts with validation rules (emails, IPs, phone numbers)
- ✅ Values needing normalization (case-insensitive emails, MAC addresses)
- ✅ Multiple representations (IP strings vs tuples)
- ❌ Simple strings, primitives, one-off validations

**Migration**: No database changes needed - column stays primitive, Ecto handles conversion.

### Background Jobs (Oban)

PostgreSQL-backed with cluster-wide coordination. Oban Pro 1.7.0 is **vendored**
at `vendor/oban_pro/` (not fetched from hex) — see `vendor/README.md` for the
update procedure.

**Queues** (concurrency scaled at runtime by `OBAN_SCALE` env var):
- `default` (10) — general tasks
- `discovery` (10) — SNMP discovery
- `pollers` (50, override via `POLLER_CONCURRENCY`) — SNMP polling (per-device)
- `monitors` (50) — health checks (per-device)
- `checks` (50) / `check_executors` (50) — monitoring checks pipeline
- `notifications` (25) — email/push fan-out
- `weather` (2) — HRRR / forecast fetchers
- `maintenance` (5) — periodic cleanup

**Key Workers**:
1. **Self-Scheduling** (per-device):
   - `DeviceMonitorWorker` — health checks (60s default)
   - `DevicePollerWorker` — SNMP data collection (60s default)
   - Auto-created/cancelled when device settings change

2. **Oban Cron** (cluster-wide): ~28 cron jobs covering insights, billing sync,
   vendor integrations (Preseem, Gaiia, NetBox, Sonar, Splynx, VISP, UISP,
   CnMaestro), agent latency probes, and cleanup. See the `:crontab` block in
   `config/runtime.exs` for the authoritative list and schedules.

**Resilience**: Oban Cron uses PostgreSQL locking; self-scheduling jobs are
recovered every 10 minutes by `JobHealthCheckWorker`.
**Dashboard**: `/admin/oban` (superuser), `/dev/dashboard` → Oban tab (dev)

### SNMP Polling

**Two mechanisms**:
1. **Discovery** (`Towerops.Snmp.Discovery`) - One-time/manual, collects full device info
2. **Polling** (`DevicePollerWorker`) - Continuous (60s default), time-series data

**Load Distribution**: Staggered polling using hash-based offsets:
- `offset = :erlang.phash2(device_id) |> rem(interval_seconds)`
- Prevents thundering herd, stable across restarts
- Implementation: `lib/towerops/workers/polling_offset.ex`

### MikroTik API Integration

RouterOS API access alongside SNMP.

**Architecture**:
- **3-tier credential cascade**: Organization → Site → Device (like SNMP)
- **Encryption**: Passwords encrypted at rest (Cloak AES-256-GCM)
- **Detection**: Auto-detected from SNMP manufacturer field
- **Transport**: API-SSL (8729, default) or plain API (8728, insecure)
- **Security**: Plain API blocked for cloud pollers, passwords require `CLOAK_KEY` env var

**Key Files**:
- `lib/towerops/vault.ex` - Cloak vault
- `lib/towerops/ecto_types/encrypted_binary.ex` - Encrypted field type
- `lib/towerops/devices.ex` - Config resolution, propagation
- `priv/proto/agent.proto` - MikrotikDevice protobuf messages

**Credential Resolution**: Each field resolves independently up the hierarchy.

### MIB Name Resolution (C NIF)

Pure C NIF calling libnetsnmp directly for fast in-process MIB resolution
(replaces unreliable Erlang SNMP and an earlier Rust attempt).

- **NIF source**: `c_src/towerops_nif.c` (built via `c_src/Makefile`)
- **Wrapper**: `lib/towerops_native.ex` (loads `priv/towerops_nif`)
- **MIB Files**: `priv/mibs/` (~560 vendor and standard MIBs)
- **Dependencies**: `brew install net-snmp` (macOS), `apt-get install libsnmp-dev snmp-mibs-downloader` (Docker)
- **Usage**: Try `ToweropsNative.resolve_oid/1` first, fallback to `SnmpKit.resolve/1`
- **Performance**: ~1–20µs per OID (no shell-out)

Note: SnmpKit still used for SNMP protocol operations (get, walk, etc.).

### LIDAR Elevation Catalog

Catalog-only system for accessing LIDAR-derived DEMs covering Texas. We do
not mirror raster data — the DB stores tile metadata (URL + footprint
geometry) and elevation values are streamed on-demand from public USGS
3DEP / TNRIS COGs via GDAL `/vsicurl/` HTTP byte-range reads.

- **Catalog**: `lib/towerops/lidar/catalog.ex`, schemas in `lib/towerops/lidar/`
- **Sources**: `lib/towerops/lidar/sources/three_dep.ex` (primary),
  `lib/towerops/lidar/sources/tnris.ex` (fallback for areas not in 3DEP)
- **Reader**: `lib/towerops/lidar/reader.ex` shells to `gdallocationinfo` /
  `gdal_translate` against `/vsicurl/<url>` — only fetches the bytes needed.
- **Sync**: `Towerops.Workers.LidarCatalogSyncWorker` runs monthly via Oban cron
- **Dependencies**: `brew install gdal postgis` (macOS), `apt-get install
  gdal-bin postgis` (Docker — `gdal-bin` is in `k8s/Dockerfile`)
- **PostGIS**: enabled by migration `20260504123340_enable_postgis`, used for
  spatial indexes on tile footprints (GiST) and `ST_Contains` dedup checks
- **Geometry types**: registered via `lib/towerops/postgrex_types.ex`, wired
  into the Repo via `config :towerops, Towerops.Repo, types: ...`

## Admin Features

### User Impersonation
- Location: `/admin/users` (superuser only)
- Superusers CAN impersonate other superusers (feature, not bug)
- All events logged to audit_logs
- Stop link appears in user menu when active

### GeoIP Database
- Import MaxMind GeoLite2-City for IP-based country/city detection (GDPR cookies)
- Local: `make geoip-import DIR=~/Downloads/GeoLite2-City-CSV_20260127/`
- Production: `make geoip-import-prod DIR=...` (requires TOWEROPS_KEY)
- API: `POST /admin/api/geoip/import` (superuser only)

## Project-Specific Constraints

See `AGENTS.md` for the full set of coding constraints. Key reminders:
- Use `mix precommit` before committing (formats, compiles with warnings-as-errors, runs tests)
- Use `:req` (Req) for HTTP — never `:httpoison`, `:tesla`, `:httpc`
- Never use `daisyUI` — custom Tailwind components only
- Never put Ecto queries directly in LiveViews — use context modules
- Always run `mix format` after Elixir changes

### API Documentation

- Available at `/docs/api` (Tailwind UI Protocol template)
- Update controller `@doc` comments and `/docs/api` template when changing endpoints
- Wrap examples in `<%= raw(~S"""...""") %>` to prevent HEEx parsing

## Kubernetes Deployment

**Prerequisites**: cert-manager, Traefik, MetalLB, ArgoCD (with `argocd-image-updater`), NFS Provisioner

**Secrets** (1Password, `towerops` namespace):
- `forgejo-registry` - Docker registry credentials (image pull)
- `towerops-secrets` - RELEASE_COOKIE, SECRET_KEY_BASE, CLOAK_KEY
- `towerops-db` - PostgreSQL connection
- `towerops-aws` - AWS credentials
- `towerops-billing` - Stripe keys (STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET, STRIPE_PRICE_ID, STRIPE_METER_ID)
- `towerops-redis` - Redis connection (envFrom)

**Create CLOAK_KEY**:
```bash
# Generate and store in 1Password first
CLOAK_KEY=$(openssl rand -base64 32)

# Create secret
kubectl create secret generic towerops-secrets \
  --from-literal=RELEASE_COOKIE=$(openssl rand -base64 32) \
  --from-literal=SECRET_KEY_BASE=$(mix phx.gen.secret) \
  --from-literal=CLOAK_KEY="$CLOAK_KEY" \
  -n towerops

# Or patch existing
kubectl patch secret towerops-secrets -n towerops \
  --type='json' \
  -p="[{'op': 'add', 'path': '/data/CLOAK_KEY', 'value': '$(echo -n "$CLOAK_KEY" | base64)'}]"

# Restart pods
kubectl rollout restart deployment/towerops -n towerops
```

**Important**: Losing CLOAK_KEY makes encrypted data unrecoverable.

**Deployment**: ArgoCD reconciles `k8s/` manifests automatically. Manual: `kubectl apply -k k8s/`

## Deployment

### Deployment Strategy

Towerops uses Forgejo CI/CD with two trigger paths (see `.forgejo/workflows/`):

**Staging (Dokku) — pull requests:**
- Opening / updating a PR triggers `staging.yaml`
- The PR branch is force-pushed to Dokku via SSH (no Docker build, buildpack-based)
- URL: staging.towerops.app (or configured Dokku domain)

**Production (Kubernetes) — push to `main`:**
- Push to `main` triggers `production.yaml`
- **Test gate:** ExUnit suite must pass before any image is built
- After tests pass:
  - Builds Docker image from `k8s/Dockerfile` — `FROM codeberg.org/gmcintire/towerops-base:latest`
  - Pushes to `codeberg.org/gmcintire/towerops` (tagged `main-<ts>-<sha>` and `production`)
  - `argocd-image-updater` picks up the new tag (~2 min), writes it to the
    ArgoCD Application's `spec.source.kustomize.images`, and ArgoCD rolls the
    Deployment. No commit to `k8s/deployment.yaml`.

There is no `production` branch — `main` IS production. Pushing to `main` will
ship to prod once tests pass.

### Base image (`k8s/Dockerfile.base`)

The slow runtime apt installs (`gdal-bin`, `snmp`, `libsnmp40`, `locales`,
BEAM runtime libs) live in a prebuilt base image —
`codeberg.org/gmcintire/towerops-base:latest` — so day-to-day app builds
skip ~500 MB of GIS dep installation.

- **Source**: `k8s/Dockerfile.base`
- **Built by**: `.forgejo/workflows/build-base.yaml`
- **Triggers**: changes to `k8s/Dockerfile.base` or the workflow itself,
  weekly cron (Sundays 06:00 UTC for Debian security updates),
  or `workflow_dispatch`.
- **Cache busting**: weekly cron passes `CACHE_BUST=<ISO year+week>` so
  apt layers re-execute on a week boundary; same-week reruns reuse the
  layer cache.

Bumping the Debian release: edit `ARG DEBIAN_VERSION=` in
`k8s/Dockerfile.base` and push. The base workflow rebuilds and re-tags
`:latest`; the next app push picks it up automatically.

If the registry GCs base layers and the next app build fails with
"could not fetch content descriptor … not found", trigger
`Build base image` via `workflow_dispatch` (or push any change to
`Dockerfile.base`) to re-upload the layers.

### Deploying Changes

**To deploy:**
```bash
git push origin main
# CI runs ExUnit → builds image → pushes → argocd-image-updater rolls out
# Watch: https://git.mcintire.me/graham/towerops-web/actions
```

**Manual production deploy (rare, e.g. config-only change to k8s/ that image-updater can't pick up):**
```bash
kubectl apply -k k8s/
# Or trigger rollout restart
kubectl rollout restart deployment/towerops -n towerops
```

**Direct Dokku deploy (bypass CI, e.g. testing without a PR):**
```bash
# Add Dokku remote (one-time)
git remote add dokku dokku@204.110.191.231:towerops

# Force push to deploy
git push dokku main:main --force
```

### Deployment Verification

**Staging:**
```bash
# Check Dokku logs
ssh dokku@204.110.191.231 logs towerops --tail 100

# Check app status
ssh dokku@204.110.191.231 ps:report towerops
```

**Production:**
```bash
# Check pod status
kubectl get pods -n towerops

# Check logs
kubectl logs -n towerops -l app=towerops --tail=100 -f

# Check deployment status
kubectl rollout status deployment/towerops -n towerops
```

### Rollback

**Staging (Dokku):**
```bash
# Revert git commit, push to main
git revert <bad-commit>
git push origin main

# Or rebuild previous release
ssh dokku@204.110.191.231 ps:rebuild towerops <release-id>
```

**Production (Kubernetes):**
```bash
# Rollback deployment
kubectl rollout undo deployment/towerops -n towerops

# Or revert git commit on main (will trigger a new production build)
git revert <bad-commit>
git push origin main
```

## Common Patterns

### Pagination in LiveView

Use `<.pagination>` component from `CoreComponents`:

```elixir
def handle_params(params, _url, socket) do
  page = params |> Map.get("page", "1") |> String.to_integer()
  per_page = 20

  all_items = MyContext.list_items(org_id)
  total_count = length(all_items)
  total_pages = ceil(total_count / per_page)
  page = max(1, min(page, max(1, total_pages)))

  offset = (page - 1) * per_page
  items = Enum.slice(all_items, offset, per_page)

  {:noreply, socket
   |> assign(:items, items)
   |> assign(:pagination, %{page: page, per_page: per_page,
                             total_count: total_count, total_pages: total_pages})}
end
```

Template:
```heex
<.pagination meta={@pagination} path={~p"/devices"} params={%{"tab" => @tab}} />
```

For large datasets (>10K), use database-level `limit/offset` instead of in-memory slicing.

## Testing Notes

- Never use npm — esbuild is built into Phoenix
- Run `cargo fmt` before committing Rust changes
- Never open test coverage HTML files — read results in terminal
- See `AGENTS.md` for full test guidelines, SNMP mocking patterns, and LiveView test helpers

## Dialyzer

- `mix dialyzer` - Run analysis (builds PLT first time)
- `mix dialyzer --format dialyzer` - Detailed error locations
- Never suppress valid warnings — fix root cause
- PLT files in `priv/plts/` — don't commit

## Changelog

**Two changelog files to maintain:**

### CHANGELOG.txt (Technical/Internal)
After every code change, append to `CHANGELOG.txt`:
- Date (YYYY-MM-DD)
- Short description (e.g. "fix: update last_snmp_poll_at for agent-polled devices")
- Files changed and brief explanation
- Technical details about implementation

Keep reverse chronological (newest at top). Never remove entries.

### priv/static/changelog.txt (User-Facing)
After significant changes, update `priv/static/changelog.txt`:
- Group changes by date (YYYY-MM-DD)
- Brief bullet points without code specifics
- Generic descriptions (no function names, modules, file paths, or language details)
- Focus on what changed for the user, not how it was implemented
- Follow existing pattern: "* Feature description" or "* Bug fix: brief description"

**Example conversions:**
- ❌ "fix: update last_snmp_poll_at in agent_channel.ex"
- ✅ "Poll time tracking improvements"

- ❌ "feat: add discover_wireless_sensors/1 to MikroTik vendor module"
- ✅ "Enhanced wireless monitoring for MikroTik devices"

**When to update user-facing changelog:**
- User-visible features or improvements
- Bug fixes affecting user experience
- New vendor/device support
- Performance improvements
- Security improvements
- Skip: Test-only changes, internal refactoring, documentation updates