towerops/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## CRITICAL: Read AGENTS.md First

**Before starting any work in this repository**, you MUST read `AGENTS.md` in the project root.

- `AGENTS.md` contains mandatory Phoenix/LiveView/Elixir coding guidelines AND project-specific patterns for this codebase
- Covers: Elixir/OTP conventions, Phoenix/LiveView best practices, browser navigation URL state, JS hook memory management, form handling patterns, test guidelines, quality/security checks
- These guidelines take precedence over general development practices when there's a conflict
- Always read `AGENTS.md` at the start of a new conversation or when resuming work

## Project Overview

Towerops is a Phoenix 1.8 web application built with Elixir, using Ecto for database operations (PostgreSQL), LiveView for real-time interactions, and Tailwind CSS v4 for styling.

### Data Model Relationships

```
User (Accounts) → owns/member of → Organization (Organizations)
                                    ├─ Site (Sites) → Equipment
                                    ├─ Equipment → SNMPDevice, MonitoringCheck, Alert
                                    └─ AgentToken → AgentAssignment → Equipment

Key Relationships:
- User can own/belong to multiple Organizations
- Organization has default_agent_token_id (optional)
- Equipment belongs to both Site and Organization (denormalized)
- Equipment can be assigned to one AgentToken via AgentAssignment
- Equipment has one SNMPDevice with Sensors and Interfaces
- Equipment has many MonitoringChecks (polling results) and Alerts
- AgentToken authenticates remote agents for local SNMP polling
```

**Note**: Update this diagram when making data model changes.

## Essential Commands

### Setup and Development
- `mix setup` - Install dependencies, create/migrate database, build assets
- `mix phx.server` - Start Phoenix server (http://localhost:4000)
- `iex -S mix phx.server` - Start server with IEx shell

### Testing and Quality
- `mix test` - Run all tests
- `mix test --failed` - Re-run failed tests
- `mix test --cover` - Run with coverage (target: 90% minimum)
- `mix precommit` - **Run before committing**: compiles with warnings as errors, formats, runs tests
- `mix dialyzer` - Static type analysis
- `cd e2e && npm test` - Run end-to-end tests (Playwright)

**E2E Testing**: When adding or modifying user-facing features, ALWAYS add corresponding e2e tests in `e2e/tests/`. E2E tests ensure the full user experience works across browsers (chromium, firefox, webkit). Tests should be defensive (use `if (await element.isVisible())` checks) and handle edge cases like missing data or sudo verification redirects.

### Database
- `mix ecto.create/migrate/reset` - Database operations
- `mix ecto.gen.migration name_using_underscores` - Generate migration

### Assets
- `mix assets.build/deploy` - Build CSS/JS assets (auto-rebuilds on save in dev)

## Architecture

### Application Structure

Standard Phoenix conventions: business logic in `lib/towerops/`, web interface in `lib/towerops_web/`.

**Key Configuration**:
- **Binary IDs**: UUID primary keys by default (`binary_id: true`)
- **Timestamps**: `:utc_datetime` for all timestamps
- **Web server**: Bandit adapter
- **HTTP client**: `:req` library (Req module) - ONLY approved client
- **Ecto repos**: `[Towerops.Repo]`

### Web Layer

All LiveViews get these imports via `html_helpers/0`:
- `ToweropsWeb.CoreComponents` - Core UI (`<.button>`, `<.input>`, `<.form>`)
- `ToweropsWeb.Layouts`, `Phoenix.LiveView.JS`, Gettext, verified routes (`~p`)

### Rate Limiting

Uses Hammer (ETS-backed):
- **Auth endpoints**: 10 req/min per IP (`/users/log-in`, `/users/register`, TOTP)
- **API v1**: 1000 req/min per IP (`/api/v1/*`)
- **Admin API**: Not rate limited (superuser only)
- Returns `429 Too Many Requests` with `Retry-After` header
- Disabled in test: `config :towerops, :rate_limiting_enabled, false`

### Asset Pipeline

- **Tailwind CSS v4**: Uses `@import "tailwindcss"` in `assets/css/app.css` (no config file)
- **esbuild**: Bundles `assets/js/app.js`
- Import all vendor assets into app.js/app.css - no external src/href in layouts
- No inline `<script>` tags - use LiveView hooks

### Custom Ecto Types

Avoid "primitive obsession" by using custom types:

**Available Types**:
1. **`Towerops.EctoTypes.IpAddress`** - IPv4/IPv6 validation, struct with version/tuple
2. **`Towerops.EctoTypes.EmailAddress`** - Normalized emails _(Planned)_
3. **`Towerops.EctoTypes.MacAddress`** - MAC address normalization _(Planned)_

**Pattern**: Implement `Ecto.Type` with `type/0`, `cast/1`, `load/1`, `dump/1`.

**When to use**:
- ✅ Domain concepts with validation rules (emails, IPs, phone numbers)
- ✅ Values needing normalization (case-insensitive emails, MAC addresses)
- ✅ Multiple representations (IP strings vs tuples)
- ❌ Simple strings, primitives, one-off validations

**Migration**: No database changes needed - column stays primitive, Ecto handles conversion.

### Background Jobs (Oban)

PostgreSQL-backed with cluster-wide coordination.

**Queues**:
- `default` (10) - General tasks
- `discovery` (10) - SNMP discovery
- `pollers` (50) - SNMP polling (per-device)
- `monitors` (50) - Health checks (per-device)
- `maintenance` (5) - Periodic cleanup

**Key Workers**:
1. **Self-Scheduling** (per-device):
   - `DeviceMonitorWorker` - Health checks (60s default)
   - `DevicePollerWorker` - SNMP data collection (60s default)
   - Auto-created/cancelled when device settings change

2. **Oban Cron** (cluster-wide):
   - `NeighborCleanupWorker` - Hourly stale data cleanup
   - `StaleAgentWorker` - Detect agents offline 10+ minutes (every minute)
   - `AgentLatencyEvaluator` - Latency-based reassignment (every 5 minutes)
   - `JobHealthCheckWorker` - Recover missing jobs (every 10 minutes)

**Resilience**: Oban Cron uses PostgreSQL locking, self-scheduling recovered every 10 minutes.
**Dashboard**: `/admin/oban` (superuser), `/dev/dashboard` → Oban tab (dev)

### SNMP Polling

**Two mechanisms**:
1. **Discovery** (`Towerops.Snmp.Discovery`) - One-time/manual, collects full device info
2. **Polling** (`DevicePollerWorker`) - Continuous (60s default), time-series data

**Load Distribution**: Staggered polling using hash-based offsets:
- `offset = :erlang.phash2(device_id) |> rem(interval_seconds)`
- Prevents thundering herd, stable across restarts
- Implementation: `lib/towerops/workers/polling_offset.ex`

### MikroTik API Integration

RouterOS API access alongside SNMP.

**Architecture**:
- **3-tier credential cascade**: Organization → Site → Device (like SNMP)
- **Encryption**: Passwords encrypted at rest (Cloak AES-256-GCM)
- **Detection**: Auto-detected from SNMP manufacturer field
- **Transport**: API-SSL (8729, default) or plain API (8728, insecure)
- **Security**: Plain API blocked for cloud pollers, passwords require `CLOAK_KEY` env var

**Key Files**:
- `lib/towerops/vault.ex` - Cloak vault
- `lib/towerops/ecto_types/encrypted_binary.ex` - Encrypted field type
- `lib/towerops/devices.ex` - Config resolution, propagation
- `priv/proto/agent.proto` - MikrotikDevice protobuf messages

**Credential Resolution**: Each field resolves independently up the hierarchy.

### MIB Name Resolution (Rust NIF)

Uses Rust NIF via Rustler for fast MIB resolution (replaces unreliable Erlang SNMP).

- **NIF Module**: `lib/towerops_native.ex`
- **Rust**: `native/towerops_native/src/mib.rs` (uses `snmptranslate` command)
- **MIB Files**: `priv/mibs/` (570+ vendor and standard MIBs)
- **Dependencies**: `brew install net-snmp` (macOS), `apt-get install libsnmp-dev snmp-mibs-downloader` (Docker)
- **Usage**: Try `ToweropsNative.resolve_oid/1` first, fallback to `SnmpKit.resolve/1`

Note: SnmpKit still used for SNMP protocol operations (get, walk, etc.).

## Admin Features

### User Impersonation
- Location: `/admin/users` (superuser only)
- Superusers CAN impersonate other superusers (feature, not bug)
- All events logged to audit_logs
- Stop link appears in user menu when active

### GeoIP Database
- Import MaxMind GeoLite2-City for IP-based country/city detection (GDPR cookies)
- Local: `make geoip-import DIR=~/Downloads/GeoLite2-City-CSV_20260127/`
- Production: `make geoip-import-prod DIR=...` (requires TOWEROPS_KEY)
- API: `POST /admin/api/geoip/import` (superuser only)

## Project-Specific Constraints

See `AGENTS.md` for the full set of coding constraints. Key reminders:
- Use `mix precommit` before committing (formats, compiles with warnings-as-errors, runs tests)
- Use `:req` (Req) for HTTP — never `:httpoison`, `:tesla`, `:httpc`
- Never use `daisyUI` — custom Tailwind components only
- Never put Ecto queries directly in LiveViews — use context modules
- Always run `mix format` after Elixir changes

### API Documentation

- Available at `/docs/api` (Tailwind UI Protocol template)
- Update controller `@doc` comments and `/docs/api` template when changing endpoints
- Wrap examples in `<%= raw(~S"""...""") %>` to prevent HEEx parsing

## Kubernetes Deployment

**Prerequisites**: cert-manager, Traefik, MetalLB, ArgoCD (with `argocd-image-updater`), NFS Provisioner

**Secrets** (1Password, `towerops` namespace):
- `gitlab-registry` - Docker credentials
- `towerops-secrets` - RELEASE_COOKIE, SECRET_KEY_BASE, CLOAK_KEY
- `towerops-db` - PostgreSQL connection
- `towerops-aws` - AWS credentials

**Create CLOAK_KEY**:
```bash
# Generate and store in 1Password first
CLOAK_KEY=$(openssl rand -base64 32)

# Create secret
kubectl create secret generic towerops-secrets \
  --from-literal=RELEASE_COOKIE=$(openssl rand -base64 32) \
  --from-literal=SECRET_KEY_BASE=$(mix phx.gen.secret) \
  --from-literal=CLOAK_KEY="$CLOAK_KEY" \
  -n towerops

# Or patch existing
kubectl patch secret towerops-secrets -n towerops \
  --type='json' \
  -p="[{'op': 'add', 'path': '/data/CLOAK_KEY', 'value': '$(echo -n "$CLOAK_KEY" | base64)'}]"

# Restart pods
kubectl rollout restart deployment/towerops -n towerops
```

**Important**: Losing CLOAK_KEY makes encrypted data unrecoverable.

**Deployment**: ArgoCD reconciles `k8s/` manifests automatically. Manual: `kubectl apply -k k8s/`

## Deployment

### Branch-Based Deployment Strategy

Towerops uses branch-based deployment with Forgejo CI/CD:

**Staging (Dokku):**
- Push to `main` branch triggers automatic deploy to Dokku staging server
- Direct git push to Dokku (no Docker build)
- Buildpack-based deployment (detects Elixir/Phoenix)
- URL: staging.towerops.app (or configured Dokku domain)

**Production (Kubernetes):**
- Push to `production` branch triggers CI/CD pipeline
- **Test Gates (must pass):**
  - All ExUnit tests (`mix test`)
  - All e2e tests (Playwright)
- Only after tests pass:
  - Builds Docker image from `k8s/Dockerfile`
  - Pushes to container registry
  - `argocd-image-updater` picks up the new tag (~2 min) and writes it to the
    Application's `spec.source.kustomize.images`; ArgoCD then rolls the
    Deployment. No commit to `k8s/deployment.yaml`.

### Deploying Changes

**CRITICAL GIT WORKFLOW RULE:**
- **ALWAYS** push to `main` after commits (default behavior)
- **NEVER** push to `production` unless explicitly instructed by the user
- Production deployments must be manually authorized each time
- When in doubt, only push to `main`

**To staging:**
```bash
git push origin main
# CI automatically deploys to Dokku
# Watch: https://git.mcintire.me/graham/towerops-web/actions
```

**To production (ONLY when explicitly requested):**
```bash
# Merge main → production
git checkout production
git merge main
git push origin production

# Or fast-forward if no conflicts
git push origin main:production

# CI/CD Pipeline:
# 1. Runs all ExUnit tests (must pass)
# 2. Runs all e2e tests (must pass)
# 3. Builds Docker image
# 4. Pushes to container registry
# 5. argocd-image-updater picks up the new tag (~2 min)
# 6. ArgoCD rolls the Deployment
#
# Watch progress: https://git.mcintire.me/graham/towerops-web/actions
```

**Direct Dokku deploy (bypass CI):**
```bash
# Add Dokku remote (one-time)
git remote add dokku dokku@204.110.191.231:towerops

# Force push to deploy
git push dokku main:main --force
```

**Manual production deploy:**
```bash
# Apply k8s manifests directly
kubectl apply -k k8s/

# Or trigger rollout restart
kubectl rollout restart deployment/towerops -n towerops
```

### Deployment Verification

**Staging:**
```bash
# Check Dokku logs
ssh dokku@204.110.191.231 logs towerops --tail 100

# Check app status
ssh dokku@204.110.191.231 ps:report towerops
```

**Production:**
```bash
# Check pod status
kubectl get pods -n towerops

# Check logs
kubectl logs -n towerops -l app=towerops --tail=100 -f

# Check deployment status
kubectl rollout status deployment/towerops -n towerops
```

### Rollback

**Staging (Dokku):**
```bash
# Revert git commit, push to main
git revert <bad-commit>
git push origin main

# Or rebuild previous release
ssh dokku@204.110.191.231 ps:rebuild towerops <release-id>
```

**Production (Kubernetes):**
```bash
# Rollback deployment
kubectl rollout undo deployment/towerops -n towerops

# Or revert git commit on production branch
git checkout production
git revert <bad-commit>
git push origin production
```

## Common Patterns

### Pagination in LiveView

Use `<.pagination>` component from `CoreComponents`:

```elixir
def handle_params(params, _url, socket) do
  page = params |> Map.get("page", "1") |> String.to_integer()
  per_page = 20

  all_items = MyContext.list_items(org_id)
  total_count = length(all_items)
  total_pages = ceil(total_count / per_page)
  page = max(1, min(page, max(1, total_pages)))

  offset = (page - 1) * per_page
  items = Enum.slice(all_items, offset, per_page)

  {:noreply, socket
   |> assign(:items, items)
   |> assign(:pagination, %{page: page, per_page: per_page,
                             total_count: total_count, total_pages: total_pages})}
end
```

Template:
```heex
<.pagination meta={@pagination} path={~p"/devices"} params={%{"tab" => @tab}} />
```

For large datasets (>10K), use database-level `limit/offset` instead of in-memory slicing.

## Testing Notes

- Never use npm — esbuild is built into Phoenix
- Run `cargo fmt` before committing Rust changes
- Never open test coverage HTML files — read results in terminal
- See `AGENTS.md` for full test guidelines, SNMP mocking patterns, and LiveView test helpers

## Dialyzer

- `mix dialyzer` - Run analysis (builds PLT first time)
- `mix dialyzer --format dialyzer` - Detailed error locations
- Never suppress valid warnings — fix root cause
- PLT files in `priv/plts/` — don't commit

## Changelog

**Two changelog files to maintain:**

### CHANGELOG.txt (Technical/Internal)
After every code change, append to `CHANGELOG.txt`:
- Date (YYYY-MM-DD)
- Short description (e.g. "fix: update last_snmp_poll_at for agent-polled devices")
- Files changed and brief explanation
- Technical details about implementation

Keep reverse chronological (newest at top). Never remove entries.

### priv/static/changelog.txt (User-Facing)
After significant changes, update `priv/static/changelog.txt`:
- Group changes by date (YYYY-MM-DD)
- Brief bullet points without code specifics
- Generic descriptions (no function names, modules, file paths, or language details)
- Focus on what changed for the user, not how it was implemented
- Follow existing pattern: "* Feature description" or "* Bug fix: brief description"

**Example conversions:**
- ❌ "fix: update last_snmp_poll_at in agent_channel.ex"
- ✅ "Poll time tracking improvements"

- ❌ "feat: add discover_wireless_sensors/1 to MikroTik vendor module"
- ✅ "Enhanced wireless monitoring for MikroTik devices"

**When to update user-facing changelog:**
- User-visible features or improvements
- Bug fixes affecting user experience
- New vendor/device support
- Performance improvements
- Security improvements
- Skip: Test-only changes, internal refactoring, documentation updates