towerops/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## CRITICAL: Read AGENTS.md First

**Before starting any work in this repository**, you MUST read `AGENTS.md` in the project root.

- `AGENTS.md` contains comprehensive Phoenix/LiveView/Elixir guidelines that are mandatory for this project
- These guidelines take precedence over general development practices when there's a conflict
- Always read `AGENTS.md` at the start of a new conversation or when resuming work
- Follow the project-specific patterns, conventions, and constraints documented there

## Project Overview

Towerops is a Phoenix 1.8 web application built with Elixir, using Ecto for database operations (PostgreSQL), LiveView for real-time interactions, and Tailwind CSS v4 for styling.

### Data Model Relationships

```
┌─────────────┐
│    User     │
│ (Accounts)  │
└──────┬──────┘
       │ owns/member of (many-to-many)
       │
       ↓
┌──────────────────────────────────────────────────────┐
│                   Organization                        │
│                  (Organizations)                      │
└────┬────────────┬─────────────┬──────────────────────┘
     │            │             │
     │ has many   │ has many    │ has many
     │            │             │
     ↓            ↓             ↓
┌─────────┐  ┌──────────┐  ┌────────────────┐
│  Site   │  │Equipment │  │  AgentToken    │
│ (Sites) │  │(Equipment│  │   (Agents)     │
└────┬────┘  └────┬─────┘  └────────┬───────┘
     │            │                  │
     │ has many   │ has one          │ assigned via
     │            │                  │
     ↓            ↓                  ↓
┌──────────┐ ┌──────────────┐  ┌──────────────────┐
│Equipment │ │  SNMPDevice  │  │ AgentAssignment  │◀──┐
│(Equipment│ │(Snmp.Devices)│  │    (Agents)      │   │
└────┬─────┘ └──────────────┘  └──────────────────┘   │
     │                                  │               │
     │ has many                         └───────────────┘
     ├────────────────┬─────────────────┘
     │                │               assigns equipment
     ↓                ↓                  to agent
┌──────────────┐ ┌─────────┐
│MonitoringCheck│ │ Alert   │
│ (Monitoring) │ │ (Alerts)│
└──────────────┘ └─────────┘

Key Relationships:
- User can own/belong to multiple Organizations
- Organization has default_agent_token_id (optional)
- Equipment belongs to both Site and Organization (denormalized)
- Equipment can be assigned to one AgentToken via AgentAssignment
- Equipment has one SNMPDevice with Sensors and Interfaces
- Equipment has many MonitoringChecks (polling results)
- Equipment has many Alerts (equipment_down, equipment_up)
- AgentToken authenticates remote agents for local SNMP polling
```

**Note to AI assistants**: When you make changes to the data model (migrations, schema additions, new contexts), update this diagram to reflect the current state.

## Essential Commands

### Setup and Development
- `mix setup` - Install dependencies, create/migrate database, setup and build assets
- `mix phx.server` - Start the Phoenix server (accessible at http://localhost:4000)
- `iex -S mix phx.server` - Start server with interactive Elixir shell

### Testing and Quality
- `mix test` - Run all tests (automatically creates and migrates test database)
- `mix test test/path/to/specific_test.exs` - Run a specific test file
- `mix test --failed` - Re-run only previously failed tests
- `mix test --cover` - Run tests with coverage report (generates HTML report in `cover/` directory)
- `mix precommit` - **Run before committing**: compiles with warnings as errors, unlocks unused deps, formats code, and runs tests

**Coverage Target**: 90% minimum (currently configured threshold)
- Coverage reports show line-by-line coverage in `cover/` directory
- View detailed HTML report: `open cover/modules_*.html`
- Current status: 60.40% overall (as of Jan 12, 2026, 6:35 PM)
- Recent improvements (Session 1):
  - Towerops.Snmp: 10.87% → 97.83% (+87%)
  - Towerops.Snmp.Profiles.NetSnmp: 56.79% → 92.59% (+35.8%)
  - Towerops.Monitoring.Supervisor: 21.21% → 63.64% (+42%)
  - Towerops.Snmp.Profiles.Cisco: 20.22% → 57.30% (+37%)
  - ToweropsWeb.EquipmentLive.Show: 35.51% → 46.38% (+10.87%)

**Coverage Notes**:
- Protobuf-generated modules (Towerops.Agent.*) show 0% but don't need tests
- Focus areas for improvement (in priority order):
  - Towerops.Monitoring context (33.33%)
  - SNMP profiles: NetSnmp (56.79%), Base (69.52%), Mikrotik (77.97%)
  - LiveView modules: EquipmentLive.Show (35.51%), EquipmentLive.Form (46.73%)
  - Accounts modules: UserCredentialController (33.93%), Accounts context (61.82%)
  - 0% coverage modules (Admin, WebAuthn, etc.) - may be intentionally untested

### Database
- `mix ecto.create` - Create the database
- `mix ecto.migrate` - Run pending migrations
- `mix ecto.reset` - Drop, create, and migrate database
- `mix ecto.gen.migration migration_name_using_underscores` - Generate a new migration file

### Assets
- `mix assets.build` - Build CSS and JS assets (Tailwind + esbuild)
- `mix assets.deploy` - Build minified assets for production

## Architecture

### Application Structure

The application follows standard Phoenix conventions with clear separation between business logic (`lib/towerops/`) and web interface (`lib/towerops_web/`):

- **Towerops.Application** - OTP application that supervises:
  - Telemetry for metrics
  - Repo (Ecto) for database
  - DNSCluster for service discovery
  - PubSub for pub/sub messaging
  - Endpoint (web server)

- **ToweropsWeb** - Main web module that provides `use` macros for:
  - `:router` - Route definitions
  - `:controller` - Traditional request/response controllers
  - `:live_view` - LiveView modules
  - `:live_component` - LiveView component modules
  - `:html` - Phoenix.Component modules

### Key Configuration Details

- **Binary IDs**: Generators use binary (UUID) primary keys by default (`binary_id: true`)
- **Timestamps**: Use `:utc_datetime` for all timestamps
- **Web server**: Uses Bandit adapter (not Cowboy)
- **HTTP client**: Uses `:req` library (Req module) - this is the only approved HTTP client
- **Ecto repos**: `[Towerops.Repo]`

### Web Layer Patterns

All LiveViews, LiveComponents, and HTML modules automatically get these imports/aliases via `html_helpers/0`:
- `ToweropsWeb.CoreComponents` - Core UI components (`<.button>`, `<.input>`, `<.form>`, etc.)
- `ToweropsWeb.Layouts` - Layout components (aliased, no need to re-alias)
- `Phoenix.LiveView.JS` - Client-side JS commands
- Gettext for translations
- Verified routes with `~p` sigil

### Asset Pipeline

- **Tailwind CSS v4**: Uses new `@import "tailwindcss"` syntax in `assets/css/app.css`, no `tailwind.config.js` needed
- **esbuild**: Bundles JavaScript from `assets/js/app.js`
- All vendor scripts/styles must be imported into app.js/app.css - cannot reference external src/href in layouts
- No inline `<script>` tags in templates - use colocated LiveView hooks instead

### Development Environment

- Dev routes enabled for:
  - LiveDashboard at `/dev/dashboard`
  - Swoosh mailbox preview at `/dev/mailbox`
- Phoenix LiveReload watches for file changes
- Code reloader enabled via `listeners: [Phoenix.CodeReloader]`

## Project-Specific Constraints

Key constraints from AGENTS.md (see that file for complete details):

- Use `mix precommit` before committing changes
- Use `:req` (Req) for all HTTP requests - never use `:httpoison`, `:tesla`, or `:httpc`
- Never use `daisyUI` - write custom Tailwind components for world-class design
- LiveView templates must start with `<Layouts.app flash={@flash}>`
- Use `<.icon name="hero-x-mark">` component for icons, never Heroicons modules
- Use LiveView streams for collections to avoid memory issues
- Forms must use `to_form/2` in LiveView and `<.form for={@form}>` in templates
- Never access changesets directly in templates - always use the form assign
- Tailwind v4 uses new import syntax - never use `@apply` in CSS
- when adding/updating hex modules, only use https://hex.pm
- assets are rebuilt on save and don't need to be built with mix assets.build
- be sure to always run mix format after you modify an elixir file
- when you run mix format, don't specify a file, let it format everything

## Kubernetes Deployment

### Prerequisites for Talos Kubernetes Cluster

The application requires the following infrastructure components to be installed in the cluster before deployment:

#### 1. Core Infrastructure

**cert-manager** - Manages SSL/TLS certificates
- Required for automatic Let's Encrypt certificate provisioning
- Must have ClusterIssuers configured: `letsencrypt-prod` and `letsencrypt-staging`

**Traefik** - Ingress controller
- Handles HTTP/HTTPS traffic routing
- Must be configured with `web` (HTTP) and `websecure` (HTTPS) entrypoints
- The application uses IngressRoute CRDs (not standard Ingress resources)

**MetalLB** - Load balancer for bare metal clusters
- Provides LoadBalancer service type support in on-premises environments
- Required for Traefik to get an external IP

**FluxCD** - GitOps continuous delivery
- Automatically syncs Kubernetes resources from Git repository
- Required components: source-controller, kustomize-controller, helm-controller, notification-controller
- Must have GitRepository resource pointing to this repository
- Must have Kustomization resource for the `k8s/` directory

#### 2. Storage

**NFS Provisioner** - Dynamic persistent volume provisioning
- Provides StorageClass for persistent volume claims
- Required for stateful workloads (if needed in the future)

#### 3. CI/CD Integration

**GitLab Agent** - Connects cluster to GitLab for CI/CD
- Enables GitLab CI/CD pipelines to deploy to the cluster
- Required for automated deployments from GitLab CI
- Agent configuration managed in `.gitlab/agents/` directory
- Agent must be registered in GitLab project settings

#### 4. Network Services

**Newt** - Pangolin tunnel forwarding
- Enables external access to internal services
- Required for Pangolin protocol forwarding
- Configuration specific to network topology

**CoreDNS** - Cluster DNS
- Standard Kubernetes DNS service (included in Talos by default)

**Tailscale Operator** - VPN mesh networking (optional)
- Provides secure access to cluster services
- Useful for development and administrative access

### Secrets Management

All secrets are stored in 1Password and must be created manually in the cluster before deployment.

#### Required Secrets in `towerops` Namespace

1. **`gitlab-registry`** - Docker registry credentials
   - Type: `kubernetes.io/dockerconfigjson`
   - Used to pull images from GitLab Container Registry
   - Retrieve from 1Password or GitLab project settings

2. **`towerops-secrets`** - Application secrets
   - Type: `Opaque`
   - Fields:
     - `RELEASE_COOKIE` - Erlang distributed cookie for node clustering
     - `SECRET_KEY_BASE` - Phoenix secret key base for session encryption
   - Stored in 1Password: "Towerops K8s Secrets"

3. **`towerops-db`** - Database connection
   - Type: `Opaque`
   - Fields:
     - `POSTGRES_HOST` - Database hostname
     - `POSTGRES_PORT` - Database port
     - `POSTGRES_DB` - Database name
     - `POSTGRES_USER` - Database username
     - `POSTGRES_PASSWORD` - Database password
     - `DATABASE_URL` - Full connection URL
   - Stored in 1Password: "Towerops K8s Database"

4. **`towerops-aws`** - AWS credentials
   - Type: `Opaque`
   - Fields:
     - `AWS_ACCESS_KEY_ID` - AWS access key
     - `AWS_SECRET_ACCESS_KEY` - AWS secret key
     - `AWS_REGION` - AWS region (us-east-1)
   - Stored in 1Password: "Towerops K8s AWS"

#### Creating Secrets from 1Password

```bash
# Set 1Password account
export OP_ACCOUNT=YOOATCZZSVGH7AD6VABUVPORLI

# Create towerops-secrets
kubectl create secret generic towerops-secrets -n towerops \
  --from-literal=RELEASE_COOKIE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=RELEASE_COOKIE)" \
  --from-literal=SECRET_KEY_BASE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=SECRET_KEY_BASE)"

# Create towerops-db
kubectl create secret generic towerops-db -n towerops \
  --from-literal=POSTGRES_HOST="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=server)" \
  --from-literal=POSTGRES_PORT="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=port)" \
  --from-literal=POSTGRES_DB="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=database)" \
  --from-literal=POSTGRES_USER="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=username)" \
  --from-literal=POSTGRES_PASSWORD="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=password)" \
  --from-literal=DATABASE_URL="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=DATABASE_URL)"

# Create towerops-aws
kubectl create secret generic towerops-aws -n towerops \
  --from-literal=AWS_ACCESS_KEY_ID="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_ACCESS_KEY_ID)" \
  --from-literal=AWS_SECRET_ACCESS_KEY="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_SECRET_ACCESS_KEY)" \
  --from-literal=AWS_REGION="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_REGION)"
```

### Deployment Process

#### FluxCD Automatic Deployment

The application is automatically deployed via FluxCD GitOps:

1. FluxCD monitors the Git repository for changes
2. When changes are detected, it applies the manifests in `k8s/` directory
3. Kustomize builds the manifests and applies them to the cluster

**GitRepository Configuration:**
```yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: towerops
  namespace: flux-system
spec:
  interval: 1m0s
  url: ssh://git@gitlab.com/graham/towerops.git
  ref:
    branch: main
  secretRef:
    name: towerops-git
```

**Kustomization Configuration:**
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: towerops-app
  namespace: flux-system
spec:
  interval: 1m0s
  path: ./k8s
  prune: true
  sourceRef:
    kind: GitRepository
    name: towerops
  targetNamespace: towerops
```

#### Manual Deployment

If FluxCD is not available or for testing:

```bash
# Apply all resources using kustomize
kubectl apply -k k8s/

# Or apply individually
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/service-headless.yaml
kubectl apply -f k8s/certificate.yaml
kubectl apply -f k8s/ingressroute.yaml
```

### Cluster Setup Checklist

When recreating a Talos cluster from scratch:

- [ ] Install Talos on nodes and bootstrap cluster
- [ ] Install MetalLB and configure IP address pool
- [ ] Install Traefik ingress controller
- [ ] Install cert-manager and configure ClusterIssuers
- [ ] Install FluxCD and configure GitRepository source
- [ ] Install GitLab Agent and connect to GitLab project
- [ ] Install Newt for Pangolin forwarding
- [ ] Install NFS provisioner (if using persistent storage)
- [ ] Create `towerops` namespace
- [ ] Create all required secrets from 1Password
- [ ] Create FluxCD Kustomization resource for `k8s/` directory
- [ ] Verify deployment with `kubectl get pods -n towerops`
- [ ] Check certificate provisioning with `kubectl get certificate -n towerops`
- [ ] Test ingress with `curl https://towerops.net/health`

### Troubleshooting

**FluxCD reconciliation errors:**
- Check kustomization status: `kubectl get kustomization -n flux-system`
- View detailed status: `kubectl describe kustomization towerops-app -n flux-system`
- Force reconciliation: `kubectl annotate kustomization towerops-app -n flux-system reconcile.fluxcd.io/requestedAt="$(date +%Y-%m-%dT%H:%M:%S%z)" --overwrite`

**Certificate issues:**
- Check certificate status: `kubectl describe certificate towerops-net-cert -n towerops`
- View cert-manager logs: `kubectl logs -n cert-manager deployment/cert-manager`
- Verify ClusterIssuer: `kubectl get clusterissuer letsencrypt-prod -o yaml`

**Pod not starting:**
- Check pod status: `kubectl describe pod -n towerops -l app=towerops`
- View logs: `kubectl logs -n towerops deployment/towerops`
- Verify secrets exist: `kubectl get secrets -n towerops`

## Testing Patterns

### SNMP Mocking with Mox

The application uses Mox for SNMP client mocking in tests. Key patterns:

**SNMP Adapter Mock Format**:
- `snmp_adapter().get/3` returns `{:ok, value}` or `{:error, reason}`
- `snmp_adapter().walk/3` returns `{:ok, [%{oid: "...", value: ...}]}` (list of maps, NOT a map)
- Values from `get/3` are already extracted (no type wrapper needed in mocks)

**Mock Expectations**:
```elixir
# Mock get_multiple - it calls get/3 once per OID
expect(SnmpMock, :get, 6, fn _target, oid, _opts ->
  case oid do
    "1.3.6.1.2.1.1.1.0" -> {:ok, "Cisco IOS Software"}  # sysDescr
    "1.3.6.1.2.1.1.2.0" -> {:ok, [1, 3, 6, 1, 4, 1, 9]}  # sysObjectID
    "1.3.6.1.2.1.1.3.0" -> {:ok, 12_345}  # sysUpTime (integer)
    "1.3.6.1.2.1.1.4.0" -> {:ok, "admin@example.com"}  # sysContact
    "1.3.6.1.2.1.1.5.0" -> {:ok, "test-device"}  # sysName
    "1.3.6.1.2.1.1.6.0" -> {:ok, "Test Location"}  # sysLocation
  end
end)

# Mock walk - returns list of OID/value maps
expect(SnmpMock, :walk, fn _target, _oid, _opts ->
  {:ok, [
    %{oid: "1.3.6.1.4.1.9.9.91.1.1.1.1.1.1000", value: 8}
  ]}
end)

# Empty walk result (no data found)
expect(SnmpMock, :walk, fn _, _, _ ->
  {:ok, []}  # NOT {:ok, %{}}
end)
```

**Common Pitfalls**:
- ❌ Returning `{:ok, %{}}` from walk - should be `{:ok, []}`
- ❌ Returning type wrappers like `{:integer, 123}` - values are already extracted
- ❌ Not matching OIDs in get expectations - use pattern matching on OID string
- ❌ Expecting wrong number of calls - `get_multiple/2` calls `get/3` once per OID

**TimescaleDB Tests**:
- Tests that query continuous aggregates should be tagged with `@tag :skip`
- Reason: Test database doesn't have TimescaleDB continuous aggregates configured
- Example: `get_hourly_stats/3`, `get_daily_stats/3`, `get_uptime_percentage/1`

### Test Organization

- Use `DataCase` for tests that need database access
- Use `ConnCase` for controller/LiveView tests
- Use `async: true` for tests that can run in parallel (most unit tests)
- Use `async: false` for tests with shared state (supervisor tests, integration tests)
- never open the test coverage html files
- remember when working in rust to always run cargo fmt before committing