towerops/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## CRITICAL: Read AGENTS.md First

**Before starting any work in this repository**, you MUST read `AGENTS.md` in the project root.

- `AGENTS.md` contains comprehensive Phoenix/LiveView/Elixir guidelines that are mandatory for this project
- These guidelines take precedence over general development practices when there's a conflict
- Always read `AGENTS.md` at the start of a new conversation or when resuming work
- Follow the project-specific patterns, conventions, and constraints documented there

## Project Overview

Towerops is a Phoenix 1.8 web application built with Elixir, using Ecto for database operations (PostgreSQL), LiveView for real-time interactions, and Tailwind CSS v4 for styling.

### Data Model Relationships

```
┌─────────────┐
│    User     │
│ (Accounts)  │
└──────┬──────┘
       │ owns/member of (many-to-many)
       │
       ↓
┌──────────────────────────────────────────────────────┐
│                   Organization                        │
│                  (Organizations)                      │
└────┬────────────┬─────────────┬──────────────────────┘
     │            │             │
     │ has many   │ has many    │ has many
     │            │             │
     ↓            ↓             ↓
┌─────────┐  ┌──────────┐  ┌────────────────┐
│  Site   │  │Equipment │  │  AgentToken    │
│ (Sites) │  │(Equipment│  │   (Agents)     │
└────┬────┘  └────┬─────┘  └────────┬───────┘
     │            │                  │
     │ has many   │ has one          │ assigned via
     │            │                  │
     ↓            ↓                  ↓
┌──────────┐ ┌──────────────┐  ┌──────────────────┐
│Equipment │ │  SNMPDevice  │  │ AgentAssignment  │◀──┐
│(Equipment│ │(Snmp.Devices)│  │    (Agents)      │   │
└────┬─────┘ └──────────────┘  └──────────────────┘   │
     │                                  │               │
     │ has many                         └───────────────┘
     ├────────────────┬─────────────────┘
     │                │               assigns equipment
     ↓                ↓                  to agent
┌──────────────┐ ┌─────────┐
│MonitoringCheck│ │ Alert   │
│ (Monitoring) │ │ (Alerts)│
└──────────────┘ └─────────┘

Key Relationships:
- User can own/belong to multiple Organizations
- Organization has default_agent_token_id (optional)
- Equipment belongs to both Site and Organization (denormalized)
- Equipment can be assigned to one AgentToken via AgentAssignment
- Equipment has one SNMPDevice with Sensors and Interfaces
- Equipment has many MonitoringChecks (polling results)
- Equipment has many Alerts (equipment_down, equipment_up)
- AgentToken authenticates remote agents for local SNMP polling
```

**Note to AI assistants**: When you make changes to the data model (migrations, schema additions, new contexts), update this diagram to reflect the current state.

## Essential Commands

### Setup and Development
- `mix setup` - Install dependencies, create/migrate database, setup and build assets
- `mix phx.server` - Start the Phoenix server (accessible at http://localhost:4000)
- `iex -S mix phx.server` - Start server with interactive Elixir shell

### Testing and Quality
- `mix test` - Run all tests (automatically creates and migrates test database)
- `mix test test/path/to/specific_test.exs` - Run a specific test file
- `mix test --failed` - Re-run only previously failed tests
- `mix test --cover` - Run tests with coverage report (generates HTML report in `cover/` directory)
- `mix precommit` - **Run before committing**: compiles with warnings as errors, unlocks unused deps, formats code, and runs tests

**Coverage Target**: 90% minimum (currently configured threshold)
- Coverage reports show line-by-line coverage in `cover/` directory
- View detailed HTML report: `open cover/modules_*.html`
- Current status: 60.40% overall (as of Jan 12, 2026, 6:35 PM)
- Recent improvements (Session 1):
  - Towerops.Snmp: 10.87% → 97.83% (+87%)
  - Towerops.Snmp.Profiles.NetSnmp: 56.79% → 92.59% (+35.8%)
  - Towerops.Monitoring.Supervisor: 21.21% → 63.64% (+42%)
  - Towerops.Snmp.Profiles.Cisco: 20.22% → 57.30% (+37%)
  - ToweropsWeb.EquipmentLive.Show: 35.51% → 46.38% (+10.87%)

**Coverage Notes**:
- Protobuf-generated modules (Towerops.Agent.*) show 0% but don't need tests
- Focus areas for improvement (in priority order):
  - Towerops.Monitoring context (33.33%)
  - SNMP profiles: NetSnmp (56.79%), Base (69.52%), Mikrotik (77.97%)
  - LiveView modules: EquipmentLive.Show (35.51%), EquipmentLive.Form (46.73%)
  - Accounts modules: UserCredentialController (33.93%), Accounts context (61.82%)
  - 0% coverage modules (Admin, WebAuthn, etc.) - may be intentionally untested

### Database
- `mix ecto.create` - Create the database
- `mix ecto.migrate` - Run pending migrations
- `mix ecto.reset` - Drop, create, and migrate database
- `mix ecto.gen.migration migration_name_using_underscores` - Generate a new migration file

### Assets
- `mix assets.build` - Build CSS and JS assets (Tailwind + esbuild)
- `mix assets.deploy` - Build minified assets for production

## Architecture

### Application Structure

The application follows standard Phoenix conventions with clear separation between business logic (`lib/towerops/`) and web interface (`lib/towerops_web/`):

- **Towerops.Application** - OTP application that supervises:
  - Telemetry for metrics
  - Repo (Ecto) for database
  - DNSCluster for service discovery
  - PubSub for pub/sub messaging
  - Endpoint (web server)

- **ToweropsWeb** - Main web module that provides `use` macros for:
  - `:router` - Route definitions
  - `:controller` - Traditional request/response controllers
  - `:live_view` - LiveView modules
  - `:live_component` - LiveView component modules
  - `:html` - Phoenix.Component modules

### Key Configuration Details

- **Binary IDs**: Generators use binary (UUID) primary keys by default (`binary_id: true`)
- **Timestamps**: Use `:utc_datetime` for all timestamps
- **Web server**: Uses Bandit adapter (not Cowboy)
- **HTTP client**: Uses `:req` library (Req module) - this is the only approved HTTP client
- **Ecto repos**: `[Towerops.Repo]`

### Web Layer Patterns

All LiveViews, LiveComponents, and HTML modules automatically get these imports/aliases via `html_helpers/0`:
- `ToweropsWeb.CoreComponents` - Core UI components (`<.button>`, `<.input>`, `<.form>`, etc.)
- `ToweropsWeb.Layouts` - Layout components (aliased, no need to re-alias)
- `Phoenix.LiveView.JS` - Client-side JS commands
- Gettext for translations
- Verified routes with `~p` sigil

### Asset Pipeline

- **Tailwind CSS v4**: Uses new `@import "tailwindcss"` syntax in `assets/css/app.css`, no `tailwind.config.js` needed
- **esbuild**: Bundles JavaScript from `assets/js/app.js`
- All vendor scripts/styles must be imported into app.js/app.css - cannot reference external src/href in layouts
- No inline `<script>` tags in templates - use colocated LiveView hooks instead

### Development Environment

- Dev routes enabled for:
  - LiveDashboard at `/dev/dashboard`
  - Swoosh mailbox preview at `/dev/mailbox`
- Phoenix LiveReload watches for file changes
- Code reloader enabled via `listeners: [Phoenix.CodeReloader]`

### Background Job Architecture

The application uses Oban for all background job processing with PostgreSQL-backed queuing for cluster-wide coordination and resilience.

**Job Types:**

1. **Self-Scheduling Workers** (per-device recurring jobs):
   - `DeviceMonitorWorker` - Health check polling (60s default interval)
   - `DevicePollerWorker` - SNMP data collection (60s default interval)
   - Schedule next run before completing via `schedule_next_poll/2`
   - One job per enabled device, automatically created/cancelled when device settings change

2. **Oban Cron Workers** (cluster-wide periodic maintenance):
   - `NeighborCleanupWorker` - Hourly cleanup of stale neighbors/ARP/MAC entries (runs every hour via cron)
   - `StaleAgentWorker` - Detects agents that haven't checked in for 10+ minutes (runs every minute)
   - `AgentLatencyEvaluator` - Latency-based agent reassignment (runs every 5 minutes)
   - `JobHealthCheckWorker` - Safety net to recover missing monitor/poller jobs (runs every 10 minutes)
   - Scheduled via `Oban.Plugins.Cron` in `config/dev.exs` and `config/runtime.exs`
   - Run cluster-wide (only one instance executes at a time across all pods)

**Queues:**
- `default` (10 workers) - General background tasks
- `discovery` (10 workers) - SNMP discovery operations
- `pollers` (50 workers) - SNMP polling jobs (one per device)
- `monitors` (50 workers) - Health check jobs (one per device)
- `maintenance` (5 workers) - Periodic cleanup and health check workers

**Resilience:**
- Oban Cron jobs run cluster-wide via PostgreSQL-based locking
- If a pod dies, another pod immediately picks up scheduled cron jobs
- Self-scheduling workers (monitor/poller) are recovered by `JobHealthCheckWorker` every 10 minutes
- All jobs visible in Oban dashboard at `/dev/dashboard` → Oban tab
- Failed jobs automatically retried with exponential backoff

**Configuration:**
- Dev: `config/dev.exs` - Oban config with cron plugin
- Prod: `config/runtime.exs` - Same cron schedule as dev
- Application: `lib/towerops/application.ex` - Oban supervision, no GenServer workers for periodic tasks

### LiveDashboard and Telemetry

LiveDashboard is available at `/dashboard` (requires authentication in production, `/dev/dashboard` in development).

**Available Metrics:**
- **Phoenix Metrics**: Request duration, route timings, channel/socket metrics
- **Database Metrics**: Query time, decode time, queue time, idle time
- **VM Metrics**: Memory usage, run queue lengths
- **Oban/Background Jobs**:
  - Queue sizes (default, discovery, pollers, monitors, maintenance)
  - Currently executing jobs
  - Available jobs waiting to be processed
- **Redis/Valkey**:
  - Connected clients
  - Memory usage (MB)
  - Total commands processed

Metrics are collected every 10 seconds via `telemetry_poller` and published using `:telemetry.execute/3`.

To view metrics:
1. Navigate to `/dashboard` (or `/dev/dashboard` in development)
2. Click the "Metrics" tab
3. Scroll to see Oban and Redis sections

### Oban Web Dashboard

Oban Web provides a dedicated dashboard for job monitoring and management at `/oban` (superuser access required).

**Access Control**: Only superusers can access the Oban Web dashboard. The route is protected by `:require_superuser` plug.

**Features**:
- Real-time job queue monitoring
- Job filtering and search
- Job details and execution history
- Queue statistics and insights
- Job cancellation and retry
- Worker and plugin configuration

**Router Configuration**:
```elixir
# lib/towerops_web/router.ex
scope "/" do
  pipe_through [:browser, :require_authenticated_user, :require_superuser]

  oban_dashboard "/oban"
end
```

**Vendored Dependencies**:
All Oban packages are vendored in `vendor/` to avoid requiring authentication during CI/CD:
- `oban_pro` (1.6.11) - Commercial Oban Pro features
- `oban_web` (2.11.7) - Web dashboard
- `oban_met` (1.0.5) - Metrics support

See `vendor/README.md` for update instructions.

## Admin Features

### User Impersonation

**Location**: `/admin/users` (superuser only)

**Files**:
- LiveView: `lib/towerops_web/live/admin/user_live/index.ex`
- Template: `lib/towerops_web/live/admin/user_live/index.html.heex`
- Controller: `lib/towerops_web/controllers/admin_controller.ex`
- Auth: `lib/towerops_web/user_auth.ex` (start_impersonation/2, stop_impersonation/1)

**Implementation Notes**:
- Impersonate button is disabled only when `user.id == @current_scope.user.id` (prevents impersonating yourself)
- Superusers CAN impersonate other superusers (security feature, not bug)
- Backend checks in `UserAuth.start_impersonation/2` prevent self-impersonation
- All impersonation events are logged to audit_logs table via `Towerops.Admin.create_audit_log/1`
- Stop impersonation: "Stop Impersonating" link appears in user menu when impersonating

### GeoIP Database Management

**Purpose**: Import MaxMind GeoLite2-City database for IP-based country/city detection (used for GDPR cookie consent).

**Files**:
- Controller: `lib/towerops_web/controllers/api/v1/geoip_controller.ex`
- Mix Task: `lib/mix/tasks/geoip.import.ex`
- Schemas: `lib/towerops/geoip/location.ex`, `lib/towerops/geoip/block.ex`
- Lookup: `lib/towerops/geoip.ex`
- Migration: `priv/repo/migrations/*_create_geoip_tables.exs`

**Database Tables**:
- `geoip_locations` (~130,000 rows) - Countries, cities, regions with coordinates
- `geoip_blocks` (~3.5M rows) - IP ranges mapped to locations

**Import Methods**:

1. **Local Import (Development)**:
```bash
make geoip-import DIR=~/Downloads/GeoLite2-City-CSV_20260127/
```

2. **Production Import via API** (requires `TOWEROPS_KEY` env var):
```bash
# Using Makefile (recommended)
make geoip-import-prod DIR=~/Downloads/GeoLite2-City-CSV_20260127/

# Or directly with mix
mix geoip.import ~/Downloads/GeoLite2-City-CSV_20260127/ --production
```

**How It Works**:
- **Local mode** (`make geoip-import`): Direct database import via Ecto
- **Production mode** (`make geoip-import-prod` or `--production` flag): Processes CSVs locally, sends data to API in batches of 5,000 records
- API endpoint: `POST /admin/api/geoip/import` (superuser only)
- No need to upload large CSV files to production - only processed data is sent over HTTPS
- Production mode requires `TOWEROPS_KEY` environment variable (superuser API token)
- Default API URL: `https://towerops.net` (override with `TOWEROPS_URL` env var)

**Implementation Notes**:
- Uses batch inserts of 5,000 rows to stay within PostgreSQL's 65,535 parameter limit
- Automatically filters orphaned IP blocks (those without matching location references)
- Mix task processes CSVs locally and sends batches via API when `TOWEROPS_KEY` is set
- CSV files are NOT committed to git - imported separately in each environment
- Download GeoLite2-City CSV from MaxMind: https://dev.maxmind.com/geoip/geolite2-free-geolocation-data
- Required files: `GeoLite2-City-Blocks-IPv4.csv`, `GeoLite2-City-Locations-en.csv`

## Project-Specific Constraints

Key constraints from AGENTS.md (see that file for complete details):

- Use `mix precommit` before committing changes
- Use `:req` (Req) for all HTTP requests - never use `:httpoison`, `:tesla`, or `:httpc`
- Never use `daisyUI` - write custom Tailwind components for world-class design
- LiveView templates must start with `<Layouts.app flash={@flash}>`
- Use `<.icon name="hero-x-mark">` component for icons, never Heroicons modules
- Use LiveView streams for collections to avoid memory issues
- Forms must use `to_form/2` in LiveView and `<.form for={@form}>` in templates
- Never access changesets directly in templates - always use the form assign
- Tailwind v4 uses new import syntax - never use `@apply` in CSS
- when adding/updating hex modules, only use https://hex.pm
- assets are rebuilt on save and don't need to be built with mix assets.build
- be sure to always run mix format after you modify an elixir file
- when you run mix format, don't specify a file, let it format everything

### Database Schema Critical Notes

**Binary UUID Primary Keys**: All tables MUST use `:binary_id` for primary keys, not `bigint`.

When creating new migrations with tables that have UUID primary keys:
```elixir
create table(:table_name, primary_key: false) do
  add :id, :binary_id, primary_key: true
  # ...
end
```

**Issue History**: The `snmp_neighbors` table was initially created with a `bigint` primary key instead of `:binary_id`, causing `DBConnection.EncodeError` when trying to insert records. This happened because the migration file was correct but the table was created before the fix. Always verify the actual database schema matches the migration:

```bash
psql towerops_dev -c "\d table_name"  # Check actual schema
```

If a table has the wrong primary key type, create a fix migration that drops and recreates the table.

**SNMP Binary Data Handling**: All SNMP values (chassis IDs, port IDs, system names) must be converted to printable strings before saving to the database. Non-printable binaries (like MAC addresses in binary format) should be converted to colon-separated hex format using `String.printable?/1` checks. See `lib/towerops/snmp/neighbor_discovery.ex:sanitize_string_field/1` for the pattern.

### API Documentation

The API documentation is available at `/docs/api` and documents all public API endpoints.

**Location**:
- Controller: `lib/towerops_web/controllers/api_docs_controller.ex`
- HTML Module: `lib/towerops_web/controllers/api_docs_html.ex`
- Template: `lib/towerops_web/controllers/api_docs_html/index.html.heex`
- Route: `get "/docs/api", ApiDocsController, :index` in `router.ex`

**Template Structure**:
- Adapted from Tailwind UI Protocol template
- Sidebar navigation with section links
- Two-column layout: descriptions on left, code examples on right
- Sections: Introduction, Authentication, Errors, Sites API, Devices API

**IMPORTANT: When updating API endpoints**:
1. Update the corresponding controller documentation (`@doc` comments in controller)
2. Update `/docs/api` template with new endpoint documentation
3. Follow the existing pattern:
   - Add endpoint to navigation sidebar if it's a new resource
   - Document the endpoint with HTTP method badge (GET/POST/PATCH/DELETE)
   - Include request example with `curl`
   - Include JSON response example
   - Document all parameters (required and optional)
4. Wrap all code examples in `<%= raw(~S"""...""") %>` to prevent HEEx from parsing curly braces as Elixir code

**Code Example Pattern**:
```heex
<pre class="p-4 text-sm text-zinc-100 overflow-x-auto"><code><%= raw(~S"""
curl https://towerops.net/api/v1/sites \
  -H "Authorization: Bearer {token}"
""") %></code></pre>
```

**JSON Example Pattern**:
```heex
<pre class="p-4 text-sm text-zinc-100 overflow-x-auto"><code><%= raw(~S"""
{
  "id": "uuid",
  "name": "Site Name"
}
""") %></code></pre>
```

**Current Documented Endpoints**:
- Sites: GET /api/v1/sites, POST /api/v1/sites, GET /api/v1/sites/:id, PATCH /api/v1/sites/:id, DELETE /api/v1/sites/:id
- Devices: GET /api/v1/devices, POST /api/v1/devices, GET /api/v1/devices/:id, PATCH /api/v1/devices/:id, DELETE /api/v1/devices/:id

### Background Job Architecture (Oban)

The application uses **Oban** (PostgreSQL-backed job queue) for all background processing. Jobs are distributed across the cluster and automatically retried on failure.

**Oban Queues:**
- `default` (10 workers) - General background tasks
- `discovery` (10 workers) - SNMP discovery jobs
- `pollers` (50 workers) - SNMP polling (one job per device)
- `monitors` (50 workers) - Device health monitoring (one job per device)
- `maintenance` (5 workers) - Cleanup and housekeeping tasks

**Key Oban Workers:**

1. **`DevicePollerWorker`** - SNMP polling for time-series data
   - Runs every 60 seconds per device (configurable via `check_interval_seconds`)
   - Collects: sensor readings, interface stats, neighbor topology, ARP/MAC tables
   - Uses Oban `unique` constraint to prevent duplicate jobs
   - Self-schedules next poll after completion
   - Broadcasts PubSub events for real-time UI updates

2. **`DeviceMonitorWorker`** - Health monitoring via ping
   - Runs every 30 seconds per device
   - Performs ICMP ping checks to detect device up/down status
   - Creates alerts on status changes
   - Self-schedules next check after completion

3. **`DiscoveryWorker`** - One-time SNMP discovery
   - Triggered manually or on device creation
   - Collects: system info, interfaces, sensors, neighbors
   - Creates/updates SNMPDevice record with full topology

4. **`NeighborCleanupWorker`** - Maintenance task
   - Runs periodically to clean up stale neighbor data
   - Deletes neighbors older than 24 hours

**Automatic Job Management:**
- Jobs are automatically started when `monitoring_enabled` or `snmp_enabled` is set to `true`
- Jobs are automatically stopped when disabled or device is deleted
- All job lifecycle managed in `Devices` context (create_device, update_device, delete_device)

### SNMP Polling Architecture

The application has two separate SNMP collection mechanisms:

**1. Discovery (`Towerops.Snmp.Discovery`)** - One-time or manual collection
- Triggered manually by user or via `DiscoveryWorker` Oban job
- Collects: system info, device identification, interfaces, sensors, **neighbors**
- Creates/updates the device record and all associated data
- Typically runs when equipment is first added or when user clicks "Rediscover"

**2. Polling (`DevicePollerWorker`)** - Continuous time-series collection
- Runs automatically every 60 seconds (configurable per equipment)
- Collects: sensor readings, interface statistics, **neighbor topology**
- Updates time-series data without recreating device records
- Keeps neighbor data fresh by polling LLDP/CDP every minute
- **Architecture**: Direct Oban worker (no GenServer or coordinator layer)

**Neighbor Data Lifecycle:**
- Discovery saves initial neighbors with 5-minute stale threshold
- Poller continuously updates neighbors every 60 seconds with 5-minute stale threshold
- `NeighborCleanupWorker` deletes neighbors older than 24 hours (safety net)
- This ensures neighbors stay current and topology changes are detected quickly

#### Polling Scheduling Details

**How Polling Works** (`lib/towerops/workers/device_poller_worker.ex`):

1. **Job Creation**:
   - `DevicePollerWorker.start_polling/1` (lines 66-70) - Creates initial Oban job for a device
   - Called from `Devices.create_device/1` (line 275-278) when `snmp_enabled=true`
   - Called from `Devices.update_device/2` (line 379-396) when SNMP is enabled

2. **Self-Scheduling Pattern**:
   - After each poll completes, `schedule_next_poll/2` (lines 1066-1070) creates the next job
   - Uses `new(schedule_in: interval_seconds)` to schedule the next run
   - Interval comes from `device.check_interval_seconds` field (default 300 seconds / 5 minutes)
   - Minimum interval enforced by `Application.get_env(:towerops, :snmp_min_poll_interval, 300)`

3. **Unique Job Constraint**:
   - Uses Oban unique constraint with 60-second period (lines 53-63)
   - Ensures only one polling job per device exists across the cluster
   - Prevents duplicate jobs if multiple pods try to schedule simultaneously

4. **Current Problem - Synchronized Polling**:
   - **All devices start polling at the same time** when first created
   - Each device reschedules exactly `check_interval_seconds` after previous poll completes
   - Results in "thundering herd" - all devices poll at clock intervals (1:00, 1:05, 1:10, etc.)
   - This creates load spikes on the server and Oban queue

5. **Job Recovery** (`lib/towerops/workers/job_health_check_worker.ex`):
   - Runs every 10 minutes via Oban Cron
   - Finds devices with `snmp_enabled=true` but no active polling job
   - Re-creates missing jobs (lines 63-80)
   - Provides resilience after pod failures/crashes

**Key Configuration**:
- Default interval: 300 seconds (5 minutes) - defined in `lib/towerops/devices/device.ex:31`
- Validation: 1-3600 seconds allowed (line 95)
- Oban queue: `:pollers` with 50 concurrent workers (`config/dev.exs:39`)
- Queue concurrency can handle ~50 devices polling simultaneously

6. **Load Distribution - Staggered Polling**:
   - **Offset Calculation**: Uses deterministic hash-based offsets via `PollingOffset.calculate_offset/2`
   - **Formula**: `offset = :erlang.phash2(device_id) |> rem(interval_seconds)`
   - **Distribution**: Devices spread evenly across full interval (0 to interval-1 seconds)
   - **Example**: With 300s interval and 100 devices:
     - Device A → offset 94s → polls at :01:34, :06:34, :11:34
     - Device B → offset 171s → polls at :02:51, :07:51, :12:51
     - Device C → offset 247s → polls at :04:07, :09:07, :14:07
   - **Deterministic**: Same device always gets same offset (stable across restarts)
   - **Applied to**: Both `DevicePollerWorker` (SNMP polling) and `DeviceMonitorWorker` (health checks)
   - **Implementation**: `lib/towerops/workers/polling_offset.ex`

**Related Files**:
- Offset calculator: `lib/towerops/workers/polling_offset.ex`
- Poller worker: `lib/towerops/workers/device_poller_worker.ex`
- Monitor worker: `lib/towerops/workers/device_monitor_worker.ex`
- Device context: `lib/towerops/devices.ex` (lines 264-285, 379-416)
- Device schema: `lib/towerops/devices/device.ex`
- Config: `config/dev.exs` (lines 31-61), `config/runtime.exs` (lines 103-131)

## Kubernetes Deployment

### Prerequisites for Talos Kubernetes Cluster

The application requires the following infrastructure components to be installed in the cluster before deployment:

#### 1. Core Infrastructure

**cert-manager** - Manages SSL/TLS certificates
- Required for automatic Let's Encrypt certificate provisioning
- Must have ClusterIssuers configured: `letsencrypt-prod` and `letsencrypt-staging`

**Traefik** - Ingress controller
- Handles HTTP/HTTPS traffic routing
- Must be configured with `web` (HTTP) and `websecure` (HTTPS) entrypoints
- The application uses IngressRoute CRDs (not standard Ingress resources)

**MetalLB** - Load balancer for bare metal clusters
- Provides LoadBalancer service type support in on-premises environments
- Required for Traefik to get an external IP

**FluxCD** - GitOps continuous delivery
- Automatically syncs Kubernetes resources from Git repository
- Required components: source-controller, kustomize-controller, helm-controller, notification-controller
- Must have GitRepository resource pointing to this repository
- Must have Kustomization resource for the `k8s/` directory

#### 2. Storage

**NFS Provisioner** - Dynamic persistent volume provisioning
- Provides StorageClass for persistent volume claims
- Required for stateful workloads (if needed in the future)

#### 3. CI/CD Integration

**GitLab Agent** - Connects cluster to GitLab for CI/CD
- Enables GitLab CI/CD pipelines to deploy to the cluster
- Required for automated deployments from GitLab CI
- Agent configuration managed in `.gitlab/agents/` directory
- Agent must be registered in GitLab project settings

#### 4. Network Services

**Newt** - Pangolin tunnel forwarding
- Enables external access to internal services
- Required for Pangolin protocol forwarding
- Configuration specific to network topology

**CoreDNS** - Cluster DNS
- Standard Kubernetes DNS service (included in Talos by default)

**Tailscale Operator** - VPN mesh networking (optional)
- Provides secure access to cluster services
- Useful for development and administrative access

### Secrets Management

All secrets are stored in 1Password and must be created manually in the cluster before deployment.

#### Required Secrets in `towerops` Namespace

1. **`gitlab-registry`** - Docker registry credentials
   - Type: `kubernetes.io/dockerconfigjson`
   - Used to pull images from GitLab Container Registry
   - Retrieve from 1Password or GitLab project settings

2. **`towerops-secrets`** - Application secrets
   - Type: `Opaque`
   - Fields:
     - `RELEASE_COOKIE` - Erlang distributed cookie for node clustering
     - `SECRET_KEY_BASE` - Phoenix secret key base for session encryption
   - Stored in 1Password: "Towerops K8s Secrets"

3. **`towerops-db`** - Database connection
   - Type: `Opaque`
   - Fields:
     - `POSTGRES_HOST` - Database hostname
     - `POSTGRES_PORT` - Database port
     - `POSTGRES_DB` - Database name
     - `POSTGRES_USER` - Database username
     - `POSTGRES_PASSWORD` - Database password
     - `DATABASE_URL` - Full connection URL
   - Stored in 1Password: "Towerops K8s Database"

4. **`towerops-aws`** - AWS credentials
   - Type: `Opaque`
   - Fields:
     - `AWS_ACCESS_KEY_ID` - AWS access key
     - `AWS_SECRET_ACCESS_KEY` - AWS secret key
     - `AWS_REGION` - AWS region (us-east-1)
   - Stored in 1Password: "Towerops K8s AWS"

#### Creating Secrets from 1Password

```bash
# Set 1Password account
export OP_ACCOUNT=YOOATCZZSVGH7AD6VABUVPORLI

# Create towerops-secrets
kubectl create secret generic towerops-secrets -n towerops \
  --from-literal=RELEASE_COOKIE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=RELEASE_COOKIE)" \
  --from-literal=SECRET_KEY_BASE="$(op item get '74ske37wsm5tb4mjd4kiphwtji' --vault Private --account=$OP_ACCOUNT --fields label=SECRET_KEY_BASE)"

# Create towerops-db
kubectl create secret generic towerops-db -n towerops \
  --from-literal=POSTGRES_HOST="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=server)" \
  --from-literal=POSTGRES_PORT="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=port)" \
  --from-literal=POSTGRES_DB="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=database)" \
  --from-literal=POSTGRES_USER="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=username)" \
  --from-literal=POSTGRES_PASSWORD="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=password)" \
  --from-literal=DATABASE_URL="$(op item get 'w2rg6bbstm5bxlskcrmn3g3on4' --vault Private --account=$OP_ACCOUNT --fields label=DATABASE_URL)"

# Create towerops-aws
kubectl create secret generic towerops-aws -n towerops \
  --from-literal=AWS_ACCESS_KEY_ID="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_ACCESS_KEY_ID)" \
  --from-literal=AWS_SECRET_ACCESS_KEY="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_SECRET_ACCESS_KEY)" \
  --from-literal=AWS_REGION="$(op item get 's4xwql6bp4huqxvjs6jkx5by6q' --vault Private --account=$OP_ACCOUNT --fields label=AWS_REGION)"
```

### Deployment Process

#### FluxCD Automatic Deployment

The application is automatically deployed via FluxCD GitOps:

1. FluxCD monitors the Git repository for changes
2. When changes are detected, it applies the manifests in `k8s/` directory
3. Kustomize builds the manifests and applies them to the cluster

**GitRepository Configuration:**
```yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: towerops
  namespace: flux-system
spec:
  interval: 1m0s
  url: ssh://git@gitlab.com/graham/towerops.git
  ref:
    branch: main
  secretRef:
    name: towerops-git
```

**Kustomization Configuration:**
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: towerops-app
  namespace: flux-system
spec:
  interval: 1m0s
  path: ./k8s
  prune: true
  sourceRef:
    kind: GitRepository
    name: towerops
  targetNamespace: towerops
```

#### Manual Deployment

If FluxCD is not available or for testing:

```bash
# Apply all resources using kustomize
kubectl apply -k k8s/

# Or apply individually
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/service-headless.yaml
kubectl apply -f k8s/certificate.yaml
kubectl apply -f k8s/ingressroute.yaml
```

### Cluster Setup Checklist

When recreating a Talos cluster from scratch:

- [ ] Install Talos on nodes and bootstrap cluster
- [ ] Install MetalLB and configure IP address pool
- [ ] Install Traefik ingress controller
- [ ] Install cert-manager and configure ClusterIssuers
- [ ] Install FluxCD and configure GitRepository source
- [ ] Install GitLab Agent and connect to GitLab project
- [ ] Install Newt for Pangolin forwarding
- [ ] Install NFS provisioner (if using persistent storage)
- [ ] Create `towerops` namespace
- [ ] Create all required secrets from 1Password
- [ ] Create FluxCD Kustomization resource for `k8s/` directory
- [ ] Verify deployment with `kubectl get pods -n towerops`
- [ ] Check certificate provisioning with `kubectl get certificate -n towerops`
- [ ] Test ingress with `curl https://towerops.net/health`

### Troubleshooting

**FluxCD reconciliation errors:**
- Check kustomization status: `kubectl get kustomization -n flux-system`
- View detailed status: `kubectl describe kustomization towerops-app -n flux-system`
- Force reconciliation: `kubectl annotate kustomization towerops-app -n flux-system reconcile.fluxcd.io/requestedAt="$(date +%Y-%m-%dT%H:%M:%S%z)" --overwrite`

**Certificate issues:**
- Check certificate status: `kubectl describe certificate towerops-net-cert -n towerops`
- View cert-manager logs: `kubectl logs -n cert-manager deployment/cert-manager`
- Verify ClusterIssuer: `kubectl get clusterissuer letsencrypt-prod -o yaml`

**Pod not starting:**
- Check pod status: `kubectl describe pod -n towerops -l app=towerops`
- View logs: `kubectl logs -n towerops deployment/towerops`
- Verify secrets exist: `kubectl get secrets -n towerops`

## Testing Patterns

### SNMP Mocking with Mox

The application uses Mox for SNMP client mocking in tests. Key patterns:

**SNMP Adapter Mock Format**:
- `snmp_adapter().get/3` returns `{:ok, value}` or `{:error, reason}`
- `snmp_adapter().walk/3` returns `{:ok, [%{oid: "...", value: ...}]}` (list of maps, NOT a map)
- Values from `get/3` are already extracted (no type wrapper needed in mocks)

**Mock Expectations**:
```elixir
# Mock get_multiple - it calls get/3 once per OID
expect(SnmpMock, :get, 6, fn _target, oid, _opts ->
  case oid do
    "1.3.6.1.2.1.1.1.0" -> {:ok, "Cisco IOS Software"}  # sysDescr
    "1.3.6.1.2.1.1.2.0" -> {:ok, [1, 3, 6, 1, 4, 1, 9]}  # sysObjectID
    "1.3.6.1.2.1.1.3.0" -> {:ok, 12_345}  # sysUpTime (integer)
    "1.3.6.1.2.1.1.4.0" -> {:ok, "admin@example.com"}  # sysContact
    "1.3.6.1.2.1.1.5.0" -> {:ok, "test-device"}  # sysName
    "1.3.6.1.2.1.1.6.0" -> {:ok, "Test Location"}  # sysLocation
  end
end)

# Mock walk - returns list of OID/value maps
expect(SnmpMock, :walk, fn _target, _oid, _opts ->
  {:ok, [
    %{oid: "1.3.6.1.4.1.9.9.91.1.1.1.1.1.1000", value: 8}
  ]}
end)

# Empty walk result (no data found)
expect(SnmpMock, :walk, fn _, _, _ ->
  {:ok, []}  # NOT {:ok, %{}}
end)
```

**Common Pitfalls**:
- ❌ Returning `{:ok, %{}}` from walk - should be `{:ok, []}`
- ❌ Returning type wrappers like `{:integer, 123}` - values are already extracted
- ❌ Not matching OIDs in get expectations - use pattern matching on OID string
- ❌ Expecting wrong number of calls - `get_multiple/2` calls `get/3` once per OID

**TimescaleDB Tests**:
- Tests that query continuous aggregates should be tagged with `@tag :skip`
- Reason: Test database doesn't have TimescaleDB continuous aggregates configured
- Example: `get_hourly_stats/3`, `get_daily_stats/3`, `get_uptime_percentage/1`

### Test Organization

- Use `DataCase` for tests that need database access
- Use `ConnCase` for controller/LiveView tests
- Use `async: true` for tests that can run in parallel (most unit tests)
- Use `async: false` for tests with shared state (supervisor tests, integration tests)
- never open the test coverage html files
- remember when working in rust to always run cargo fmt before committing
- never try to use npm to install js things, use esbuild built in to phoenix