towerops/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## CRITICAL: Read AGENTS.md First

**Before starting any work in this repository**, you MUST read `AGENTS.md` in the project root.

- `AGENTS.md` contains comprehensive Phoenix/LiveView/Elixir guidelines that are mandatory for this project
- These guidelines take precedence over general development practices when there's a conflict
- Always read `AGENTS.md` at the start of a new conversation or when resuming work
- Follow the project-specific patterns, conventions, and constraints documented there

## Project Overview

Towerops is a Phoenix 1.8 web application built with Elixir, using Ecto for database operations (PostgreSQL), LiveView for real-time interactions, and Tailwind CSS v4 for styling.

### Data Model Relationships

```
┌─────────────┐
│    User     │
│ (Accounts)  │
└──────┬──────┘
       │ owns/member of (many-to-many)
       │
       ↓
┌──────────────────────────────────────────────────────┐
│                   Organization                        │
│                  (Organizations)                      │
└────┬────────────┬─────────────┬──────────────────────┘
     │            │             │
     │ has many   │ has many    │ has many
     │            │             │
     ↓            ↓             ↓
┌─────────┐  ┌──────────┐  ┌────────────────┐
│  Site   │  │Equipment │  │  AgentToken    │
│ (Sites) │  │(Equipment│  │   (Agents)     │
└────┬────┘  └────┬─────┘  └────────┬───────┘
     │            │                  │
     │ has many   │ has one          │ assigned via
     │            │                  │
     ↓            ↓                  ↓
┌──────────┐ ┌──────────────┐  ┌──────────────────┐
│Equipment │ │  SNMPDevice  │  │ AgentAssignment  │◀──┐
│(Equipment│ │(Snmp.Devices)│  │    (Agents)      │   │
└────┬─────┘ └──────────────┘  └──────────────────┘   │
     │                                  │               │
     │ has many                         └───────────────┘
     ├────────────────┬─────────────────┘
     │                │               assigns equipment
     ↓                ↓                  to agent
┌──────────────┐ ┌─────────┐
│MonitoringCheck│ │ Alert   │
│ (Monitoring) │ │ (Alerts)│
└──────────────┘ └─────────┘

Key Relationships:
- User can own/belong to multiple Organizations
- Organization has default_agent_token_id (optional)
- Equipment belongs to both Site and Organization (denormalized)
- Equipment can be assigned to one AgentToken via AgentAssignment
- Equipment has one SNMPDevice with Sensors and Interfaces
- Equipment has many MonitoringChecks (polling results)
- Equipment has many Alerts (equipment_down, equipment_up)
- AgentToken authenticates remote agents for local SNMP polling
```

**Note to AI assistants**: When you make changes to the data model (migrations, schema additions, new contexts), update this diagram to reflect the current state.

## Essential Commands

### Setup and Development
- `mix setup` - Install dependencies, create/migrate database, setup and build assets
- `mix phx.server` - Start the Phoenix server (accessible at http://localhost:4000)
- `iex -S mix phx.server` - Start server with interactive Elixir shell

### Testing and Quality
- `mix test` - Run all tests (automatically creates and migrates test database)
- `mix test test/path/to/specific_test.exs` - Run a specific test file
- `mix test --failed` - Re-run only previously failed tests
- `mix test --cover` - Run tests with coverage report (view: `open cover/modules_*.html`)
- `mix precommit` - **Run before committing**: compiles with warnings as errors, unlocks unused deps, formats code, and runs tests
- `mix dialyzer` - Run static type analysis

**Coverage Notes**:
- Target: 90% minimum
- Protobuf-generated modules (Towerops.Agent.*) show 0% but don't need tests
- TimescaleDB aggregate tests should be tagged `@tag :skip` (test DB lacks continuous aggregates)

### Database
- `mix ecto.create` - Create the database
- `mix ecto.migrate` - Run pending migrations
- `mix ecto.reset` - Drop, create, and migrate database
- `mix ecto.gen.migration migration_name_using_underscores` - Generate a new migration file

### Assets
- `mix assets.build` - Build CSS and JS assets (Tailwind + esbuild)
- `mix assets.deploy` - Build minified assets for production

## Architecture

### Application Structure

Standard Phoenix conventions with separation between business logic (`lib/towerops/`) and web interface (`lib/towerops_web/`).

**Key Configuration**:
- **Binary IDs**: Generators use binary (UUID) primary keys by default (`binary_id: true`)
- **Timestamps**: Use `:utc_datetime` for all timestamps
- **Web server**: Bandit adapter (not Cowboy)
- **HTTP client**: `:req` library (Req module) - this is the only approved HTTP client
- **Ecto repos**: `[Towerops.Repo]`

### Web Layer Patterns

All LiveViews, LiveComponents, and HTML modules automatically get these imports/aliases via `html_helpers/0`:
- `ToweropsWeb.CoreComponents` - Core UI components (`<.button>`, `<.input>`, `<.form>`, etc.)
- `ToweropsWeb.Layouts` - Layout components
- `Phoenix.LiveView.JS` - Client-side JS commands
- Gettext for translations
- Verified routes with `~p` sigil

### Rate Limiting

Uses Hammer (ETS-backed) for rate limiting to prevent brute-force attacks.

**Endpoints Protected:**
- **Auth endpoints** (10 req/min per IP): `/users/log-in`, `/users/register`, `/users/reset-password`, TOTP verification, mobile auth QR flows
- **API v1 endpoints** (1000 req/min per IP): `/api/v1/*` - relaxed for Terraform/automation
- **Admin API endpoints**: Not rate limited (superuser only, different attack profile)

**Implementation:**
- Plug: `ToweropsWeb.Plugs.RateLimit` with `:auth` or `:api` type
- Uses client's real IP from `X-Forwarded-For` or `X-Real-IP` headers (for proxy/load balancer)
- Returns `429 Too Many Requests` with `Retry-After` header when limit exceeded

**Configuration:**
- Disabled in test environment via `config :towerops, :rate_limiting_enabled, false`
- ETS backend configured in `config/config.exs`
- Hammer process started in `application.ex` supervision tree

### Asset Pipeline

- **Tailwind CSS v4**: Uses new `@import "tailwindcss"` syntax in `assets/css/app.css`, no `tailwind.config.js` needed
- **esbuild**: Bundles JavaScript from `assets/js/app.js`
- All vendor scripts/styles must be imported into app.js/app.css - cannot reference external src/href in layouts
- No inline `<script>` tags in templates - use colocated LiveView hooks instead
- Assets are rebuilt on save automatically

### Development Environment

- Dev routes: LiveDashboard at `/dev/dashboard`, Swoosh mailbox at `/dev/mailbox`
- Phoenix LiveReload watches for file changes
- Code reloader enabled

### Custom Ecto Types

Uses custom Ecto types to create rich domain objects and avoid "primitive obsession" - the anti-pattern of representing domain concepts as primitive types (strings, integers) instead of structured data.

**Pattern**: Implement `Ecto.Type` behavior with 4 required callbacks:
- `type/0` - Database storage type (e.g., `:string`, `:jsonb`)
- `cast/1` - Transform user input to domain struct (validation happens here)
- `load/1` - Convert database value to domain struct
- `dump/1` - Convert domain struct back to database primitive

**Benefits**:
- **Type Safety**: Work with `%IPAddress{}` instead of raw strings
- **Centralized Validation**: One place for format checking and normalization
- **Automatic Conversion**: Happens transparently in changesets
- **Rich Domain Behavior**: Pattern matching, helper functions, computed properties
- **Zero Migration Cost**: Column types stay as primitives (varchar, jsonb, etc.)

**Available Custom Types**:

1. **`Towerops.EctoTypes.IpAddress`** - IPv4/IPv6 address handling
   - Validates format using `:inet.parse_address/1`
   - Struct: `%IPAddress{address: String.t, version: :ipv4 | :ipv6, tuple: ip_tuple}`
   - Helpers: `ipv4?/1`, `ipv6?/1`, `to_tuple/1`, `to_string/1`
   - Handles edge cases: nil, IPv6 zone IDs, rejects CIDR notation

2. **`Towerops.EctoTypes.EmailAddress`** - Normalized email addresses _(Planned - Phase 2)_
   - Normalizes to lowercase
   - Splits into local + domain parts
   - Consistent validation across all user-facing schemas

3. **`Towerops.EctoTypes.MacAddress`** - MAC address handling _(Planned - Phase 3)_
   - Handles multiple input formats (colon, hyphen, dot-separated)
   - Normalizes to colon-separated lowercase
   - SNMP binary conversion support

**Usage Example**:

```elixir
# Schema definition
defmodule Towerops.Devices.Device do
  use Ecto.Schema

  schema "devices" do
    field :name, :string
    field :ip_address, Towerops.EctoTypes.IpAddress  # Custom type
  end
end

# Changeset - validation handled automatically by custom type
def changeset(device, attrs) do
  device
  |> cast(attrs, [:name, :ip_address])
  |> validate_required([:name, :ip_address])
  # No need for manual IP validation - handled in IPAddress.cast/1
end

# Pattern matching in functions
def ipv4_only?(%Device{ip_address: %IPAddress{version: :ipv4}}), do: true
def ipv4_only?(_), do: false
```

**When to Create a Custom Type**:

✅ **Good candidates**:
- Domain concepts with specific validation rules (emails, phone numbers, URLs)
- Values that benefit from normalization (case-insensitive emails, MAC addresses)
- Data with multiple representations (IP addresses as strings vs tuples)
- Types that need computed properties (IP version derived from format)

❌ **Avoid for**:
- Simple strings with no validation (names, descriptions)
- Values that are truly primitive (booleans, simple integers)
- One-off validations (use changeset validation instead)

**Implementation Pattern**:

```elixir
defmodule Towerops.EctoTypes.YourType do
  use Ecto.Type

  # Define your domain struct
  defstruct [:field1, :field2]

  @impl Ecto.Type
  def type, do: :string  # Database storage type

  @impl Ecto.Type
  def cast(nil), do: {:ok, nil}
  def cast(%__MODULE__{} = value), do: {:ok, value}  # Idempotence
  def cast(value) when is_binary(value) do
    # Validation and transformation logic here
    case validate_and_parse(value) do
      {:ok, parsed} -> {:ok, struct(__MODULE__, parsed)}
      {:error, _} -> :error
    end
  end
  def cast(_), do: :error

  @impl Ecto.Type
  def load(value) when is_binary(value), do: cast(value)
  def load(nil), do: {:ok, nil}

  @impl Ecto.Type
  def dump(%__MODULE__{} = struct), do: {:ok, struct.field1}  # Extract primitive
  def dump(nil), do: {:ok, nil}
  def dump(_), do: :error

  # Helper functions
  def new(value), do: cast(value)
  def new!(value) do
    case cast(value) do
      {:ok, result} -> result
      :error -> raise ArgumentError, "invalid value"
    end
  end
end
```

**Migration Notes**:

Adopting custom types requires NO database migrations:

1. Create custom type module in `lib/towerops/ecto_types/`
2. Create comprehensive tests in `test/towerops/ecto_types/`
3. Change schema field type: `field :ip_address, :string` → `field :ip_address, Towerops.EctoTypes.IpAddress`
4. Remove manual validation from changeset (now handled in `cast/1`)
5. Update tests to assert on struct fields instead of raw strings
6. Database column stays as varchar/jsonb - Ecto handles conversion transparently

**Testing Pattern**:

```elixir
defmodule Towerops.EctoTypes.YourTypeTest do
  use ExUnit.Case, async: true

  alias Towerops.EctoTypes.YourType

  describe "type/0" do
    test "returns storage type" do
      assert YourType.type() == :string
    end
  end

  describe "cast/1" do
    test "accepts valid values" do
      assert {:ok, %YourType{}} = YourType.cast("valid-value")
    end

    test "rejects invalid values" do
      assert :error = YourType.cast("invalid")
    end

    test "handles nil" do
      assert {:ok, nil} = YourType.cast(nil)
    end
  end

  describe "roundtrip" do
    test "cast -> dump -> load preserves value" do
      {:ok, original} = YourType.cast("value")
      {:ok, dumped} = YourType.dump(original)
      {:ok, loaded} = YourType.load(dumped)
      assert loaded == original
    end
  end
end
```

### Background Job Architecture (Oban)

Uses PostgreSQL-backed Oban for all background processing with cluster-wide coordination.

**Queues:**
- `default` (10 workers) - General background tasks
- `discovery` (10 workers) - SNMP discovery operations
- `pollers` (50 workers) - SNMP polling jobs (one per device)
- `monitors` (50 workers) - Health check jobs (one per device)
- `maintenance` (5 workers) - Periodic cleanup and health check workers

**Key Workers:**
1. **Self-Scheduling Workers** (per-device recurring jobs):
   - `DeviceMonitorWorker` - Health check polling (60s default)
   - `DevicePollerWorker` - SNMP data collection (60s default)
   - Schedule next run before completing, one job per enabled device
   - Automatically created/cancelled when device settings change

2. **Oban Cron Workers** (cluster-wide periodic maintenance):
   - `NeighborCleanupWorker` - Hourly cleanup of stale neighbors/ARP/MAC entries
   - `StaleAgentWorker` - Detects agents that haven't checked in for 10+ minutes (every minute)
   - `AgentLatencyEvaluator` - Latency-based agent reassignment (every 5 minutes)
   - `JobHealthCheckWorker` - Safety net to recover missing monitor/poller jobs (every 10 minutes)

**Resilience:**
- Oban Cron jobs run cluster-wide via PostgreSQL-based locking
- Self-scheduling workers recovered by `JobHealthCheckWorker` every 10 minutes
- All jobs visible in Oban dashboard at `/dev/dashboard` → Oban tab
- Failed jobs automatically retried with exponential backoff

**Configuration:**
- Dev: `config/dev.exs` - Oban config with cron plugin
- Prod: `config/runtime.exs` - Same cron schedule as dev
- Application: `lib/towerops/application.ex` - Oban supervision

**Oban Web Dashboard**:
- Location: `/admin/oban` (superuser access required)
- Vendored dependencies in `vendor/`: `oban_pro`, `oban_web`, `oban_met`
- See `vendor/README.md` for update instructions

### LiveDashboard and Telemetry

Available at `/admin/dashboard` (or `/dev/dashboard` in development).

**Metrics**: Phoenix requests, DB queries, VM stats, Oban queue sizes, Redis/Valkey stats
- Collected every 10 seconds via `telemetry_poller`
- Access: Navigate to dashboard → "Metrics" tab

### SNMP Polling Architecture

Two separate SNMP collection mechanisms:

**1. Discovery (`Towerops.Snmp.Discovery`)** - One-time or manual
- Triggered manually or via `DiscoveryWorker` Oban job
- Collects: system info, device identification, interfaces, sensors, neighbors
- Creates/updates device record and associated data

**2. Polling (`DevicePollerWorker`)** - Continuous time-series
- Runs automatically every 60 seconds (configurable per equipment)
- Collects: sensor readings, interface stats, neighbor topology
- Updates time-series data, keeps neighbor data fresh
- Direct Oban worker (no GenServer layer)

**Load Distribution - Staggered Polling**:
- Uses deterministic hash-based offsets via `PollingOffset.calculate_offset/2`
- Formula: `offset = :erlang.phash2(device_id) |> rem(interval_seconds)`
- Distributes devices evenly across full interval (prevents thundering herd)
- Same device always gets same offset (stable across restarts)
- Applied to both `DevicePollerWorker` and `DeviceMonitorWorker`
- Implementation: `lib/towerops/workers/polling_offset.ex`

### MikroTik API Integration

Provides MikroTik RouterOS API access alongside SNMP for enhanced device management.

**Architecture:**
- **3-tier credential cascade**: Organization → Site → Device (identical to SNMP pattern)
- **Encryption**: Passwords encrypted at rest using Cloak (AES-256-GCM)
- **Detection**: Auto-detected from SNMP manufacturer field (contains "MikroTik" or "RouterOS")
- **Transport**: Supports both API-SSL (port 8729, default) and plain API (port 8728, insecure)
- **Access**: Superuser-only in UI (configurable)
- **Alongside SNMP**: Both can be enabled simultaneously

**Key Files:**
- `lib/towerops/vault.ex` - Cloak encryption vault
- `lib/towerops/ecto_types/encrypted_binary.ex` - Encrypted field type
- `lib/towerops/devices.ex` - `get_mikrotik_config/1`, `propagate_site_mikrotik_change/2`, `propagate_organization_mikrotik_change/2`
- `lib/towerops_web/channels/agent_channel.ex` - `build_mikrotik_job/1`, `build_mikrotik_commands/0`
- `lib/towerops_web/live/device_live/form.ex` - Conditional UI display for MikroTik devices
- `priv/proto/agent.proto` - MikrotikDevice, MikrotikCommand, MikrotikResult protobuf messages

**Database Schema:**
- `organizations.mikrotik_*` - Organization-level defaults
- `sites.mikrotik_*` - Site-level overrides (all nullable)
- `devices.mikrotik_*` - Device-level overrides (all nullable)
- `devices.mikrotik_credential_source` - Tracks inheritance ("device", "site", or default)

**Credential Resolution:**
Each field (username, password, port, use_ssl, enabled) resolves independently:
```elixir
username = device.mikrotik_username ||
           device.site.mikrotik_username ||
           device.site.organization.mikrotik_username
```

**Security:**
- Passwords encrypted at rest using AES-256-GCM via Cloak
- Encryption key must be set in `CLOAK_KEY` environment variable (base64-encoded 32-byte key)
- Generate key: `openssl rand -base64 32`
- Development/test use fixed key in config files (DO NOT use in production)
- **Production**: See "Kubernetes Deployment" section below for secret creation commands
- Plain API (port 8728) is blocked when using cloud pollers (enforced in backend validation)
- UI warns about plain API security risks

**Agent Integration:**
- MikroTik jobs sent via protobuf alongside SNMP jobs
- Agent executes RouterOS API commands (e.g., `/system/identity/print`, `/system/resource/print`)
- Results sent back via `MikrotikResult` protobuf message

### MIB Name Resolution (Rust NIF)

Uses Rust NIF via Rustler for fast SNMP MIB name resolution (replaces unreliable Erlang SNMP modules).

**Architecture:**
- NIF Module: `lib/towerops_native.ex`
- Rust Implementation: `native/towerops_native/src/mib.rs` (uses `snmptranslate` command)
- Build Integration: `mix.exs` - Rustler compiler configured via `rustler_crates`

**How It Works:**
1. Application startup calls `ToweropsNative.load_mib_directory/1` with `priv/mibs` path
2. Client calls `resolve_mib_name/1` before SNMP operations
3. Tries `ToweropsNative.resolve_oid/1` first, falls back to `SnmpKit.resolve/1`
4. Uses `snmptranslate -On` command-line tool with auto-detected system MIB directories

**System Dependencies:**
- **Development** (macOS): `brew install net-snmp` (keg-only, requires PKG_CONFIG_PATH)
- **Production** (Docker): `apt-get install libsnmp-dev snmp-mibs-downloader`
- **Runtime**: `snmptranslate` command must be in PATH

**MIB Files:**
- Located in `priv/mibs/` directory (570+ vendor and standard MIB files)
- Standard MIBs: SNMPv2-MIB, IF-MIB, etc.
- Vendor MIBs: Cisco, Ubiquiti, IEEE, etc.

**Important**:
- SnmpKit still used for SNMP protocol operations (get, walk, get_next, get_bulk)
- NIF only handles MIB name resolution
- Rust compilation happens automatically via Mix
- Clean Rust builds: `cd native/towerops_native && cargo clean`

## Admin Features

### User Impersonation

**Location**: `/admin/users` (superuser only)

**Files**: `lib/towerops_web/live/admin/user_live/index.ex`, `lib/towerops_web/user_auth.ex`

**Notes**:
- Superusers CAN impersonate other superusers (security feature, not bug)
- Backend checks prevent self-impersonation
- All events logged to audit_logs table
- Stop impersonation link appears in user menu when active

### GeoIP Database Management

**Purpose**: Import MaxMind GeoLite2-City database for IP-based country/city detection (GDPR cookie consent).

**Files**: `lib/towerops_web/controllers/api/v1/geoip_controller.ex`, `lib/mix/tasks/geoip.import.ex`

**Import Methods**:
```bash
# Local Import (Development)
make geoip-import DIR=~/Downloads/GeoLite2-City-CSV_20260127/

# Production Import via API (requires TOWEROPS_KEY env var)
make geoip-import-prod DIR=~/Downloads/GeoLite2-City-CSV_20260127/
```

**Implementation**:
- Local mode: Direct database import via Ecto
- Production mode: Processes CSVs locally, sends data to API in 5,000 record batches
- API endpoint: `POST /admin/api/geoip/import` (superuser only)
- Download from: https://dev.maxmind.com/geoip/geolite2-free-geolocation-data

## Project-Specific Constraints

Key constraints from AGENTS.md (see that file for complete details):

- Use `mix precommit` before committing changes
- Use `:req` (Req) for all HTTP requests - never use `:httpoison`, `:tesla`, or `:httpc`
- Never use `daisyUI` - write custom Tailwind components
- LiveView templates must start with `<Layouts.app flash={@flash}>`
- Use `<.icon name="hero-x-mark">` component for icons, never Heroicons modules
- Use LiveView streams for collections to avoid memory issues
- Forms must use `to_form/2` in LiveView and `<.form for={@form}>` in templates
- Never access changesets directly in templates - always use the form assign
- Tailwind v4 uses new import syntax - never use `@apply` in CSS
- When adding/updating hex modules, only use https://hex.pm
- Always run `mix format` after modifying Elixir files (don't specify file, format everything)

### Database Schema Critical Notes

**Binary UUID Primary Keys**: All tables MUST use `:binary_id` for primary keys, not `bigint`.

```elixir
create table(:table_name, primary_key: false) do
  add :id, :binary_id, primary_key: true
  # ...
end
```

**Issue History**: The `snmp_neighbors` table was initially created with `bigint` instead of `:binary_id`, causing `DBConnection.EncodeError`. Always verify actual schema matches migration:
```bash
psql towerops_dev -c "\d table_name"  # Check actual schema
```

**SNMP Binary Data Handling**: All SNMP values must be converted to printable strings before saving. Non-printable binaries should be converted to colon-separated hex format. See `lib/towerops/snmp/neighbor_discovery.ex:sanitize_string_field/1`.

### API Documentation

Available at `/docs/api` (adapted from Tailwind UI Protocol template).

**Files**: `lib/towerops_web/controllers/api_docs_controller.ex`, `lib/towerops_web/controllers/api_docs_html/index.html.heex`

**When updating API endpoints**:
1. Update controller documentation (`@doc` comments)
2. Update `/docs/api` template
3. Wrap code examples in `<%= raw(~S"""...""") %>` to prevent HEEx parsing curly braces

**Current Documented Endpoints**: Sites API, Devices API (GET/POST/PATCH/DELETE)

## Kubernetes Deployment

**Prerequisites**: cert-manager, Traefik, MetalLB, FluxCD, GitLab Agent, NFS Provisioner

**Secrets**: All stored in 1Password, created manually in `towerops` namespace
- `gitlab-registry` - Docker registry credentials
- `towerops-secrets` - RELEASE_COOKIE, SECRET_KEY_BASE, CLOAK_KEY
- `towerops-db` - PostgreSQL connection details
- `towerops-aws` - AWS credentials

### Creating/Updating Secrets

**Initial setup** - Generate and store encryption key:
```bash
# Generate CLOAK_KEY (store in 1Password)
openssl rand -base64 32

# Create towerops-secrets with all required keys
kubectl create secret generic towerops-secrets \
  --from-literal=RELEASE_COOKIE=$(openssl rand -base64 32) \
  --from-literal=SECRET_KEY_BASE=$(mix phx.gen.secret) \
  --from-literal=CLOAK_KEY=$(openssl rand -base64 32) \
  -n towerops
```

**Add CLOAK_KEY to existing secret**:
```bash
# Generate new encryption key
CLOAK_KEY=$(openssl rand -base64 32)

# Store in 1Password first, then add to Kubernetes
kubectl create secret generic towerops-secrets \
  --from-literal=CLOAK_KEY="$CLOAK_KEY" \
  --dry-run=client -o yaml | \
  kubectl apply -f - -n towerops

# Or patch existing secret
kubectl patch secret towerops-secrets \
  -n towerops \
  --type='json' \
  -p="[{'op': 'add', 'path': '/data/CLOAK_KEY', 'value': '$(echo -n "$CLOAK_KEY" | base64)'}]"
```

**Important**:
- Always store the CLOAK_KEY in 1Password before deploying
- Losing the encryption key makes encrypted data (MikroTik passwords, etc.) unrecoverable
- After adding CLOAK_KEY, restart pods: `kubectl rollout restart deployment/towerops -n towerops`

**Deployment**: FluxCD monitors Git repository, automatically applies manifests in `k8s/` directory
**Manual**: `kubectl apply -k k8s/`

**Troubleshooting**:
- FluxCD: `kubectl get kustomization -n flux-system`
- Certificates: `kubectl describe certificate towerops-net-cert -n towerops`
- Pods: `kubectl describe pod -n towerops -l app=towerops`
- Secrets: `kubectl describe secret towerops-secrets -n towerops` (shows keys but not values)

## Common Patterns

### Pagination in LiveView

Simple offset-based pagination using the reusable `<.pagination>` component from `CoreComponents`.

**1. Add pagination metadata to socket assigns in `handle_params` or `apply_action`:**

```elixir
def handle_params(params, _url, socket) do
  page = params |> Map.get("page", "1") |> String.to_integer()
  per_page = 20

  # Fetch all items (or use a paginated query for large datasets)
  all_items = MyContext.list_items(socket.assigns.current_scope.organization.id)
  total_count = length(all_items)
  total_pages = ceil(total_count / per_page)

  # Ensure page is within valid range
  page = max(1, min(page, max(1, total_pages)))

  # Slice items for current page
  offset = (page - 1) * per_page
  items = Enum.slice(all_items, offset, per_page)

  {:noreply,
   socket
   |> assign(:items, items)
   |> assign(:pagination, %{
     page: page,
     per_page: per_page,
     total_count: total_count,
     total_pages: total_pages
   })}
end
```

**2. Use the `<.pagination>` component in your template:**

```heex
<!-- Your list/table here -->
<div id="items">
  <%= for item <- @items do %>
    <!-- item rendering -->
  <% end %>
</div>

<!-- Pagination component -->
<.pagination meta={@pagination} path={~p"/your-path"} />
```

**Preserving query parameters:**

If you need to preserve other query params (like tabs or filters), pass them via the `params` attribute:

```heex
<.pagination
  meta={@pagination}
  path={~p"/devices"}
  params={%{"tab" => @current_tab, "filter" => @filter}}
/>
```

**Component Reference:**

The `<.pagination>` component (defined in `CoreComponents`) accepts:
- `meta` (required) - Map with `:page`, `:per_page`, `:total_count`, `:total_pages`
- `path` (required) - Base path for pagination links (e.g., `~p"/admin"`)
- `params` (optional) - Additional query params to preserve (default: `%{}`)

**Features:**
- Mobile: Simple Previous/Next buttons
- Desktop: Full page numbers with smart ellipsis (shows [1] ... [4] [5] [6] ... [20])
- Automatically builds pagination URLs preserving query params
- Only renders if `total_pages > 1`
- Uses `patch` navigation to avoid full page reload

**Notes**:
- For very large datasets (>10,000 items), use database-level pagination with `limit/offset` instead of in-memory slicing
- The component handles URL building automatically - just provide the base path and params
- See `lib/towerops_web/live/admin/dashboard_live.ex` for a complete example

## Testing Patterns

### SNMP Mocking with Mox

**SNMP Adapter Mock Format**:
- `snmp_adapter().get/3` returns `{:ok, value}` or `{:error, reason}`
- `snmp_adapter().walk/3` returns `{:ok, [%{oid: "...", value: ...}]}` (list of maps, NOT a map)
- Values from `get/3` are already extracted (no type wrapper needed)

**Common Pitfalls**:
- ❌ Returning `{:ok, %{}}` from walk - should be `{:ok, []}`
- ❌ Returning type wrappers like `{:integer, 123}` - values are already extracted
- ❌ Not matching OIDs in get expectations - use pattern matching on OID string

### Test Organization

- Use `DataCase` for database tests, `ConnCase` for controller/LiveView tests
- Use `async: true` for tests that can run in parallel
- Use `async: false` for tests with shared state
- Never open test coverage HTML files
- Never try to use npm to install JS things, use esbuild built in to Phoenix
- When working in Rust, always run `cargo fmt` before committing

## Dialyzer: Static Type Analysis

**Commands:**
- `mix dialyzer` - Run analysis (builds PLT on first run)
- `mix dialyzer --format dialyzer` - Show detailed error locations
- `mix dialyzer --format ignore_file` - Generate `.dialyzer_ignore.exs` for gradual adoption

**Configuration** (`mix.exs`):
```elixir
defp dialyzer do
  [
    plt_add_deps: :apps_direct,  # Faster incremental analysis
    plt_file: {:no_warn, "priv/plts/dialyzer.plt"},
    flags: [:error_handling, :underspecs]
  ]
end
```

**Common Warning Types**:
1. Invalid Contracts - Return type doesn't match all code paths
2. Function Application Arguments - Type alignment issues
3. Opaque Type Mismatches - External module accessing opaque type internals
4. Range Errors - Incomplete specs (e.g., not accounting for empty list case)

**Workflow**:
- Start with broad types, narrow gradually
- Use ignore file for gradual adoption in legacy code
- Combine with runtime type checking for complete safety
- PLT files in `priv/plts/` - don't commit
- Never suppress valid warnings - fix the root cause

## Memory Leak Prevention

### JavaScript Hook Event Listeners

**Problem**: Event listeners added in LiveView hooks without cleanup cause memory leaks when the hook is destroyed.

**Solution**: Always store event listener references and remove them in `destroyed()`:

```typescript
const MyHook = {
  handleClick: null as ((e: Event) => void) | null,
  handleDrag: null as ((e: DragEvent) => void) | null,

  mounted(this: any) {
    // Store reference to handler
    this.handleClick = (e: Event) => {
      // Handler logic
    }

    this.el.addEventListener("click", this.handleClick)
  },

  destroyed(this: any) {
    // Clean up event listeners
    if (this.handleClick) {
      this.el.removeEventListener("click", this.handleClick)
      this.handleClick = null
    }

    // Clean up any other state
    this.someProperty = null
  }
}
```

**Examples Fixed**:
- `CopyToClipboard` - Added `destroyed()` hook to remove click listener
- `DeviceListReorder` - Added `destroyed()` to remove dragstart, dragend, dragover, drop listeners

**Chart.js Cleanup** (already correct):
```typescript
const SensorChart = {
  destroyed() {
    if (this.chart) {
      this.chart.destroy()  // Prevents canvas memory leaks
      this.chart = null
    }
  }
}
```

**Cytoscape Cleanup** (already correct):
```typescript
const NetworkMap = {
  destroyed(this: any) {
    if (this.cy) {
      this.cy.destroy()  // Cleans up Cytoscape instance
      this.cy = null
    }
  }
}
```

### LiveView Assigns vs Streams

**Problem**: Large collections in assigns are kept in memory for the entire LiveView session.

**When to use streams instead of assigns**:
- Lists with hundreds or thousands of items
- Lists that receive frequent updates via PubSub
- Lists where you only need to add/update/remove individual items

**Example - AlertLive.Index should use streams**:

❌ **Current (memory inefficient)**:
```elixir
def mount(_params, _session, socket) do
  {:ok, socket |> assign(:alerts, load_all_alerts())}
end

def handle_info({:new_alert, _device_id, _alert}, socket) do
  # Reloads entire list into memory
  {:noreply, assign(socket, :alerts, load_all_alerts())}
end
```

✅ **Better (use streams)**:
```elixir
def mount(_params, _session, socket) do
  {:ok, socket |> stream(:alerts, load_all_alerts())}
end

def handle_info({:new_alert, _device_id, alert}, socket) do
  # Only adds one alert to the stream
  {:noreply, stream_insert(socket, :alerts, alert, at: 0)}
end

def handle_info({:alert_resolved, _device_id, alert}, socket) do
  # Only removes one alert from the stream
  {:noreply, stream_delete(socket, :alerts, alert)}
end
```

Template change:
```heex
<!-- Old -->
<%= for alert <- @alerts do %>
  <div id={"alert-#{alert.id}"}>{alert.message}</div>
<% end %>

<!-- New with streams -->
<div id="alerts" phx-update="stream">
  <%= for {id, alert} <- @streams.alerts do %>
    <div id={id}>{alert.message}</div>
  <% end %>
</div>
```

### Pagination for Large Lists

**Problem**: Loading thousands of items into memory for display.

**Solution**: Use pagination (see "Pagination in LiveView" section above).

**When pagination is appropriate**:
- Discovered devices (already paginated - 20 per page)
- Audit logs
- Historical alerts
- Any list that could grow to hundreds of items

**Example** (from DeviceLive.Index):
```elixir
# Don't keep all items in assigns
all_items = Context.list_all_items()  # Temporary variable
total_count = length(all_items)

# Only store current page in assigns
items_page = Enum.slice(all_items, offset, per_page)

socket
|> assign(:items, items_page)  # Only current page
|> assign(:pagination, %{...})  # Metadata
```

### Phoenix.PubSub Subscriptions

**Good news**: Phoenix automatically unsubscribes when a LiveView process terminates - no manual cleanup needed.

**Pattern** (already correct in codebase):
```elixir
def mount(_params, _session, socket) do
  if connected?(socket) do
    Phoenix.PubSub.subscribe(Towerops.PubSub, "topic")
  end
  # No need for terminate/2 callback
end
```

### Global Event Listeners (app.ts)

**Safe patterns** (already correct):
- `document.addEventListener('DOMContentLoaded', ...)` - Only fires once, remains for page lifetime
- `window.addEventListener("phx:page-loading-start", ...)` - Global events for the application
- `window.addEventListener("phx:copy", ...)` - Phoenix JS commands

These are acceptable because:
1. They're registered once when the page loads
2. They remain active for the entire page lifetime
3. LiveView patch navigation doesn't reload the page, so no duplicates

### Memory Leak Checklist

When writing new LiveView code or JavaScript hooks:

**LiveView (Elixir)**:
- [ ] Use `stream/3` for large or frequently-updated lists
- [ ] Use pagination for lists that could grow large
- [ ] Avoid storing entire datasets in assigns when only a subset is displayed
- [ ] PubSub subscriptions clean up automatically (no action needed)

**JavaScript Hooks**:
- [ ] Every `addEventListener` in `mounted()` has a matching `removeEventListener` in `destroyed()`
- [ ] Store event handler references (don't use anonymous functions)
- [ ] Clean up third-party library instances (Chart.js, Cytoscape, etc.) in `destroyed()`
- [ ] Set hook properties to `null` in `destroyed()` to release references

**Testing for Memory Leaks**:
- Navigate to a page with a large list
- Use browser DevTools → Memory → Take heap snapshot
- Navigate away and back multiple times
- Take another snapshot
- Compare - memory should not continuously grow
- assets are rebuilt automatically on filesystem change
- never try to run npm, it's not included in phoenix
- assets build themselves on file change, you do not need to ever run mix assets.build