infra/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Repository Overview

This is a multi-environment infrastructure-as-code repository managing:
- **Ansible**: Configuration management for servers across multiple environments (VNTX, home, app servers)
- **OpenTofu**: Proxmox/Talos VM provisioning and Porkbun domain NS management
- **Home Lab**: Server configurations with Tailscale integration

## Common Commands

### Proxmox VM Provisioning

Infrastructure VMs are provisioned using OpenTofu and configured via Butane/Ignition (Flatcar) or Ansible (traditional Linux):

```bash
# Infrastructure VMs are managed by OpenTofu (tofu/proxmox.tf)
cd tofu && tofu plan && tofu apply

# Post-provision configuration via Ansible
cd ansible && source .envrc

# Configure BIND9 nameserver at 204.110.191.222
ansible-playbook playbook.yml --tags provision,bind9
```

**Flatcar Linux VMs** (managed by `lucidsolns/flatcar-vm/proxmox` module):
- No manual template needed — module downloads the Flatcar image automatically
- VM definition: `tofu/proxmox.tf` (module `flatcar_resolver`, VMID 110)
- Configuration: `tofu/flatcar-resolver.bu` (Butane → Ignition)
- Config files: `tofu/unbound.conf` (rendered from group_vars), `tofu/blocklist.rpz`
- Static IP: 10.0.0.30/24, 8GB RAM, 2 cores, UEFI boot
- Default user: `core` (passwordless sudo)
- Unbound runs as a Docker container (`mvance/unbound`) with config + blocklist mounted
- DoH endpoint on 127.0.0.1:8080 for future Caddy proxy
- To deploy: update `flatcar-resolver.bu` or config files, then `tofu apply`
- Proxmox storage `local` must have `snippets` and `import` content types enabled (`/etc/pve/storage.cfg`)
```

### Ansible Operations
```bash
# Run main playbook
cd ansible && ./run.sh

# Bootstrap new host
make bootstrap-init HOSTNAME=hostname ANSIBLE_HOST=ip_address

# Run specific playbook
ansible-playbook general.yml

# Run with specific tags
ansible-playbook -t caddy general.yml

# Gather facts from all hosts
make facts
```

### OpenTofu Operations
**IMPORTANT**: Use OpenTofu (tofu) instead of terraform for all operations.

```bash
cd tofu
tofu init
tofu plan
tofu apply

# Format terraform files before committing
tofu fmt
```

### Talos Operations (Home Cluster)
```bash
# Change to talos directory
cd /Users/graham/dev/infra/talos

# Check cluster health
talosctl --talosconfig talosconfig health

# View cluster nodes
kubectl get nodes

# View all pods
kubectl get pods -A

# Check talos services
talosctl --talosconfig talosconfig service

# View logs
talosctl --talosconfig talosconfig logs -f kubelet
```


## Architecture

### Directory Structure
- `ansible/` - Ansible playbooks and roles for server configuration
  - `roles/` - Reusable roles (base, caddy, docker, resolvers, etc.)
  - `group_vars/` - Group-specific variables
  - `host_vars/` - Host-specific variables
  - Main playbooks: `playbook.yml`, `general.yml`, `bootstrap.yml`

- `tofu/` - OpenTofu infrastructure management
  - Proxmox/Talos VM provisioning (main.tf, talos_vms.tf)
  - Porkbun domain NS management (dns.tf)
    - Provider: `marcfrederick/porkbun ~> 1.3`
    - Credentials via `PORKBUN_API_KEY` and `PORKBUN_SECRET_API_KEY` env vars
    - Manages NS records for all Porkbun domains (as393837, cloudflare, or porkbun default NS)
  - Backend: Local state (committed to repo)

- `home/` - Home lab configurations
  - `cluster/` - Kubernetes application deployments
    - `argocd/` - ArgoCD install + Application manifests (app-of-apps via `root-app.yaml`)
    - `aprs/` - APRS.me deployment manifests (legacy; live manifests in the `aprs.me` repo's `k8s/`)

### Key Infrastructure Services

**Monitoring**: Icinga2 with MariaDB backend
**DNS**: Unbound resolvers, PowerDNS for reverse zones
**VPN**: Tailscale integration across environments
**Web Proxy**: Caddy for reverse proxy and SSL termination

### Security Notes
- Ansible user (UID 10001) with passwordless sudo
- SSH keys fetched from GitHub for authentication
- Tailscale requires `TAILSCALE_KEY` environment variable
- OpenTofu credentials should use environment variables (PORKBUN_API_KEY, PORKBUN_SECRET_API_KEY)

## Development Workflow

### Adding New Hosts
1. Add host to `ansible/hosts` inventory
2. Create host_vars file if needed
3. Run bootstrap: `make bootstrap-init HOSTNAME=x ANSIBLE_HOST=y`
4. Apply configuration: `ansible-playbook -l hostname general.yml`

### DNS Zone Changes (BIND9)
1. Edit zone YAML in `ansible/group_vars/bind9_servers/<domain>.yml`
2. Run ansible playbook to deploy zone changes

### Domain NS Changes (Porkbun Registrar)
1. Edit `tofu/dns.tf` — update the domain's NS in the `local.domains` map
2. Run `tofu fmt` to format
3. Review with `tofu plan`
4. Apply with `tofu apply`


## Testing

### Ansible
```bash
# Syntax check
ansible-playbook --syntax-check playbook.yml

# Dry run
ansible-playbook --check playbook.yml

# Test on specific host
ansible-playbook -l hostname playbook.yml
```

### Terraform
```bash
tofu validate
tofu plan
```


## Recent Infrastructure Updates

### Talos Home Cluster (10.0.19.1-6)
- **Purpose**: Primary home lab Kubernetes cluster
- **Cluster Name**: home-cluster
- **Kubernetes Version**: v1.36.1
- **Control Plane Nodes**: 3 (10.0.19.1-3)
- **Worker Nodes**: 3 (10.0.19.4-6)
- **API Endpoint**: https://10.0.19.10:6443
- **Proxmox Cluster**: "home" (node1, node2, node3 at 10.0.19.101-103)
- **VM Disks**: local-lvm on each Proxmox node
- **Persistent Storage**: TBD (Longhorn was removed — evaluating alternatives)
- **CNI**: Flannel
- **Talos Extensions** (workers only):
  - `iscsi-tools` v0.2.0
  - `util-linux-tools` 2.41.2
- Installer image: `factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.13.3`
- Control plane nodes use the stock `ghcr.io/siderolabs/installer:v1.13.3`
- **Deployment**: Managed via OpenTofu in `/tofu/`
- **Configuration**: `/talos/` directory
  - `talosconfig` - Talosctl client config
  - `secrets.yaml` - Cluster secrets
  - `controlplane.yaml` / `worker.yaml` - Base node configs
  - `worker.yaml` / `controlplane.yaml` - Base node configs (applied with per-node patches)
  - `patches/*.yaml` - Per-node patches (hostname, disk overrides)
- **GitOps**: ArgoCD (replaced Flux on 2026-04-29). See `home/cluster/argocd/` and the GitOps section below.
- **Applications**: deployed via ArgoCD `Application` manifests in `home/cluster/argocd/apps/`
  - **APRS.me** (namespace `aprs`, https://aprs.me) — sourced from `codeberg.org/gmcintire/aprs.me.git` `./k8s`
  - **gridmap** (namespace `gridmap`) — sourced from `codeberg.org/gmcintire/gridmap.git` `./k8s`
  - **prop** (namespace `prop`) — sourced from `codeberg.org/gmcintire/prop.git` `./k8s`
  - **towerops** (namespace `towerops`) — sourced from `codeberg.org/gmcintire/towerops-web.git` `./k8s`
- **Documentation**: See `/talos/README.md` and `/cluster/README.md` for detailed operations
- **Config Notes**:
  - `install.extraKernelArgs` and `install.grubUseUKICmdline` cannot be used together
  - Worker disks with existing data (e.g. old Ceph LVM) must be wiped before Talos will partition them: `talosctl reset --user-disks-to-wipe /dev/sdb --wipe-mode user-disks --reboot`

### GitOps via ArgoCD (home-cluster)
- **Install**: `home/cluster/argocd/install/` — upstream ArgoCD manifests + a Kustomize patch enabling `server.insecure=true` (cloudflared terminates TLS) and `reposerver.git.modules.enabled=false` (the `aprs.me` repo has a stale submodule pin that breaks fetch otherwise)
- **App-of-apps**: `home/cluster/argocd/root-app.yaml` watches `home/cluster/argocd/apps/` in this repo and reconciles all child Applications. Add a new YAML to `apps/` and push — Argo picks it up.
- **Repo creds**: `forgejo-mcintire-creds` Secret in `argocd` ns is a credential template for `https://codeberg.org/gmcintire` — username/password copied from `flux-system/forgejo-git-credentials`.
- **UI**: `argo.w5isp.com` via cloudflared tunnel → `argocd-server.argocd.svc.cluster.local:80`. Initial admin password is in `argocd/argocd-initial-admin-secret`.
- **Image auto-update**: Flux's `ImagePolicy`/`ImageUpdateAutomation` is gone. Annotations like `# {"$imagepolicy": "flux-system:prop"}` in app repo manifests are inert. To restore, install `argocd-image-updater`.

### Caddy server: w5isp.w5isp.com (204.110.191.200)
- **Purpose**: Single-node server running Debian 13
- **Features**:
  - Web services with automatic SSL
  - Single public IP handles all services

### DNS Configuration
- **Photos**: photos.w5isp.com → 204.110.191.212 (Caddy) → 100.107.11.77:2283
- **Mailcow**: mcintire.me mail records → mail.w5isp.com (sync.w5isp.com at 204.110.191.216)
  - Full mail setup with MX, SPF, DKIM, DMARC records
  - Autodiscover/Autoconfig for mail clients
  - Backup MX: mail.nsnw.ca (priority 20)
  - **Note**: sync.w5isp.com is the Mailcow server (not using Caddy proxy)

## Code Guidelines

### Communication Practices
- Do not ever reply with "you're right". Just fix the issue
- Never say you're right, just accept it and move on

### Notes for Claude
- As you learn new things, keep claude.md updated

## Cluster
- **home-cluster** (Talos): 10.0.19.1-6 — only Kubernetes cluster. External traffic enters via the `cloudflared` tunnel deployment in the `cloudflared` namespace; per-hostname routing rules live in Cloudflare's dashboard, not in the cluster.

## Application Deployment Patterns

### Standard App Deployment Process
1. App repos contain their own k8s manifests in `./k8s/`
2. ArgoCD `Application` in `home/cluster/argocd/apps/<name>.yaml` points at the repo
3. Push the app repo → Argo auto-syncs (when `automated.selfHeal=true` is set)
4. Use a CSI storage driver for persistent volumes (set default StorageClass)
5. Use ClusterIP services + Cloudflare tunnel for external access

### Common Issues & Solutions
- **PodSecurity violations**: Set namespace to privileged: `kubectl label namespace [ns] pod-security.kubernetes.io/enforce=privileged --overwrite`
- **Database connectivity**: Use short service names (`postgres` not `postgres.namespace.svc.cluster.local`)
- **Init container permissions**: Required for volume ownership fixes
- **Service DNS**: Services accessible via `service-name` within same namespace
- **Argo can't fetch a repo with a submodule**: confirm `reposerver.git.modules.enabled=false` is set on `argocd-cmd-params-cm` (the `aprs.me` repo had a stale submodule pin that broke clones)
- **Argo refuses repo for "out-of-bounds symlinks"**: the repo has a tracked symlink pointing outside the working tree (e.g. into `/nix/store/...`). Fix in the source repo — Argo has no override.