Graham McIntire
ae5ceebf44
- install argocd at home/cluster/argocd/, app-of-apps via root-app.yaml - 4 child Applications: aprs, gridmap, prop, towerops - argo.w5isp.com via cloudflared tunnel (CNAME + ingress_rule in tofu) - update CLAUDE.md: drop K3s/Wavelog/n8n/Node-RED, document GitOps + ArgoCD
249 lines
No EOL
10 KiB
Markdown
249 lines
No EOL
10 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Repository Overview
|
|
|
|
This is a multi-environment infrastructure-as-code repository managing:
|
|
- **Ansible**: Configuration management for servers across multiple environments (VNTX, home, app servers)
|
|
- **OpenTofu**: Proxmox/Talos VM provisioning and Porkbun domain NS management
|
|
- **Home Lab**: Server configurations with Tailscale integration
|
|
|
|
## Common Commands
|
|
|
|
### Proxmox VM Provisioning
|
|
|
|
BIND9 and future infrastructure VMs are provisioned using the main Ansible playbook:
|
|
|
|
```bash
|
|
cd ansible
|
|
source .envrc # Load Proxmox credentials from .envrc
|
|
|
|
# Provision and configure BIND9 nameserver at 204.110.191.222
|
|
ansible-playbook playbook.yml --tags provision,bind9
|
|
|
|
# VM settings (aligned with terraform/variables.tf):
|
|
# - Node: vm2-380 (var.proxmox_host)
|
|
# - Template: debian-13-cloudinit (VMID 9000)
|
|
# - Storage: local-lvm (for BIND9 and general infrastructure VMs)
|
|
# - Network: vmbr0
|
|
#
|
|
# Note: Talos nodes use a different template and Longhorn for persistent storage
|
|
```
|
|
|
|
### Ansible Operations
|
|
```bash
|
|
# Run main playbook
|
|
cd ansible && ./run.sh
|
|
|
|
# Bootstrap new host
|
|
make bootstrap-init HOSTNAME=hostname ANSIBLE_HOST=ip_address
|
|
|
|
# Run specific playbook
|
|
ansible-playbook general.yml
|
|
|
|
# Run with specific tags
|
|
ansible-playbook -t caddy general.yml
|
|
|
|
# Gather facts from all hosts
|
|
make facts
|
|
```
|
|
|
|
### OpenTofu Operations
|
|
**IMPORTANT**: Use OpenTofu (tofu) instead of terraform for all operations.
|
|
|
|
```bash
|
|
cd tofu
|
|
tofu init
|
|
tofu plan
|
|
tofu apply
|
|
|
|
# Format terraform files before committing
|
|
tofu fmt
|
|
```
|
|
|
|
### Talos Operations (Home Cluster)
|
|
```bash
|
|
# Change to talos directory
|
|
cd /Users/graham/dev/infra/talos
|
|
|
|
# Check cluster health
|
|
talosctl --talosconfig talosconfig health
|
|
|
|
# View cluster nodes
|
|
kubectl get nodes
|
|
|
|
# View all pods
|
|
kubectl get pods -A
|
|
|
|
# Check talos services
|
|
talosctl --talosconfig talosconfig service
|
|
|
|
# View logs
|
|
talosctl --talosconfig talosconfig logs -f kubelet
|
|
```
|
|
|
|
|
|
## Architecture
|
|
|
|
### Directory Structure
|
|
- `ansible/` - Ansible playbooks and roles for server configuration
|
|
- `roles/` - Reusable roles (base, caddy, docker, resolvers, etc.)
|
|
- `group_vars/` - Group-specific variables
|
|
- `host_vars/` - Host-specific variables
|
|
- Main playbooks: `playbook.yml`, `general.yml`, `bootstrap.yml`
|
|
|
|
- `tofu/` - OpenTofu infrastructure management
|
|
- Proxmox/Talos VM provisioning (main.tf, talos_vms.tf)
|
|
- Porkbun domain NS management (dns.tf)
|
|
- Provider: `marcfrederick/porkbun ~> 1.3`
|
|
- Credentials via `PORKBUN_API_KEY` and `PORKBUN_SECRET_API_KEY` env vars
|
|
- Manages NS records for all Porkbun domains (as393837, cloudflare, or porkbun default NS)
|
|
- Backend: Local state (committed to repo)
|
|
|
|
- `home/` - Home lab configurations
|
|
- `cluster/` - Kubernetes application deployments
|
|
- `argocd/` - ArgoCD install + Application manifests (app-of-apps via `root-app.yaml`)
|
|
- `aprs/` - APRS.me deployment manifests (legacy; live manifests in the `aprs.me` repo's `k8s/`)
|
|
|
|
### Key Infrastructure Services
|
|
|
|
**Monitoring**: Icinga2 with MariaDB backend
|
|
**DNS**: Unbound resolvers, PowerDNS for reverse zones
|
|
**VPN**: Tailscale integration across environments
|
|
**Web Proxy**: Caddy for reverse proxy and SSL termination
|
|
|
|
### Security Notes
|
|
- Ansible user (UID 10001) with passwordless sudo
|
|
- SSH keys fetched from GitHub for authentication
|
|
- Tailscale requires `TAILSCALE_KEY` environment variable
|
|
- OpenTofu credentials should use environment variables (PORKBUN_API_KEY, PORKBUN_SECRET_API_KEY)
|
|
|
|
## Development Workflow
|
|
|
|
### Adding New Hosts
|
|
1. Add host to `ansible/hosts` inventory
|
|
2. Create host_vars file if needed
|
|
3. Run bootstrap: `make bootstrap-init HOSTNAME=x ANSIBLE_HOST=y`
|
|
4. Apply configuration: `ansible-playbook -l hostname general.yml`
|
|
|
|
### DNS Zone Changes (BIND9)
|
|
1. Edit zone YAML in `ansible/group_vars/bind9_servers/<domain>.yml`
|
|
2. Run ansible playbook to deploy zone changes
|
|
|
|
### Domain NS Changes (Porkbun Registrar)
|
|
1. Edit `tofu/dns.tf` — update the domain's NS in the `local.domains` map
|
|
2. Run `tofu fmt` to format
|
|
3. Review with `tofu plan`
|
|
4. Apply with `tofu apply`
|
|
|
|
|
|
## Testing
|
|
|
|
### Ansible
|
|
```bash
|
|
# Syntax check
|
|
ansible-playbook --syntax-check playbook.yml
|
|
|
|
# Dry run
|
|
ansible-playbook --check playbook.yml
|
|
|
|
# Test on specific host
|
|
ansible-playbook -l hostname playbook.yml
|
|
```
|
|
|
|
### Terraform
|
|
```bash
|
|
tofu validate
|
|
tofu plan
|
|
```
|
|
|
|
|
|
## Recent Infrastructure Updates
|
|
|
|
### Talos Home Cluster (10.0.15.1-6)
|
|
- **Purpose**: Primary home lab Kubernetes cluster
|
|
- **Cluster Name**: home-cluster
|
|
- **Kubernetes Version**: v1.35.0
|
|
- **Control Plane Nodes**: 3 (10.0.15.1-3)
|
|
- **Worker Nodes**: 3 (10.0.15.4-6)
|
|
- **API Endpoint**: https://10.0.15.1:6443
|
|
- **Proxmox Cluster**: "home" (node1, node2, node3 at 10.0.15.101-103)
|
|
- **VM Disks**: Proxmox Ceph for VM boot disks
|
|
- **Persistent Storage**: Longhorn on passthrough NVMe drives (`/dev/sdb` on each worker, mounted at `/var/mnt/longhorn`)
|
|
- Worker1: 2TB, Worker2: 2TB, Worker3: 1TB
|
|
- `longhorn` StorageClass is the default
|
|
- `longhorn-system` namespace requires privileged PodSecurity policy
|
|
- **CNI**: Flannel
|
|
- **Talos Extensions** (workers only):
|
|
- `iscsi-tools` v0.2.0 — required by Longhorn
|
|
- `util-linux-tools` 2.41.2 — required by Longhorn
|
|
- Installer image: `factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.12.6`
|
|
- Control plane nodes use the stock `ghcr.io/siderolabs/installer:v1.12.6`
|
|
- **Deployment**: Managed via OpenTofu in `/tofu/`
|
|
- **Configuration**: `/talos/` directory
|
|
- `talosconfig` - Talosctl client config
|
|
- `secrets.yaml` - Cluster secrets
|
|
- `controlplane.yaml` / `worker.yaml` - Base node configs
|
|
- `configs/talos-worker{1,2,3}.yaml` - Per-node worker configs (with disk/extension settings)
|
|
- `configs/talos-cp{1,2,3}.yaml` - Per-node control plane configs
|
|
- **GitOps**: ArgoCD (replaced Flux on 2026-04-29). See `home/cluster/argocd/` and the GitOps section below.
|
|
- **Applications**: deployed via ArgoCD `Application` manifests in `home/cluster/argocd/apps/`
|
|
- **APRS.me** (namespace `aprs`, https://aprs.me) — sourced from `git.mcintire.me/graham/aprs.me.git` `./k8s`
|
|
- **gridmap** (namespace `gridmap`) — sourced from `git.mcintire.me/graham/gridmap.git` `./k8s`
|
|
- **prop** (namespace `prop`) — sourced from `git.mcintire.me/graham/prop.git` `./k8s`
|
|
- **towerops** (namespace `towerops`) — sourced from `git.mcintire.me/graham/towerops-web.git` `./k8s`
|
|
- **Documentation**: See `/talos/README.md` and `/cluster/README.md` for detailed operations
|
|
- **Config Notes**:
|
|
- `install.extraKernelArgs` and `install.grubUseUKICmdline` cannot be used together
|
|
- Worker disks with existing data (e.g. old Ceph LVM) must be wiped before Talos will partition them: `talosctl reset --user-disks-to-wipe /dev/sdb --wipe-mode user-disks --reboot`
|
|
|
|
### GitOps via ArgoCD (home-cluster)
|
|
- **Install**: `home/cluster/argocd/install/` — upstream ArgoCD manifests + a Kustomize patch enabling `server.insecure=true` (cloudflared terminates TLS) and `reposerver.git.modules.enabled=false` (the `aprs.me` repo has a stale submodule pin that breaks fetch otherwise)
|
|
- **App-of-apps**: `home/cluster/argocd/root-app.yaml` watches `home/cluster/argocd/apps/` in this repo and reconciles all child Applications. Add a new YAML to `apps/` and push — Argo picks it up.
|
|
- **Repo creds**: `forgejo-mcintire-creds` Secret in `argocd` ns is a credential template for `https://git.mcintire.me/graham` — username/password copied from `flux-system/forgejo-git-credentials`.
|
|
- **UI**: `argo.w5isp.com` via cloudflared tunnel → `argocd-server.argocd.svc.cluster.local:80`. Initial admin password is in `argocd/argocd-initial-admin-secret`.
|
|
- **Image auto-update**: Flux's `ImagePolicy`/`ImageUpdateAutomation` is gone. Annotations like `# {"$imagepolicy": "flux-system:prop"}` in app repo manifests are inert. To restore, install `argocd-image-updater`.
|
|
|
|
### Caddy server: w5isp.w5isp.com (204.110.191.200)
|
|
- **Purpose**: Single-node server running Debian 13
|
|
- **Features**:
|
|
- Web services with automatic SSL
|
|
- Single public IP handles all services
|
|
|
|
### DNS Configuration
|
|
- **Photos**: photos.w5isp.com → 204.110.191.212 (Caddy) → 100.107.11.77:2283
|
|
- **Mailcow**: mcintire.me mail records → mail.w5isp.com (sync.w5isp.com at 204.110.191.216)
|
|
- Full mail setup with MX, SPF, DKIM, DMARC records
|
|
- Autodiscover/Autoconfig for mail clients
|
|
- Backup MX: mail.nsnw.ca (priority 20)
|
|
- **Note**: sync.w5isp.com is the Mailcow server (not using Caddy proxy)
|
|
|
|
## Code Guidelines
|
|
|
|
### Communication Practices
|
|
- Do not ever reply with "you're right". Just fix the issue
|
|
- Never say you're right, just accept it and move on
|
|
|
|
### Notes for Claude
|
|
- As you learn new things, keep claude.md updated
|
|
|
|
## Cluster
|
|
- **home-cluster** (Talos): 10.0.15.1-6 — only Kubernetes cluster. External traffic enters via the `cloudflared` tunnel deployment in the `cloudflared` namespace; per-hostname routing rules live in Cloudflare's dashboard, not in the cluster.
|
|
|
|
## Application Deployment Patterns
|
|
|
|
### Standard App Deployment Process
|
|
1. App repos contain their own k8s manifests in `./k8s/`
|
|
2. ArgoCD `Application` in `home/cluster/argocd/apps/<name>.yaml` points at the repo
|
|
3. Push the app repo → Argo auto-syncs (when `automated.selfHeal=true` is set)
|
|
4. Use Longhorn for persistent volumes (default StorageClass)
|
|
5. Use ClusterIP services + Cloudflare tunnel for external access
|
|
|
|
### Common Issues & Solutions
|
|
- **PodSecurity violations**: Set namespace to privileged: `kubectl label namespace [ns] pod-security.kubernetes.io/enforce=privileged --overwrite`
|
|
- **Database connectivity**: Use short service names (`postgres` not `postgres.namespace.svc.cluster.local`)
|
|
- **Init container permissions**: Required for volume ownership fixes
|
|
- **Service DNS**: Services accessible via `service-name` within same namespace
|
|
- **Argo can't fetch a repo with a submodule**: confirm `reposerver.git.modules.enabled=false` is set on `argocd-cmd-params-cm` (the `aprs.me` repo had a stale submodule pin that broke clones)
|
|
- **Argo refuses repo for "out-of-bounds symlinks"**: the repo has a tracked symlink pointing outside the working tree (e.g. into `/nix/store/...`). Fix in the source repo — Argo has no override. |