graham/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

update

Browse source
This commit is contained in:
Graham McIntire 2025-10-01 17:06:51 -05:00
parent d5e3ce88a3
commit 8e4a9830bc
No known key found for this signature in database
179 changed files with 1952 additions and 9033 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

108
CLAUDE.md
View file

@ -7,8 +7,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
This is a multi-environment infrastructure-as-code repository managing: This is a multi-environment infrastructure-as-code repository managing:
- **Ansible**: Configuration management for servers across multiple environments (VNTX, home, app servers) - **Ansible**: Configuration management for servers across multiple environments (VNTX, home, app servers)
- **Terraform**: DNS infrastructure management across DNSimple, Porkbun, and PowerDNS - **Terraform**: DNS infrastructure management across DNSimple, Porkbun, and PowerDNS
- **Kubernetes**: K3s cluster deployments on Proxmox with Flatcar/Debian - **Home Lab**: Server configurations with Tailscale integration
- **Home Lab**: K3s configurations with Tailscale integration
## Common Commands ## Common Commands
@ -43,41 +42,39 @@ tofu apply
tofu fmt tofu fmt
``` ```
### K3s Cluster Deployment (VNTX) ### Kubernetes Operations (K3s cluster at 204.110.191.2)
```bash ```bash
cd clusters/vntx # Set kubeconfig
export TF_VAR_proxmox_password="your-password" export KUBECONFIG=/Users/graham/dev/infra/home/ansible/kubeconfig
# Deploy APRS.me application
cd home/cluster/aprs
./deploy.sh ./deploy.sh
# Manual steps if needed: # Check deployment status
cd terraform && tofu apply kubectl get pods -n aprs
cd ../ansible && ansible-playbook -i inventory.yml k3s-install.yml kubectl get svc -n aprs
export KUBECONFIG=./kubeconfig
kubectl get nodes
``` ```
## Architecture ## Architecture
### Directory Structure ### Directory Structure
- `ansible/` - Ansible playbooks and roles for server configuration - `ansible/` - Ansible playbooks and roles for server configuration
- `roles/` - Reusable roles (base, caddy, docker, k3s, resolvers, etc.) - `roles/` - Reusable roles (base, caddy, docker, resolvers, etc.)
- `group_vars/` - Group-specific variables - `group_vars/` - Group-specific variables
- `host_vars/` - Host-specific variables - `host_vars/` - Host-specific variables
- Main playbooks: `playbook.yml`, `general.yml`, `bootstrap.yml` - Main playbooks: `playbook.yml`, `general.yml`, `bootstrap.yml`
- `terraform/` - DNS infrastructure management - `home/terraform/` - DNS infrastructure management
- Manages domains: w5isp.com, vntx.net/org, manero.org, mcintire.me, gridmap.org - Manages domains: w5isp.com, vntx.net/org, manero.org, mcintire.me, gridmap.org
- Providers: DNSimple, Porkbun, PowerDNS (reverse DNS) - Providers: DNSimple, Porkbun, PowerDNS (reverse DNS)
- `clusters/vntx/` - K3s cluster on Proxmox - `home/` - Home lab configurations
- `terraform/` - VM provisioning on Proxmox - `cluster/` - Kubernetes application deployments
- `ansible/` - K3s installation and configuration - `aprs/` - APRS.me deployment with PostgreSQL/PostGIS and Redis
- `infrastructure/` - Services like Icinga2, FreeRADIUS, Unbound - `wavelog/` - Wavelog amateur radio logging application
- `system/` - Core services: MetalLB, Longhorn, cert-manager - `terraform/` - DNS infrastructure management
- `home/` - Home lab K3s configurations
- `k3s/` - K3s service definitions with Tailscale integration
- `cluster/` - Application deployments (Forgejo, Node-RED, etc.)
### Key Infrastructure Services ### Key Infrastructure Services
@ -85,8 +82,6 @@ kubectl get nodes
**DNS**: Unbound resolvers, PowerDNS for reverse zones **DNS**: Unbound resolvers, PowerDNS for reverse zones
**VPN**: Tailscale integration across environments **VPN**: Tailscale integration across environments
**Web Proxy**: Caddy for reverse proxy and SSL termination **Web Proxy**: Caddy for reverse proxy and SSL termination
**Storage**: Longhorn distributed storage for K3s
**Load Balancing**: MetalLB for bare-metal K3s
### Security Notes ### Security Notes
- Ansible user (UID 10001) with passwordless sudo - Ansible user (UID 10001) with passwordless sudo
@ -103,17 +98,11 @@ kubectl get nodes
4. Apply configuration: `ansible-playbook -l hostname general.yml` 4. Apply configuration: `ansible-playbook -l hostname general.yml`
### DNS Changes ### DNS Changes
1. Edit appropriate `terraform/dns_*.tf` file 1. Edit appropriate `home/terraform/dns_*.tf` file
2. Run `terraform fmt` to format 2. Run `tofu fmt` to format
3. Review with `terraform plan` 3. Review with `tofu plan`
4. Apply with `terraform apply` 4. Apply with `tofu apply`
### K3s Service Deployment
1. Create namespace if needed
2. Deploy storage (PVC) if stateful
3. Create ConfigMaps/Secrets
4. Deploy application
5. Configure Service/Ingress for access
## Testing ## Testing
@ -131,12 +120,51 @@ ansible-playbook -l hostname playbook.yml
### Terraform ### Terraform
```bash ```bash
terraform validate tofu validate
terraform plan tofu plan
``` ```
### Kubernetes
```bash ## Recent Infrastructure Updates
kubectl apply --dry-run=client -f manifest.yaml
kubectl diff -f manifest.yaml ### K3s Cluster at 204.110.191.2
``` - **Purpose**: Home lab Kubernetes cluster
- **Context**: Use `kubectl config use-context homelab` to access
- **Storage**: Longhorn for persistent volumes
- **Applications**:
- **APRS.me**: Amateur radio APRS tracking application (https://aprs.me)
- PostgreSQL 17 with PostGIS (100GB Longhorn PVC)
- Redis for caching
- 2-replica StatefulSet for high availability
- Erlang clustering enabled
- LoadBalancer service assigned: 10.0.101.32
- **Note**: ghcr.io/aprsme/aprs.me:latest requires authentication
- Deploy with: `cd home/cluster/aprs && ./deploy.sh`
- **Wavelog**: Amateur radio logging application (https://log.w5isp.com)
- MariaDB 11 database (10GB storage)
- 20GB storage for configs, backups, uploads, QSL cards
- PHP application with custom limits (50MB uploads, 256MB memory)
- Deploy with: `cd home/cluster/wavelog && ./deploy.sh`
### New Server: w5isp.w5isp.com (204.110.191.200)
- **Purpose**: Single-node server running Debian 13
- **Features**:
- Web services with automatic SSL
- Single public IP handles all services
### DNS Configuration
- **Photos**: photos.w5isp.com → 204.110.191.212 (Caddy) → 100.107.11.77:2283
- **Mailcow**: mcintire.me mail records → mail.w5isp.com (sync.w5isp.com at 204.110.191.216)
- Full mail setup with MX, SPF, DKIM, DMARC records
- Autodiscover/Autoconfig for mail clients
- Backup MX: mail.nsnw.ca (priority 20)
- **Note**: sync.w5isp.com is the Mailcow server (not using Caddy proxy)
## Code Guidelines
### Communication Practices
- Do not ever reply with "you're right". Just fix the issue
- Never say you're right, just accept it and move on
## Clusters
- k3s cluster at 204.110.191.2

233
ansible/K8S-DEPLOYMENT.md
View file

@ -1,233 +0,0 @@
# Kubernetes/K3s Cluster Deployment on Proxmox
This guide helps you deploy either a full Kubernetes cluster or a lightweight K3s cluster on Proxmox using Debian 12.
## Prerequisites
1. Three Proxmox hosts (lab02, lab03, lab04)
2. Proxmox API access with root credentials
3. Network configured with:
- Gateway: 10.0.19.254
- Available IPs: 10.0.19.200-202
4. Ansible collections installed:
```bash
ansible-galaxy collection install -r requirements.yml
```
## Architecture
- **k8s-master** (10.0.19.200): Control plane node
- **k8s-worker1** (10.0.19.201): Worker node
- **k8s-worker2** (10.0.19.202): Worker node
All nodes run Debian 12 with:
- 2 CPU cores
- 4GB RAM
- 40GB disk
- Containerd runtime
- Kubernetes 1.33.3
## Deployment Methods
### Method 1: Full Automated Deployment
Use this to create VMs and install Kubernetes:
```bash
export PROXMOX_PASSWORD='your-proxmox-password'
export K8S_VM_PASSWORD='secure-password' # Optional, defaults to 'changeme'
./deploy-k8s-cluster.sh
```
### Method 2: Step-by-Step Deployment
1. **Create VM Template**:
```bash
ansible-playbook create-vm-template.yml -e proxmox_password='your-password'
```
2. **Deploy Cluster**:
```bash
ansible-playbook -i inventories/k8s-cluster/hosts k8s-cluster.yml \
-e proxmox_password='your-password' \
-e vm_password='secure-password'
```
### Method 3: Install on Existing VMs
If you already have Debian 12 VMs:
```bash
ansible-playbook -i inventories/k8s-cluster/hosts k8s-install-only.yml
```
## Post-Deployment
### Access the Cluster
1. **SSH to master**:
```bash
ssh debian@10.0.19.200
kubectl get nodes
```
2. **Copy kubeconfig locally**:
```bash
scp debian@10.0.19.200:~/.kube/config ~/.kube/config-k8s
export KUBECONFIG=~/.kube/config-k8s
kubectl get nodes
```
### Verify Installation
```bash
# Check nodes
kubectl get nodes
# Check system pods
kubectl get pods -n kube-system
# Check cluster info
kubectl cluster-info
```
### Access Dashboard
The Kubernetes dashboard is installed by default:
```bash
# Create admin user
kubectl create serviceaccount dashboard-admin -n kubernetes-dashboard
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-admin
# Get token
kubectl -n kubernetes-dashboard create token dashboard-admin
# Start proxy
kubectl proxy
# Access at: http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
```
## Customization
### Change Network Settings
Edit `k8s-cluster.yml`:
```yaml
vars:
k8s_nodes:
- name: k8s-master
ip: 10.0.19.200/24 # Change IP here
gateway: 10.0.19.254 # Change gateway
nameserver: 10.0.19.254 # Change DNS
```
### Change VM Resources
Edit `k8s-cluster.yml`:
```yaml
k8s_nodes:
- name: k8s-master
cores: 4 # Increase CPU
memory: 8192 # Increase RAM
disk: 100 # Increase disk
```
### Use Different CNI
Replace Flannel with Calico:
```yaml
- name: Install Calico CNI
command: kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
```
## Troubleshooting
### VMs Won't Start
- Check Proxmox storage space
- Verify template was created successfully
- Check VM IDs aren't already in use
### Nodes Not Joining
- Verify network connectivity between nodes
- Check firewall rules
- Ensure time is synchronized
### Pods Stuck in Pending
- Check node resources: `kubectl describe nodes`
- Verify CNI is working: `kubectl get pods -n kube-system`
## K3s Deployment (Recommended for Lightweight Clusters)
K3s is a certified Kubernetes distribution designed for production workloads in resource-constrained environments.
### K3s Architecture
- **k3s-server** (10.0.19.210): Control plane + etcd
- **k3s-agent1** (10.0.19.211): Worker node
- **k3s-agent2** (10.0.19.212): Worker node
All nodes run Debian 12 with:
- 2 CPU cores
- 2GB RAM (vs 4GB for full K8s)
- 20GB disk (vs 40GB for full K8s)
- Single binary (~100MB)
- Built-in SQLite (no external etcd)
### Deploy K3s Cluster
```bash
export PROXMOX_PASSWORD='your-proxmox-password'
export K3S_VM_PASSWORD='secure-password' # Optional, defaults to 'changeme'
./deploy-k3s-cluster.sh
```
Or manually:
```bash
ansible-playbook -i inventories/k3s/hosts k3s-cluster.yml \
-e proxmox_password='your-password' \
-e vm_password='secure-password'
```
### K3s Features
Out of the box, K3s includes:
- Flannel CNI for pod networking
- CoreDNS for cluster DNS
- Traefik ingress controller (disabled in our setup)
- Local-path storage provisioner
- Metrics server
- Helm controller
### Access K3s Cluster
```bash
# SSH to server
ssh debian@10.0.19.210
kubectl get nodes
# Copy kubeconfig locally
scp debian@10.0.19.210:~/.kube/config ~/.kube/config-k3s
export KUBECONFIG=~/.kube/config-k3s
```
## Cleanup
To remove the cluster:
### K8s Cluster
```bash
# Delete K8s VMs from Proxmox
for vmid in 200 201 202; do
ssh root@lab02.w5isp.com "qm stop $vmid && qm destroy $vmid"
done
```
### K3s Cluster
```bash
# Delete K3s VMs from Proxmox
for vmid in 300 301 302; do
ssh root@lab02.w5isp.com "qm stop $vmid && qm destroy $vmid"
done
```

75
ansible/deploy-k3s-cluster.sh
View file

@ -1,75 +0,0 @@
#!/bin/bash
# Deploy a 3-node K3s cluster on Proxmox using Debian 12
set -e
echo "=== K3s Cluster Deployment Script ==="
echo "This will create a lightweight 3-node K3s cluster on your Proxmox servers"
echo
echo "VMs to be created:"
echo " - k3s-server (10.0.19.210) - 2 CPU, 2GB RAM, 20GB disk"
echo " - k3s-agent1 (10.0.19.211) - 2 CPU, 2GB RAM, 20GB disk"
echo " - k3s-agent2 (10.0.19.212) - 2 CPU, 2GB RAM, 20GB disk"
echo
# Check if proxmox password is set
if [ -z "$PROXMOX_PASSWORD" ]; then
echo "Please set PROXMOX_PASSWORD environment variable"
echo "export PROXMOX_PASSWORD='your-proxmox-password'"
exit 1
fi
# Install required collections if needed
if ! ansible-galaxy collection list | grep -q community.general; then
echo "Installing required Ansible collections..."
ansible-galaxy collection install community.general
fi
# Check if template exists
echo "Checking if Debian 12 template exists..."
TEMPLATE_EXISTS=$(ansible -i inventories/k3s/hosts proxmox -m shell -a "qm list | grep -q debian12-cloud-init && echo yes || echo no" -e proxmox_password="$PROXMOX_PASSWORD" | grep -o "yes\|no" | head -1)
if [ "$TEMPLATE_EXISTS" != "yes" ]; then
echo "Debian 12 template not found. Please run create-vm-template.yml first."
read -p "Create template now? (y/n) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
ansible-playbook create-vm-template.yml -e proxmox_password="$PROXMOX_PASSWORD"
else
exit 1
fi
fi
# Deploy K3s cluster
echo
echo "Ready to deploy K3s cluster"
read -p "Continue? (y/n) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
echo "Creating VMs and installing K3s..."
ansible-playbook -i inventories/k3s/hosts k3s-cluster.yml \
-e proxmox_password="$PROXMOX_PASSWORD" \
-e vm_password="${K3S_VM_PASSWORD:-changeme}"
fi
echo
echo "=== Deployment Complete ==="
echo "To access your K3s cluster:"
echo " ssh debian@10.0.19.210"
echo " kubectl get nodes"
echo
echo "To copy kubeconfig to your local machine:"
echo " mkdir -p ~/.kube"
echo " scp debian@10.0.19.210:~/.kube/config ~/.kube/config-k3s"
echo " export KUBECONFIG=~/.kube/config-k3s"
echo
echo "K3s includes:"
echo " - Embedded SQLite (no external datastore needed)"
echo " - Flannel CNI for networking"
echo " - CoreDNS for cluster DNS"
echo " - Local-path provisioner for storage"
echo " - Metrics server"
echo
echo "To install additional components:"
echo " - Kubernetes Dashboard: kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml"
echo " - Cert-Manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.3/cert-manager.yaml"

56
ansible/general.yml.merged
View file

@ -1,56 +0,0 @@
---
- name: Base setup
hosts: all
become: yes
become_method: sudo
roles:
- general
tags:
- base
- name: SNMP clients
hosts: snmp_clients
become: yes
become_method: sudo
roles:
- snmp_clients
tags:
- snmp
- name: k3s
hosts: k3s_servers
become: yes
become_method: sudo
tags: k3s
roles:
- k3s_servers
- name: Docker servers
hosts: docker
become: yes
become_method: sudo
tags: docker
roles:
- docker
vars:
gpg_fingerprint: "9DC858229FC7DD38854AE2D88D81803C0EBFCD88"
- name: Netboot servers
hosts: netboot
become: yes
become_method: sudo
- vars/netboot.yml
- vars/network.yml
vars:
netboot_arch_dist: "{{ netboot.dist | product(netboot.arch) }}"
tags:
- netboot
roles:
- netboot_servers
- hosts: syslog
become: yes
become_method: sudo
tags: syslog
roles:
- syslog_servers

0
ansible/group_vars/all.yml → ansible/group_vars/all/all.yml
View file

12
ansible/group_vars/all/vault.yml Normal file
View file

@ -0,0 +1,12 @@
$ANSIBLE_VAULT;1.1;AES256
31383762643636383533356463306362613965366237613862373939663362366463303363643363
6234356433653762663966616562656137313661653165320a666439323636376134306661396332
30333732336338306134313862646262323864343034633466646530393036333539666232393166
3466346531623636340a656235343462383235613637663964653664333361326632363663353636
65646639613435663934343038626264393636383961646463623664336330363137653030353533
35353934323462626562393865653366643665346263653864626236363961303030643339346265
37356335323861393434366632333366313238666462643435323634376637613966393732316233
31643266653964303137633339373439373836626366633732323731666436343462326338386134
38323334653661636236653830346531643539393839333330633939356362663065613266626536
62363934396261313330616432363561313666393637653834353132613838666636353532363965
346363633336613836626435643363396663

13
ansible/group_vars/all/vault.yml.example Normal file
View file

@ -0,0 +1,13 @@
# Example vault file for NetBox
# Copy this to vault.yml and encrypt with: ansible-vault encrypt vault.yml
#
# To edit: ansible-vault edit group_vars/all/vault.yml
# To run playbooks: ansible-playbook -i hosts --ask-vault-pass netbox.yml
# NetBox secrets
vault_netbox_db_password: "change-me-to-secure-password"
vault_netbox_secret_key: "change-me-to-very-long-random-string-at-least-50-chars"
vault_netbox_admin_password: "change-me-to-secure-admin-password"
vault_netbox_redis_password: "optional-redis-password"
vault_netbox_redis_cache_password: "optional-redis-cache-password"
vault_netbox_superuser_api_token: "optional-api-token-for-automation"

28
ansible/host_vars/netbox.vntx.net.yml Normal file
View file

@ -0,0 +1,28 @@
---
# Host-specific configuration for netbox.vntx.net
# Python interpreter
ansible_python_interpreter: /usr/bin/python3
# Override defaults for NetBox installation
netbox_port: 8001
netbox_secret_key: "{{ vault_netbox_secret_key }}"
netbox_db_password: "{{ vault_netbox_db_password }}"
netbox_admin_password: "{{ vault_netbox_admin_password }}"
netbox_admin_email: "admin@vntx.net"
# Set allowed hosts for this installation (space-separated string)
netbox_allowed_hosts: "netbox.vntx.net localhost 127.0.0.1"
# Email configuration for VNTX environment
netbox_email_from: "netbox@vntx.net"
# Time zone
netbox_timezone: "America/Chicago"
# Enable metrics for monitoring
netbox_metrics_enabled: true
# Custom branding if desired
# netbox_branding_title: "VNTX NetBox"
# netbox_branding_url: "https://vntx.net"

15
ansible/hosts
View file

@ -11,13 +11,11 @@ ns2.vntx.net
#ntp.vntx.net #ntp.vntx.net
[app_servers] [app_servers]
dallas.aprs2.net
apps.w5isp.com apps.w5isp.com
[home_servers] [home_servers]
skippy.w5isp.com skippy.w5isp.com
sync.w5isp.com ansible_host=204.110.191.216 sync.w5isp.com ansible_host=204.110.191.216
w5isp.w5isp.com ansible_host=204.110.191.200
[proxmox] [proxmox]
lab02.w5isp.com lab02.w5isp.com
@ -25,28 +23,15 @@ lab03.w5isp.com
lab04.w5isp.com lab04.w5isp.com
[tailscale_home] [tailscale_home]
lab02.w5isp.com
lab03.w5isp.com
lab04.w5isp.com
g.w5isp.com
sync.w5isp.com sync.w5isp.com
[caddy_servers] [caddy_servers]
caddy.w5isp.com caddy.w5isp.com
[k3s_servers]
home-k3s-control-1 ansible_host=10.0.19.100
home-k3s-control-2 ansible_host=10.0.19.101
home-k3s-control-3 ansible_host=10.0.19.102
home-k3s-worker-1 ansible_host=10.0.19.110
home-k3s-worker-2 ansible_host=10.0.19.111
home-k3s-worker-3 ansible_host=10.0.19.112
[syncthing_servers] [syncthing_servers]
sync.w5isp.com sync.w5isp.com
[postgresql_servers] [postgresql_servers]
#postgres ansible_host=10.0.19.224
[mailcow_servers] [mailcow_servers]
sync.w5isp.com sync.w5isp.com

23
ansible/inventories/k3s/hosts
View file

@ -1,23 +0,0 @@
# K3s cluster inventory
[proxmox]
lab02.w5isp.com
lab03.w5isp.com
lab04.w5isp.com
[proxmox:vars]
ansible_python_interpreter=/usr/bin/python3
[k3s_server]
k3s-server ansible_host=10.0.19.210 ansible_user=ansible
[k3s_agents]
k3s-agent1 ansible_host=10.0.19.211 ansible_user=ansible
k3s-agent2 ansible_host=10.0.19.212 ansible_user=ansible
[k3s_cluster:children]
k3s_server
k3s_agents
[k3s_cluster:vars]
ansible_python_interpreter=/usr/bin/python3

11
ansible/inventory.yml
View file

@ -1,11 +0,0 @@
all:
hosts:
postgres:
ansible_host: 10.0.19.224
ansible_user: graham
ansible_become: yes
ansible_python_interpreter: /usr/bin/python3
children:
postgres_servers:
hosts:
postgres:

351
ansible/k3s-cluster.yml
View file

@ -1,351 +0,0 @@
---
# Deploy K3s cluster on Proxmox
# This playbook creates VMs and installs K3s
- name: Create K3s VMs on Proxmox
hosts: proxmox
vars:
k3s_vms:
- name: k3s-server
vmid: 300
node: lab02
cores: 4
memory: 8192
disk: 64
ip: 10.0.19.210/22
gateway: 10.0.19.254
- name: k3s-agent1
vmid: 301
node: lab03
cores: 4
memory: 8192
disk: 64
ip: 10.0.19.211/22
gateway: 10.0.19.254
- name: k3s-agent2
vmid: 302
node: lab04
cores: 4
memory: 8192
disk: 64
ip: 10.0.19.212/22
gateway: 10.0.19.254
template_vmid: 9000
proxmox_api_host: "{{ ansible_host | default(inventory_hostname) }}"
proxmox_api_user: root@pam
proxmox_api_password: "{{ proxmox_password }}"
vm_user: ansible
vm_password: "{{ lookup('password', '/dev/null length=16 chars=ascii_letters,digits') }}"
ssh_key_path: "~/.ssh/id_ed25519.pub"
tasks:
- name: Clone VMs from template on lab02
community.general.proxmox_kvm:
api_host: "{{ proxmox_api_host }}"
api_user: "{{ proxmox_api_user }}"
api_password: "{{ proxmox_api_password }}"
validate_certs: false
clone: "debian12-cloud-init"
newid: "{{ item.vmid }}"
name: "{{ item.name }}"
node: "lab02"
storage: "ceph"
full: true
timeout: 600
loop: "{{ k3s_vms }}"
when: inventory_hostname == 'lab02.w5isp.com'
delegate_to: localhost
- name: Wait for clones to settle
ansible.builtin.pause:
seconds: 10
when: inventory_hostname == 'lab02.w5isp.com'
- name: Check current VM location
shell: |
qm config {{ item.vmid }} >/dev/null 2>&1 && echo "local" || echo "remote"
register: vm_location
loop: "{{ k3s_vms }}"
when: inventory_hostname == 'lab02.w5isp.com'
changed_when: false
- name: Migrate VMs to target nodes
shell: |
pvesh create /nodes/lab02/qemu/{{ item.0.vmid }}/migrate \
-target {{ item.0.node }} \
-online 0
loop: "{{ k3s_vms | zip(vm_location.results) | list }}"
when:
- inventory_hostname == 'lab02.w5isp.com'
- item.0.node != 'lab02'
- item.1.stdout == 'local'
- name: Wait for migrations to complete
ansible.builtin.pause:
seconds: 30
when: inventory_hostname == 'lab02.w5isp.com'
- name: Resize VM disk
community.general.proxmox_disk:
api_host: "{{ proxmox_api_host }}"
api_user: "{{ proxmox_api_user }}"
api_password: "{{ proxmox_api_password }}"
validate_certs: false
vmid: "{{ item.vmid }}"
disk: scsi0
size: "{{ item.disk }}G"
state: resized
loop: "{{ k3s_vms }}"
when: inventory_hostname == 'lab02.w5isp.com'
delegate_to: localhost
- name: Configure VM resources
community.general.proxmox_kvm:
api_host: "{{ proxmox_api_host }}"
api_user: "{{ proxmox_api_user }}"
api_password: "{{ proxmox_api_password }}"
validate_certs: false
vmid: "{{ item.vmid }}"
node: "{{ item.node }}"
cores: "{{ item.cores }}"
memory: "{{ item.memory }}"
onboot: true
update: true
loop: "{{ k3s_vms }}"
when: inventory_hostname == 'lab02.w5isp.com'
delegate_to: localhost
- name: Configure cloud-init
community.general.proxmox_kvm:
api_host: "{{ proxmox_api_host }}"
api_user: "{{ proxmox_api_user }}"
api_password: "{{ proxmox_api_password }}"
validate_certs: false
vmid: "{{ item.vmid }}"
node: "{{ item.node }}"
ipconfig:
ipconfig0: "ip={{ item.ip }},gw={{ item.gateway }}"
nameservers: "9.9.9.9"
ciuser: "{{ vm_user }}"
cipassword: "{{ vm_password }}"
sshkeys: "{{ lookup('file', ssh_key_path) }}"
update: true
loop: "{{ k3s_vms }}"
when: inventory_hostname == 'lab02.w5isp.com'
delegate_to: localhost
- name: Start VMs
community.general.proxmox_kvm:
api_host: "{{ proxmox_api_host }}"
api_user: "{{ proxmox_api_user }}"
api_password: "{{ proxmox_api_password }}"
validate_certs: false
vmid: "{{ item.vmid }}"
node: "{{ item.node }}"
state: started
loop: "{{ k3s_vms }}"
when: inventory_hostname == 'lab02.w5isp.com'
delegate_to: localhost
- name: Wait for VMs to be reachable
ansible.builtin.wait_for:
host: "{{ item.ip.split('/')[0] }}"
port: 22
delay: 30
timeout: 300
loop: "{{ k3s_vms }}"
when: inventory_hostname == 'lab02.w5isp.com'
delegate_to: localhost
- name: Bootstrap K3s nodes
hosts: k3s_cluster
become: true
gather_facts: false
tasks:
- name: Wait for system to be ready
ansible.builtin.wait_for_connection:
delay: 10
timeout: 300
- name: Gather facts
ansible.builtin.setup:
- name: Update apt cache
ansible.builtin.apt:
update_cache: true
cache_valid_time: 3600
- name: Install required packages
ansible.builtin.apt:
name:
- curl
- ca-certificates
- apt-transport-https
- sudo
state: present
- name: Create graham user
ansible.builtin.user:
name: graham
groups: sudo
shell: /bin/bash
create_home: true
state: present
- name: Set up SSH key for graham user
ansible.posix.authorized_key:
user: graham
key: "{{ lookup('file', '~/.ssh/id_ed25519.pub') }}"
state: present
- name: Configure sudo for graham user
ansible.builtin.lineinfile:
path: /etc/sudoers.d/graham
line: 'graham ALL=(ALL) NOPASSWD:ALL'
create: true
validate: 'visudo -cf %s'
- name: Configure DNS to use Quad9
ansible.builtin.copy:
content: |
nameserver 9.9.9.9
nameserver 149.112.112.112
dest: /etc/resolv.conf
owner: root
group: root
mode: '0644'
- name: Prevent resolv.conf from being overwritten
ansible.builtin.file:
path: /etc/resolv.conf
attr: +i
- name: Check if Tailscale service exists
ansible.builtin.stat:
path: /etc/systemd/system/tailscaled.service
register: tailscale_service
- name: Stop and disable Tailscale if present
ansible.builtin.systemd:
name: tailscaled
state: stopped
enabled: false
when: tailscale_service.stat.exists
- name: Install K3s on server
hosts: k3s_server
become: true
vars:
k3s_version: v1.28.5+k3s1
k3s_server_args: "--disable traefik --write-kubeconfig-mode 644"
tasks:
- name: Download K3s installer
ansible.builtin.get_url:
url: https://get.k3s.io
dest: /tmp/k3s-install.sh
mode: '0755'
- name: Install K3s server
ansible.builtin.shell: |
INSTALL_K3S_VERSION="{{ k3s_version }}" \
INSTALL_K3S_EXEC="{{ k3s_server_args }}" \
sh /tmp/k3s-install.sh
args:
creates: /usr/local/bin/k3s
- name: Wait for K3s to be ready
ansible.builtin.wait_for:
path: /var/lib/rancher/k3s/server/node-token
state: present
timeout: 60
- name: Get node token
ansible.builtin.slurp:
src: /var/lib/rancher/k3s/server/node-token
register: node_token
- name: Store token for agents
ansible.builtin.set_fact:
k3s_token: "{{ node_token.content | b64decode | trim }}"
- name: Create .kube directory
ansible.builtin.file:
path: /home/{{ ansible_user }}/.kube
state: directory
owner: "{{ ansible_user }}"
group: "{{ ansible_user }}"
mode: '0755'
- name: Copy kubeconfig
ansible.builtin.copy:
src: /etc/rancher/k3s/k3s.yaml
dest: /home/{{ ansible_user }}/.kube/config
remote_src: true
owner: "{{ ansible_user }}"
group: "{{ ansible_user }}"
mode: '0600'
- name: Update kubeconfig server address
ansible.builtin.replace:
path: /home/{{ ansible_user }}/.kube/config
regexp: 'https://127.0.0.1:6443'
replace: 'https://{{ ansible_host }}:6443'
- name: Install K3s on agents
hosts: k3s_agents
become: true
vars:
k3s_version: v1.28.5+k3s1
k3s_server_host: "{{ hostvars[groups['k3s_server'][0]]['ansible_host'] }}"
k3s_token: "{{ hostvars[groups['k3s_server'][0]]['k3s_token'] }}"
tasks:
- name: Download K3s installer
ansible.builtin.get_url:
url: https://get.k3s.io
dest: /tmp/k3s-install.sh
mode: '0755'
- name: Install K3s agent
ansible.builtin.shell: |
INSTALL_K3S_VERSION="{{ k3s_version }}" \
K3S_URL="https://{{ k3s_server_host }}:6443" \
K3S_TOKEN="{{ k3s_token }}" \
sh /tmp/k3s-install.sh
args:
creates: /usr/local/bin/k3s
- name: Configure K3s cluster
hosts: k3s_server
become: true
tasks:
- name: Install Helm
ansible.builtin.shell: |
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
args:
creates: /usr/local/bin/helm
- name: Wait for all nodes to be ready
ansible.builtin.shell: |
kubectl get nodes --no-headers | grep -v Ready || true
register: not_ready_nodes
until: not_ready_nodes.stdout == ""
retries: 30
delay: 10
become_user: "{{ ansible_user }}"
- name: Label worker nodes
ansible.builtin.shell: |
kubectl label node {{ item }} node-role.kubernetes.io/worker=worker --overwrite
loop:
- k3s-agent1
- k3s-agent2
become_user: "{{ ansible_user }}"
- name: Show cluster status
ansible.builtin.shell: kubectl get nodes
register: nodes_status
become_user: "{{ ansible_user }}"
- name: Display cluster status
ansible.builtin.debug:
msg: "{{ nodes_status.stdout_lines }}"

48
ansible/k3s-tailscale.yml
View file

@ -1,48 +0,0 @@
---
- name: Install Tailscale on K3s control plane
hosts: k3s_server
become: true
roles:
- role: artis3n.tailscale
vars:
# Pull Tailscale auth key from environment variable
tailscale_authkey: "{{ lookup('ansible.builtin.env', 'TAILSCALE_KEY') }}"
tailscale_args: "--ssh"
pre_tasks:
- name: Debug - Check Tailscale key (will be partially redacted)
debug:
msg:
- "Key exists: {{ 'TAILSCALE_KEY' in ansible_env }}"
- "Key length: {{ lookup('ansible.builtin.env', 'TAILSCALE_KEY') | length }}"
- "Key prefix: {{ lookup('ansible.builtin.env', 'TAILSCALE_KEY')[0:10] }}..."
when: lookup('ansible.builtin.env', 'TAILSCALE_KEY') | length > 0
- name: Fail if no Tailscale key
fail:
msg: "TAILSCALE_KEY environment variable is not set!"
when: lookup('ansible.builtin.env', 'TAILSCALE_KEY') | length == 0
- name: Ensure artis3n.tailscale role is installed
delegate_to: localhost
become: false
ansible.builtin.command:
cmd: ansible-galaxy install artis3n.tailscale
run_once: true
post_tasks:
- name: Wait for Tailscale to be connected
ansible.builtin.command:
cmd: tailscale status
register: tailscale_status
until: '"100.64" in tailscale_status.stdout'
retries: 10
delay: 5
- name: Get Tailscale IP
ansible.builtin.command:
cmd: tailscale ip -4
register: tailscale_ip
- name: Display Tailscale IP
debug:
msg: "{{ inventory_hostname }} Tailscale IP: {{ tailscale_ip.stdout }}"

8
ansible/netbox.yml Normal file
View file

@ -0,0 +1,8 @@
---
# Playbook to deploy NetBox
- name: Deploy NetBox
hosts: netbox.vntx.net
become: yes
roles:
- base
- netbox

16
ansible/playbook.yml
View file

@ -158,14 +158,6 @@
tags: tags:
- snmp - snmp
- name: k3s
hosts: k3s_servers
become: yes
become_method: sudo
tags: k3s
roles:
- k3s_servers
- name: Docker servers - name: Docker servers
hosts: docker hosts: docker
become: yes become: yes
@ -176,14 +168,6 @@
vars: vars:
gpg_fingerprint: "9DC858229FC7DD38854AE2D88D81803C0EBFCD88" gpg_fingerprint: "9DC858229FC7DD38854AE2D88D81803C0EBFCD88"
- name: Syslog servers
hosts: syslog
become: yes
become_method: sudo
tags: syslog
roles:
- syslog_servers
- name: Syncthing servers - name: Syncthing servers
hosts: syncthing_servers hosts: syncthing_servers
become: yes become: yes

2
ansible/roles/caddy/templates/Caddyfile.j2
View file

@ -21,7 +21,7 @@ truck.w5isp.com {
} }
photos.w5isp.com { photos.w5isp.com {
reverse_proxy http://10.0.16.1:2283 reverse_proxy http://100.107.11.77:2283
} }
# Mailcow configuration # Mailcow configuration

33
ansible/roles/general/tasks/debian/k3s.yml
View file

@ -1,33 +0,0 @@
- name: Install k3s.sh setup script
copy:
src: roles/k3s_servers/files/k3s.sh
dest: /usr/local/bin/k3s.sh
owner: root
group: root
mode: '00755'
tags:
- k3s_setup
- name: Install required packages
apt:
pkg:
- netscript-ipfilter
tags:
- k3s_packages
- name: Check for node token
file:
path: /var/lib/rancher/k3s/server/node-token
register: token
ignore_errors: true
tags:
- k3s_token
- name: Run k3s installer
shell:
cmd: /usr/local/bin/k3s.sh
environment:
INSTALL_K3S_EXEC: "server --flannel-backend={{ k3s.flannel_backend }} --cluster-cidr={{ k3s.cluster_cidr }} --service-cidr={{ k3s.service_cidr }}"
when: token.state == "absent"
tags:
- k3s_install

8
ansible/roles/general/tasks/main.yml
View file

@ -32,10 +32,4 @@
when: ansible_os_family == "Debian" when: ansible_os_family == "Debian"
tags: tags:
- icinga2 - icinga2
- monitoring - monitoring
- import_tasks: debian/k3s.yml
when: ansible_os_family == "Debian"
tags:
- k3s
- kubernetes

3
ansible/roles/k3s_servers/tasks/main.yml
View file

@ -1,3 +0,0 @@
---
# K3s server configuration tasks
# TODO: Implement K3s server installation and configuration

106
ansible/roles/netbox/defaults/main.yml Normal file
View file

@ -0,0 +1,106 @@
---
# Directory configuration
netbox_home: /opt/netbox
# Version configuration
netbox_version: release
netbox_docker_tag: latest
netbox_postgres_version: 15-alpine
netbox_redis_version: 7-alpine
# Network configuration
netbox_port: 8080
netbox_listen_address: 0.0.0.0
netbox_domain: netbox.vntx.net
# Database configuration
netbox_db_name: netbox
netbox_db_user: netbox
netbox_db_password: "{{ vault_netbox_db_password | default('changeme') }}"
netbox_db_host: postgres
netbox_db_port: 5432
# Redis configuration
netbox_redis_host: redis
netbox_redis_port: 6379
netbox_redis_password: "{{ vault_netbox_redis_password | default('') }}"
netbox_redis_cache_host: redis-cache
netbox_redis_cache_port: 6379
netbox_redis_cache_password: "{{ vault_netbox_redis_cache_password | default('') }}"
# NetBox configuration
netbox_secret_key: "{{ vault_netbox_secret_key | default('changeme_to_a_very_long_random_string') }}"
netbox_superuser_api_token: "{{ vault_netbox_superuser_api_token | default('') }}"
netbox_allowed_hosts: "*"
netbox_cors_origin_allow_all: true
netbox_debug: false
netbox_email_server: localhost
netbox_email_port: 25
netbox_email_username: ""
netbox_email_password: ""
netbox_email_use_ssl: false
netbox_email_use_tls: false
netbox_email_from: "netbox@{{ ansible_fqdn }}"
netbox_email_timeout: 10
# Login configuration
netbox_login_required: false
netbox_login_timeout: 1209600 # 14 days in seconds
# Media and upload configuration
netbox_media_root: /opt/netbox/netbox/media
netbox_reports_root: /opt/netbox/netbox/reports
netbox_scripts_root: /opt/netbox/netbox/scripts
netbox_max_upload_size: 26214400 # 25MB
# Housekeeping
netbox_housekeeping_interval: 86400 # 24 hours
# Metrics
netbox_metrics_enabled: false
# Plugins
netbox_plugins: []
netbox_plugins_config: {}
# Admin user configuration
netbox_create_superuser: true
netbox_admin_user: admin
netbox_admin_email: "admin@{{ ansible_fqdn }}"
netbox_admin_password: "{{ vault_netbox_admin_password | default('changeme') }}"
# NAPALM configuration (for network device integration)
netbox_napalm_username: ""
netbox_napalm_password: ""
netbox_napalm_timeout: 30
# Time zone
netbox_timezone: UTC
# Date and time formatting
netbox_date_format: "N j, Y"
netbox_short_date_format: "Y-m-d"
netbox_time_format: "g:i a"
netbox_short_time_format: "H:i:s"
netbox_datetime_format: "N j, Y g:i a"
netbox_short_datetime_format: "Y-m-d H:i"
# Pagination
netbox_paginate_count: 50
netbox_max_page_size: 1000
# GraphQL
netbox_graphql_enabled: true
# Webhook configuration
netbox_webhooks_enabled: true
netbox_webhook_redis_queue: default
# Cache timeout
netbox_cache_timeout: 900 # 15 minutes
# Changelog retention
netbox_changelog_retention: 90 # days
# Job retention
netbox_jobresult_retention: 90 # days

20
ansible/roles/netbox/handlers/main.yml Normal file
View file

@ -0,0 +1,20 @@
---
- name: restart netbox
systemd:
name: netbox-docker
state: restarted
- name: stop netbox
systemd:
name: netbox-docker
state: stopped
- name: start netbox
systemd:
name: netbox-docker
state: started
- name: restart caddy
systemd:
name: caddy
state: restarted

253
ansible/roles/netbox/tasks/main.yml Normal file
View file

@ -0,0 +1,253 @@
---
- name: Install prerequisites for Docker
apt:
name:
- ca-certificates
- curl
- gnupg
- lsb-release
- git
- python3-full
state: present
update_cache: yes
when: ansible_os_family == "Debian"
- name: Add Docker's official GPG key
shell: |
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
args:
creates: /usr/share/keyrings/docker-archive-keyring.gpg
- name: Add Docker repository
apt_repository:
repo: "deb [arch={{ ansible_architecture | replace('x86_64', 'amd64') }} signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable"
state: present
filename: docker
- name: Install Docker packages
apt:
name:
- docker-ce
- docker-ce-cli
- containerd.io
- docker-compose-plugin
state: present
update_cache: yes
- name: Ensure Docker service is started and enabled
systemd:
name: docker
state: started
enabled: yes
- name: Create netbox directories
file:
path: "{{ item }}"
state: directory
mode: '0755'
loop:
- "{{ netbox_home }}/netbox-docker"
- "{{ netbox_home }}/netbox-docker/env"
- "{{ netbox_home }}/data"
- "{{ netbox_home }}/data/redis"
- "{{ netbox_home }}/data/redis-cache"
- "{{ netbox_home }}/data/media"
- "{{ netbox_home }}/data/reports"
- "{{ netbox_home }}/data/scripts"
- name: Create postgres data directory with specific permissions
file:
path: "{{ netbox_home }}/data/postgres"
state: directory
mode: '0700'
- name: Clone netbox-docker repository
git:
repo: https://github.com/netbox-community/netbox-docker.git
dest: "{{ netbox_home }}/netbox-docker"
version: "{{ netbox_version }}"
force: yes
notify: restart netbox
# Configuration directory creation removed - using default configuration with env vars
- name: Deploy docker-compose override file
template:
src: docker-compose.override.yml.j2
dest: "{{ netbox_home }}/netbox-docker/docker-compose.override.yml"
mode: '0644'
notify: restart netbox
- name: Deploy netbox.env file
template:
src: env.j2
dest: "{{ netbox_home }}/netbox-docker/env/netbox.env"
mode: '0600'
notify: restart netbox
- name: Deploy postgres.env file
template:
src: postgres.env.j2
dest: "{{ netbox_home }}/netbox-docker/env/postgres.env"
mode: '0600'
notify: restart netbox
- name: Deploy redis.env file
template:
src: redis.env.j2
dest: "{{ netbox_home }}/netbox-docker/env/redis.env"
mode: '0600'
notify: restart netbox
- name: Deploy redis-cache.env file
template:
src: redis-cache.env.j2
dest: "{{ netbox_home }}/netbox-docker/env/redis-cache.env"
mode: '0600'
notify: restart netbox
- name: Remove custom configuration.py if exists
file:
path: "{{ netbox_home }}/netbox-docker/configuration/configuration.py"
state: absent
- name: Pull container images
command: docker compose -f docker-compose.yml -f docker-compose.override.yml pull
args:
chdir: "{{ netbox_home }}/netbox-docker"
register: pull_result
changed_when: "'Pulling' in pull_result.stdout"
- name: Deploy systemd service for NetBox
template:
src: netbox-docker.service.j2
dest: /etc/systemd/system/netbox-docker.service
owner: root
group: root
mode: '0644'
notify: restart netbox
- name: Reload systemd daemon
systemd:
daemon_reload: yes
- name: Start and enable NetBox service
systemd:
name: netbox-docker
state: started
enabled: yes
- name: Wait for initial container startup
pause:
seconds: 15
- name: Check container status
command: docker ps -a
register: container_status
changed_when: false
- name: Debug container status
debug:
var: container_status.stdout_lines
when: "'ANSIBLE_DEBUG' in ansible_env or ansible_verbosity > 0"
- name: Wait for NetBox to be ready
uri:
url: "http://localhost:{{ netbox_port }}/api/"
method: GET
status_code:
- 200
- 403 # API might return 403 if auth is required
validate_certs: no
timeout: 30
register: result
until: result.status in [200, 403]
retries: 30
delay: 10
- name: Check if superuser exists
command: >
docker exec netbox-docker-netbox-1
python /opt/netbox/netbox/manage.py shell -c
"from django.contrib.auth import get_user_model; User = get_user_model(); exit(0 if User.objects.filter(username='{{ netbox_admin_user }}').exists() else 1)"
register: superuser_check
failed_when: false
changed_when: false
- name: Create superuser
command: >
docker exec netbox-docker-netbox-1
python /opt/netbox/netbox/manage.py createsuperuser
--username {{ netbox_admin_user }}
--email {{ netbox_admin_email }}
--noinput
environment:
DJANGO_SUPERUSER_PASSWORD: "{{ netbox_admin_password }}"
when:
- netbox_create_superuser | bool
- superuser_check.rc != 0
# Caddy reverse proxy setup
- name: Install Caddy prerequisites
apt:
name:
- debian-keyring
- debian-archive-keyring
- apt-transport-https
- curl
state: present
update_cache: yes
- name: Add Caddy GPG key
shell: |
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
args:
creates: /usr/share/keyrings/caddy-stable-archive-keyring.gpg
- name: Add Caddy repository
shell: |
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | tee /etc/apt/sources.list.d/caddy-stable.list
args:
creates: /etc/apt/sources.list.d/caddy-stable.list
- name: Fix permissions on Caddy repository files
file:
path: "{{ item }}"
mode: '0644'
loop:
- /usr/share/keyrings/caddy-stable-archive-keyring.gpg
- /etc/apt/sources.list.d/caddy-stable.list
- name: Update apt cache after adding Caddy repo
apt:
update_cache: yes
- name: Install Caddy
apt:
name: caddy
state: present
- name: Create Caddy log directory
file:
path: /var/log/caddy
state: directory
owner: caddy
group: caddy
mode: '0755'
- name: Configure Caddy for NetBox
template:
src: Caddyfile.j2
dest: /etc/caddy/Caddyfile
owner: root
group: root
mode: '0644'
notify: restart caddy
- name: Ensure Caddy is started and enabled
systemd:
name: caddy
state: started
enabled: yes

54
ansible/roles/netbox/templates/Caddyfile.j2 Normal file
View file

@ -0,0 +1,54 @@
# {{ ansible_managed }}
# Caddy configuration for NetBox reverse proxy
{{ netbox_domain | default('netbox.vntx.net') }} {
# Enable compression
encode gzip
# Reverse proxy to NetBox
reverse_proxy localhost:{{ netbox_port }} {
# Set headers for proper operation
header_up Host {host}
header_up X-Real-IP {remote}
header_up X-Forwarded-For {remote}
header_up X-Forwarded-Proto {scheme}
# WebSocket support
header_up Connection {>Connection}
header_up Upgrade {>Upgrade}
}
# Logging
log {
output file /var/log/caddy/netbox-access.log
format json
}
# Security headers
header {
# Enable HSTS
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
# Disable content type sniffing
X-Content-Type-Options "nosniff"
# Enable XSS protection
X-XSS-Protection "1; mode=block"
# Clickjacking protection
X-Frame-Options "SAMEORIGIN"
# Referrer policy
Referrer-Policy "strict-origin-when-cross-origin"
}
# Handle large uploads (for NetBox file uploads)
request_body {
max_size 100MB
}
}
# Redirect HTTP to HTTPS
http://{{ netbox_domain | default('netbox.vntx.net') }} {
redir https://{{ netbox_domain | default('netbox.vntx.net') }}{uri} permanent
}

37
ansible/roles/netbox/templates/docker-compose.override.yml.j2 Normal file
View file

@ -0,0 +1,37 @@
services:
netbox:
ports:
- "{{ netbox_listen_address }}:{{ netbox_port }}:8080"
volumes:
- {{ netbox_home }}/data/media:/opt/netbox/netbox/media
- {{ netbox_home }}/data/reports:/opt/netbox/netbox/reports
- {{ netbox_home }}/data/scripts:/opt/netbox/netbox/scripts
environment:
SKIP_SUPERUSER: "false"
SUPERUSER_API_TOKEN: "{{ netbox_superuser_api_token }}"
SUPERUSER_EMAIL: "{{ netbox_admin_email }}"
SUPERUSER_NAME: "{{ netbox_admin_user }}"
SUPERUSER_PASSWORD: "{{ netbox_admin_password }}"
restart: unless-stopped
netbox-worker:
volumes:
- {{ netbox_home }}/data/media:/opt/netbox/netbox/media
- {{ netbox_home }}/data/reports:/opt/netbox/netbox/reports
- {{ netbox_home }}/data/scripts:/opt/netbox/netbox/scripts
restart: unless-stopped
postgres:
volumes:
- {{ netbox_home }}/data/postgres:/var/lib/postgresql/data
restart: unless-stopped
redis:
volumes:
- {{ netbox_home }}/data/redis:/data
restart: unless-stopped
redis-cache:
volumes:
- {{ netbox_home }}/data/redis-cache:/data
restart: unless-stopped

82
ansible/roles/netbox/templates/env.j2 Normal file
View file

@ -0,0 +1,82 @@
# NetBox environment configuration
# {{ ansible_managed }}
# PostgreSQL database configuration
DB_NAME={{ netbox_db_name }}
DB_USER={{ netbox_db_user }}
DB_PASSWORD={{ netbox_db_password }}
DB_HOST={{ netbox_db_host }}
DB_PORT={{ netbox_db_port }}
# Redis configuration
REDIS_HOST={{ netbox_redis_host }}
REDIS_PORT={{ netbox_redis_port }}
REDIS_PASSWORD={{ netbox_redis_password }}
REDIS_DATABASE=0
REDIS_SSL=false
# Redis cache configuration
REDIS_CACHE_HOST={{ netbox_redis_cache_host }}
REDIS_CACHE_PORT={{ netbox_redis_cache_port }}
REDIS_CACHE_PASSWORD={{ netbox_redis_cache_password }}
REDIS_CACHE_DATABASE=1
REDIS_CACHE_SSL=false
# NetBox configuration
SECRET_KEY={{ netbox_secret_key }}
SUPERUSER_API_TOKEN={{ netbox_superuser_api_token }}
ALLOWED_HOSTS={{ netbox_allowed_hosts }}
DEBUG={{ netbox_debug | lower }}
METRICS_ENABLED={{ netbox_metrics_enabled | lower }}
# Email configuration
EMAIL_SERVER={{ netbox_email_server }}
EMAIL_PORT={{ netbox_email_port }}
EMAIL_USERNAME={{ netbox_email_username }}
EMAIL_PASSWORD={{ netbox_email_password }}
EMAIL_USE_SSL={{ netbox_email_use_ssl | lower }}
EMAIL_USE_TLS={{ netbox_email_use_tls | lower }}
EMAIL_SSL_CERTFILE=
EMAIL_SSL_KEYFILE=
EMAIL_FROM={{ netbox_email_from }}
EMAIL_TIMEOUT={{ netbox_email_timeout }}
# NAPALM configuration
NAPALM_USERNAME={{ netbox_napalm_username }}
NAPALM_PASSWORD={{ netbox_napalm_password }}
NAPALM_TIMEOUT={{ netbox_napalm_timeout }}
# Housekeeping
HOUSEKEEPING_INTERVAL={{ netbox_housekeeping_interval }}
# Webhooks
WEBHOOKS_ENABLED={{ netbox_webhooks_enabled | lower }}
# GraphQL API
GRAPHQL_ENABLED={{ netbox_graphql_enabled | lower }}
# Time zone
TZ={{ netbox_timezone }}
# Change logging
CHANGELOG_RETENTION={{ netbox_changelog_retention }}
# Job results
JOBRESULT_RETENTION={{ netbox_jobresult_retention }}
# Maximum upload size in bytes
MAX_UPLOAD_SIZE={{ netbox_max_upload_size }}
# Pagination
PAGINATE_COUNT={{ netbox_paginate_count }}
MAX_PAGE_SIZE={{ netbox_max_page_size }}
# Cache timeout
CACHE_TIMEOUT={{ netbox_cache_timeout }}
# CORS
CORS_ORIGIN_ALLOW_ALL={{ netbox_cors_origin_allow_all | lower }}
# Login persistence
LOGIN_PERSISTENCE={{ netbox_login_timeout }}
LOGIN_REQUIRED={{ netbox_login_required | lower }}

17
ansible/roles/netbox/templates/netbox-docker.service.j2 Normal file
View file

@ -0,0 +1,17 @@
[Unit]
Description=NetBox Docker Compose Application
Requires=docker.service
After=docker.service network-online.target
RequiresMountsFor={{ netbox_home }}
[Service]
Type=exec
WorkingDirectory={{ netbox_home }}/netbox-docker
ExecStart=/usr/bin/docker compose -f docker-compose.yml -f docker-compose.override.yml up
ExecStop=/usr/bin/docker compose -f docker-compose.yml -f docker-compose.override.yml down
TimeoutStopSec=70
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target

6
ansible/roles/netbox/templates/postgres.env.j2 Normal file
View file

@ -0,0 +1,6 @@
# PostgreSQL environment configuration
# {{ ansible_managed }}
POSTGRES_USER={{ netbox_db_user }}
POSTGRES_PASSWORD={{ netbox_db_password }}
POSTGRES_DB={{ netbox_db_name }}

6
ansible/roles/netbox/templates/redis-cache.env.j2 Normal file
View file

@ -0,0 +1,6 @@
# Redis cache environment configuration
# {{ ansible_managed }}
{% if netbox_redis_cache_password %}
REDIS_PASSWORD={{ netbox_redis_cache_password }}
{% endif %}

6
ansible/roles/netbox/templates/redis.env.j2 Normal file
View file

@ -0,0 +1,6 @@
# Redis environment configuration
# {{ ansible_managed }}
{% if netbox_redis_password %}
REDIS_PASSWORD={{ netbox_redis_password }}
{% endif %}

215
ansible/roles/raid/tasks/main.yml
View file

@ -1,215 +0,0 @@
---
- name: Install RAID management tools
apt:
name:
- mdadm
- smartmontools
- parted
- nvme-cli
- lsscsi
state: present
update_cache: yes
- name: Gather disk information
shell: |
echo "=== Block devices ==="
lsblk -d -o NAME,SIZE,TYPE,MODEL
echo "=== NVMe devices ==="
nvme list 2>/dev/null || echo "No NVMe devices found"
echo "=== SCSI devices ==="
lsscsi -s 2>/dev/null || echo "No SCSI devices found"
echo "=== All disk devices ==="
ls -la /dev/sd* /dev/nvme* /dev/vd* /dev/mmcblk* 2>/dev/null | grep -E "^b" || echo "Checking other paths..."
echo "=== Detailed block device tree ==="
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL
register: disk_info
changed_when: false
- name: Display disk information
debug:
msg: "{{ disk_info.stdout_lines }}"
- name: Check if we should auto-detect RAID devices
set_fact:
auto_detect_devices: "{{ raid_devices is not defined or raid_devices | length == 0 }}"
- name: Auto-detect largest unused disks
when: auto_detect_devices
block:
- name: Find unused block devices
shell: |
lsblk -dpno NAME,SIZE,TYPE,MOUNTPOINT | grep -E "disk.*$" | grep -v -E "/(|/boot)" | sort -k2 -h -r | head -5 | awk '{print $1}'
register: detected_devices
changed_when: false
- name: Set raid_devices from detected devices
set_fact:
raid_devices: "{{ detected_devices.stdout_lines }}"
when: detected_devices.stdout_lines | length >= 1
- name: Fail if no devices detected
fail:
msg: "Could not detect any unused block devices for RAID. Please specify raid_devices manually."
when: detected_devices.stdout_lines | length < 1
- name: Display RAID devices to be used
debug:
msg: "Will use these devices for RAID0: {{ raid_devices }}"
- name: Check if RAID array already exists
command: mdadm --detail /dev/md0
register: raid_status
failed_when: false
changed_when: false
- name: Stop and remove existing RAID array if it exists
block:
- name: Stop RAID array
command: mdadm --stop /dev/md0
when: raid_status.rc == 0
- name: Zero superblocks on disks
command: mdadm --zero-superblock {{ item }}
loop: "{{ raid_devices }}"
when: raid_status.rc == 0
when: force_raid_rebuild | default(false)
- name: Create partition table on RAID devices
parted:
device: "{{ item }}"
number: 1
state: present
part_type: primary
part_start: 0%
part_end: 100%
label: gpt
loop: "{{ raid_devices }}"
when: raid_status.rc != 0
- name: Build RAID device list with partitions
set_fact:
raid_partitions: "{{ raid_devices | map('regex_replace', '$', 'p1') | list }}"
- name: Create RAID0 array with dynamic device count
command: >
mdadm --create /dev/md0
--level=0
--raid-devices={{ raid_devices | length }}
{{ raid_partitions | join(' ') }}
--force
when: raid_status.rc != 0
register: raid_create
- name: Wait for RAID array to be ready
command: mdadm --detail /dev/md0
register: raid_ready
until: raid_ready.rc == 0
retries: 10
delay: 5
when: raid_create is changed
- name: Create mdadm configuration
shell: mdadm --detail --scan >> /etc/mdadm/mdadm.conf
when: raid_create is changed
- name: Check if update-initramfs exists
stat:
path: /usr/sbin/update-initramfs
register: initramfs_exists
- name: Update initramfs
command: update-initramfs -u
when:
- raid_create is changed
- initramfs_exists.stat.exists
- name: Check if filesystem exists on RAID array
command: blkid /dev/md0
register: filesystem_check
failed_when: false
changed_when: false
- name: Create filesystem on RAID array
filesystem:
fstype: ext4
dev: /dev/md0
opts: "-E lazy_itable_init=0,lazy_journal_init=0"
when: filesystem_check.rc != 0
- name: Create mount point for PostgreSQL data
file:
path: /mnt/pgdata
state: directory
mode: '0755'
- name: Mount RAID array
mount:
path: /mnt/pgdata
src: /dev/md0
fstype: ext4
opts: noatime,nodiratime,nobarrier
state: mounted
- name: Create PostgreSQL user and group
block:
- name: Create postgres group
group:
name: postgres
state: present
system: yes
- name: Create postgres user
user:
name: postgres
group: postgres
home: /var/lib/postgresql
shell: /bin/bash
system: yes
state: present
- name: Ensure PostgreSQL directories exist on RAID
file:
path: "{{ item }}"
state: directory
owner: postgres
group: postgres
mode: '0700'
loop:
- /mnt/pgdata
- /mnt/pgdata/16
- /mnt/pgdata/16/main
- /mnt/pgdata/wal
- /mnt/pgdata/archive
- /mnt/pgdata/backups
- name: Set up RAID monitoring
lineinfile:
path: /etc/mdadm/mdadm.conf
line: "MAILADDR root"
insertafter: EOF
- name: Enable mdadm monitoring service
systemd:
name: mdmonitor
enabled: yes
state: started
- name: Create RAID monitoring script
copy:
content: |
#!/bin/bash
# RAID health check script
RAID_STATUS=$(mdadm --detail /dev/md0 | grep "State :" | awk '{print $3}')
if [ "$RAID_STATUS" != "clean" ] && [ "$RAID_STATUS" != "active" ]; then
echo "RAID array /dev/md0 is in state: $RAID_STATUS"
exit 1
fi
exit 0
dest: /usr/local/bin/check-raid-health
mode: '0755'
- name: Set up RAID health check cron job
cron:
name: "Check RAID health"
minute: "*/5"
job: "/usr/local/bin/check-raid-health || echo 'RAID health check failed' | logger -t raid-health"

3
ansible/roles/syslog_servers/tasks/main.yml
View file

@ -1,3 +0,0 @@
---
# Syslog server configuration tasks
# TODO: Implement syslog server configuration

173
clusters/vntx/DEPLOYMENT.md
View file

@ -1,173 +0,0 @@
# VNTX Cluster Deployment Guide
This guide walks through deploying the VNTX K3s cluster on Proxmox to consolidate all infrastructure services.
## Prerequisites
- Proxmox cluster with sufficient resources
- OpenTofu/Terraform installed locally
- Ansible installed locally
- SSH access to Proxmox nodes
## Step 1: Deploy Flatcar VMs with OpenTofu
```bash
cd clusters/vntx/terraform
# Copy and edit the example vars
cp terraform.tfvars.example terraform.tfvars
vim terraform.tfvars
# Set Proxmox password
export TF_VAR_proxmox_password="your-proxmox-password"
# Initialize and deploy
tofu init
tofu plan
tofu apply
```
This creates:
- 3 control plane nodes (4 CPU, 8GB RAM, 50GB disk)
- 2 worker nodes (8 CPU, 16GB RAM, 50GB + 200GB disks)
## Step 2: Bootstrap K3s Cluster
```bash
cd ../ansible
# Prepare Flatcar nodes
ansible-playbook -i inventory.yml prepare-flatcar.yml
# Install K3s cluster
ansible-playbook -i inventory.yml k3s-install.yml
# Verify cluster
export KUBECONFIG=./kubeconfig
kubectl get nodes
```
## Step 3: Deploy Core System Services
```bash
cd ..
# Install MetalLB for load balancing
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.12/config/manifests/metallb-native.yaml
kubectl apply -f system/metallb/config.yaml
# Install Longhorn for storage
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.3/deploy/longhorn.yaml
# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.3/cert-manager.yaml
# Create namespaces
kubectl apply -f system/base/namespaces.yaml
```
## Step 4: Deploy Infrastructure Services
### Deploy MariaDB (shared database)
```bash
kubectl apply -f infrastructure/monitoring/mariadb/
# Create databases for services
kubectl exec -it -n monitoring mariadb-0 -- mysql -u root -p
CREATE DATABASE icinga2;
CREATE DATABASE icingaweb2;
CREATE USER 'icinga2'@'%' IDENTIFIED BY 'secure-password';
GRANT ALL ON icinga2.* TO 'icinga2'@'%';
CREATE USER 'icingaweb2'@'%' IDENTIFIED BY 'secure-password';
GRANT ALL ON icingaweb2.* TO 'icingaweb2'@'%';
```
### Deploy Monitoring Stack
```bash
# Create secrets
kubectl create secret generic -n monitoring icinga2-mysql \
--from-literal=password='secure-password'
kubectl create secret generic -n monitoring icingaweb2-mysql \
--from-literal=password='secure-password'
# Deploy Icinga2
kubectl apply -f infrastructure/monitoring/icinga2/
```
### Deploy Network Services
```bash
# Deploy Unbound DNS
kubectl apply -f infrastructure/network-services/unbound/
# Deploy FreeRADIUS
kubectl apply -f infrastructure/network-services/freeradius/
```
## Step 5: Configure Ingress and Access
```bash
# Get MetalLB assigned IPs
kubectl get svc -A | grep LoadBalancer
# Configure DNS entries for services
# icinga.vntx.net -> Icinga2 LoadBalancer IP
# radius.vntx.net -> FreeRADIUS LoadBalancer IP
```
## Service Migration Checklist
### From monitor.vntx.net (Icinga2):
- [ ] Export Icinga2 configuration
- [ ] Export monitoring database
- [ ] Migrate custom check scripts
- [ ] Update monitored hosts to point to new IP
- [ ] Verify all checks working
### From DNS servers:
- [ ] Export zone files
- [ ] Update forwarders configuration
- [ ] Test resolution
- [ ] Update DHCP to use new DNS
### From RADIUS server:
- [ ] Export user database
- [ ] Export clients configuration
- [ ] Test authentication
- [ ] Update network devices to use new RADIUS
## Maintenance
### Cluster Updates
```bash
# Flatcar auto-updates by default
# K3s updates:
curl -sfL https://get.k3s.io | sh -s - upgrade
```
### Backup
```bash
# Install Velero for cluster backup
# Backup to S3-compatible storage
```
### Monitoring
- Access Icinga2 Web: http://icinga.vntx.net
- Kubernetes dashboard: Install via Rancher or Lens
## Troubleshooting
### Check pod logs
```bash
kubectl logs -n monitoring deployment/icinga2
kubectl logs -n network-services deployment/freeradius
```
### Node issues
```bash
# SSH to Flatcar node
ssh core@10.0.0.101
# Check K3s service
sudo systemctl status k3s
sudo journalctl -u k3s -f
```

98
clusters/vntx/QUICKSTART.md
View file

@ -1,98 +0,0 @@
# Quick Start: Deploy K3s on Proxmox
## 1. Prerequisites
```bash
# Install OpenTofu (or use Terraform)
brew install opentofu
# Install Ansible
pip install ansible
# Get your SSH public key
cat ~/.ssh/id_rsa.pub
```
## 2. Configure for Your Environment
```bash
cd clusters/vntx/terraform
# Edit terraform.tfvars
vim terraform.tfvars
```
**Important settings to update:**
- `proxmox_nodes` - Set to your actual Proxmox node names (check in Proxmox UI)
- `ssh_public_key` - Replace with your actual SSH public key
- `control_plane_ips` and `worker_ips` - Adjust for your network
- `proxmox_storage` - Set to your storage name (usually "local-lvm" or "local")
## 3. Deploy Everything
```bash
cd ../
# Set Proxmox password
export TF_VAR_proxmox_password="your-proxmox-root-password"
# Run the deployment script
./deploy.sh
```
The script will:
1. Create 5 Flatcar Linux VMs on Proxmox
2. Install K3s cluster
3. Configure kubectl access
4. Optionally install MetalLB and Longhorn
## 4. Manual Steps (if not using deploy.sh)
```bash
# Step 1: Deploy VMs
cd terraform
tofu init
tofu plan
tofu apply
# Step 2: Wait for VMs to boot (60 seconds)
# Step 3: Install K3s
cd ../ansible
ansible-playbook -i inventory.yml prepare-flatcar.yml
ansible-playbook -i inventory.yml k3s-install.yml
# Step 4: Get kubeconfig
export KUBECONFIG=./kubeconfig
kubectl get nodes
```
## 5. Access Your Cluster
```bash
# Set kubeconfig
export KUBECONFIG=~/.kube/config-vntx
# Check nodes
kubectl get nodes
# Deploy a test service
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=LoadBalancer
```
## Troubleshooting
### If VMs don't start:
- Check Proxmox has enough resources
- Verify network bridge name in terraform.tfvars
- Check Proxmox storage has space
### If Ansible fails:
- Ensure VMs are accessible: `ping 10.0.0.101`
- Check SSH key is correct
- Wait longer for VMs to fully boot
### If K3s installation fails:
- SSH to node: `ssh core@10.0.0.101`
- Check logs: `sudo journalctl -u k3s -f`

70
clusters/vntx/README-DEPLOYMENT.md
View file

@ -1,70 +0,0 @@
# Deploying K3s on Proxmox with Debian 13
## Prerequisites
1. **Create Debian 13 template on Proxmox:**
```bash
# SSH to your Proxmox host (10.0.0.1 or 10.0.0.2)
ssh root@10.0.0.1
# Run the commands from setup-debian-template.sh
cat setup-debian-template.sh
```
2. **Update Proxmox node names:**
Edit `terraform/terraform.tfvars` and set your actual Proxmox node names
(you can find these in the Proxmox web UI)
3. **Verify your SSH key is set** (already done)
## Deploy the Cluster
```bash
# Set Proxmox password
export TF_VAR_proxmox_password="your-proxmox-password"
# Run deployment
cd clusters/vntx
./deploy.sh
```
## What Gets Deployed
- **5 Debian 13 VMs:**
- 3 control plane nodes (4 CPU, 8GB RAM each)
- 2 worker nodes (8 CPU, 16GB RAM each)
- Worker nodes get extra 200GB disk for container storage
- **K3s Kubernetes cluster** with:
- High availability control plane
- MetalLB for load balancing
- Longhorn for distributed storage
## After Deployment
Your services will be accessible via MetalLB LoadBalancer IPs:
- Icinga2: http://10.0.0.200 (example)
- FreeRADIUS: 10.0.0.201:1812 (example)
- Unbound DNS: Running on all nodes
## Manual Steps Summary
If the script fails, you can run manually:
```bash
# 1. Create VMs
cd terraform
tofu init
tofu apply
# 2. Configure nodes
cd ../ansible
ansible-playbook -i inventory.yml prepare-debian.yml
# 3. Install K3s
ansible-playbook -i inventory.yml k3s-install.yml
# 4. Use cluster
export KUBECONFIG=./kubeconfig
kubectl get nodes
```

36
clusters/vntx/README.md
View file

@ -1,36 +0,0 @@
# VNTX K3s Cluster
This cluster runs all infrastructure services previously deployed as individual VMs on Proxmox.
## Architecture
- **Platform**: Flatcar Linux on Proxmox VE
- **Kubernetes**: K3s (lightweight Kubernetes)
- **Nodes**: 3 control plane + 2 worker nodes
- **Storage**: Longhorn distributed storage
- **Load Balancer**: MetalLB
## Services
### Infrastructure Services (`infrastructure/`)
- **Monitoring**: Icinga2, Prometheus, Grafana
- **DNS**: Unbound, CoreDNS
- **Authentication**: FreeRADIUS
- **Reverse Proxy**: Caddy
### System Services (`system/`)
- **Storage**: Longhorn
- **Ingress**: Traefik (K3s default)
- **Certificates**: cert-manager
- **Backups**: Velero
### Applications (`apps/`)
- User-facing applications
## Deployment Order
1. Deploy Flatcar VMs via Terraform
2. Bootstrap K3s cluster via Ansible
3. Deploy system services
4. Deploy infrastructure services
5. Migrate data from existing VMs

31
clusters/vntx/ansible/inventory.yml
View file

@ -1,31 +0,0 @@
all:
children:
k3s_cluster:
children:
control_plane:
hosts:
vntx-control-1:
ansible_host: 10.0.0.101
k3s_control_node: true
k3s_primary: true
vntx-control-2:
ansible_host: 10.0.0.102
k3s_control_node: true
vntx-control-3:
ansible_host: 10.0.0.103
k3s_control_node: true
workers:
hosts:
vntx-worker-1:
ansible_host: 10.0.0.111
vntx-worker-2:
ansible_host: 10.0.0.112
vars:
ansible_user: debian
ansible_ssh_common_args: '-o StrictHostKeyChecking=no'
ansible_python_interpreter: /usr/bin/python3
k3s_version: v1.29.0+k3s1
k3s_cluster_domain: cluster.local
k3s_cluster_cidr: 10.42.0.0/16
k3s_service_cidr: 10.43.0.0/16
systemd_dir: /etc/systemd/system

160
clusters/vntx/ansible/k3s-install.yml
View file

@ -1,160 +0,0 @@
---
- name: Prepare Debian nodes for K3s
hosts: k3s_cluster
become: yes
tasks:
- name: Configure sysctl for K3s
ansible.posix.sysctl:
name: "{{ item }}"
value: '1'
state: present
reload: yes
loop:
- net.bridge.bridge-nf-call-iptables
- net.ipv4.ip_forward
- net.bridge.bridge-nf-call-ip6tables
- name: Load required kernel modules
community.general.modprobe:
name: "{{ item }}"
state: present
loop:
- br_netfilter
- overlay
- name: Ensure kernel modules load on boot
ansible.builtin.copy:
dest: /etc/modules-load.d/k3s.conf
content: |
br_netfilter
overlay
- name: Deploy K3s on primary control node
hosts: control_plane
become: yes
tasks:
- name: Download K3s installer
ansible.builtin.get_url:
url: https://get.k3s.io
dest: /tmp/k3s-install.sh
mode: '0755'
when: k3s_primary | default(false)
- name: Install K3s on primary control node
ansible.builtin.shell: |
INSTALL_K3S_VERSION="{{ k3s_version }}" \
K3S_CLUSTER_INIT=true \
INSTALL_K3S_EXEC="server \
--cluster-cidr={{ k3s_cluster_cidr }} \
--service-cidr={{ k3s_service_cidr }} \
--cluster-domain={{ k3s_cluster_domain }} \
--disable traefik \
--disable servicelb \
--write-kubeconfig-mode=644" \
/tmp/k3s-install.sh
when: k3s_primary | default(false)
register: k3s_primary_install
- name: Get node token
ansible.builtin.slurp:
src: /var/lib/rancher/k3s/server/node-token
register: node_token
when: k3s_primary | default(false)
- name: Set token fact
ansible.builtin.set_fact:
k3s_token: "{{ node_token.content | b64decode | trim }}"
when: k3s_primary | default(false)
- name: Get primary node URL
ansible.builtin.set_fact:
k3s_url: "https://{{ ansible_host }}:6443"
when: k3s_primary | default(false)
- name: Deploy K3s on secondary control nodes
hosts: control_plane:!vntx-control-1
become: yes
tasks:
- name: Download K3s installer
ansible.builtin.get_url:
url: https://get.k3s.io
dest: /tmp/k3s-install.sh
mode: '0755'
- name: Get primary node facts
ansible.builtin.set_fact:
k3s_url: "{{ hostvars['vntx-control-1']['k3s_url'] }}"
k3s_token: "{{ hostvars['vntx-control-1']['k3s_token'] }}"
- name: Install K3s on secondary control nodes
ansible.builtin.shell: |
INSTALL_K3S_VERSION="{{ k3s_version }}" \
K3S_URL="{{ k3s_url }}" \
K3S_TOKEN="{{ k3s_token }}" \
INSTALL_K3S_EXEC="server \
--cluster-cidr={{ k3s_cluster_cidr }} \
--service-cidr={{ k3s_service_cidr }} \
--cluster-domain={{ k3s_cluster_domain }} \
--disable traefik \
--disable servicelb \
--write-kubeconfig-mode=644" \
/tmp/k3s-install.sh
- name: Deploy K3s on worker nodes
hosts: workers
become: yes
tasks:
- name: Download K3s installer
ansible.builtin.get_url:
url: https://get.k3s.io
dest: /tmp/k3s-install.sh
mode: '0755'
- name: Get primary node facts
ansible.builtin.set_fact:
k3s_url: "{{ hostvars['vntx-control-1']['k3s_url'] }}"
k3s_token: "{{ hostvars['vntx-control-1']['k3s_token'] }}"
- name: Install K3s agent on worker nodes
ansible.builtin.shell: |
INSTALL_K3S_VERSION="{{ k3s_version }}" \
K3S_URL="{{ k3s_url }}" \
K3S_TOKEN="{{ k3s_token }}" \
INSTALL_K3S_EXEC="agent" \
/tmp/k3s-install.sh
- name: Configure kubectl access
hosts: vntx-control-1
become: yes
tasks:
- name: Create .kube directory
ansible.builtin.file:
path: /home/debian/.kube
state: directory
owner: debian
group: debian
mode: '0755'
- name: Copy kubeconfig
ansible.builtin.copy:
src: /etc/rancher/k3s/k3s.yaml
dest: /home/debian/.kube/config
owner: debian
group: debian
mode: '0600'
remote_src: yes
- name: Fetch kubeconfig to local machine
ansible.builtin.fetch:
src: /etc/rancher/k3s/k3s.yaml
dest: ./kubeconfig
flat: yes
- name: Display cluster info
ansible.builtin.shell: kubectl get nodes
become_user: debian
register: nodes_output
- name: Show nodes
ansible.builtin.debug:
var: nodes_output.stdout_lines

19
clusters/vntx/ansible/kubeconfig
View file

@ -1,19 +0,0 @@
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUzTlRZNU1qVTBPRFl3SGhjTk1qVXdPVEF6TVRnMU1USTJXaGNOTXpVd09UQXhNVGcxTVRJMgpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUzTlRZNU1qVTBPRFl3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFRWC9KVDdROEV1RktCMWhWUVRHNVVZNXFmVmljcFAybkN5eHE0Tmd1T3AKZCtUSTI4SWgycmVSQ2FsL2d2enJLamxvbTR2ekc5eDJUUnpwMW1LTGJHUlFvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVVAzdG5tcFphN2RIUFluaFBZM2NXCitVNmxnNGd3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQVBJWS9jdzBNYXljV3RwNDFDbDZnZEZOY0xBUzNBelAKN21pZWJXcGFWbUNSQWlCaTdaME9mR0ZoeVRoL1BTVG5Qam5weXJzQnprcUlPeEVkU3U2aEdCMnphZz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://127.0.0.1:6443
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRlZ0F3SUJBZ0lJZUwwRzJmMDB3SkF3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOelUyT1RJMU5EZzJNQjRYRFRJMU1Ea3dNekU0TlRFeU5sb1hEVEkyTURrdwpNekU0TlRFeU5sb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJPbGhUT1V3dmVGbjlDTTcKSHZPVk94bVM1TXRsbHM2ZUNOOHN4dTg2NGQzQjBnK0grZnFBelBvbUNWdEp0UmpNdEJhSUltSU9jUWx0NFlqSQpTR1c4a05TalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCVG9nRjNxOHF5eUllclNuc1Y1dDIxaTQ5MEpCREFLQmdncWhrak9QUVFEQWdOSUFEQkYKQWlFQXFvaDJIR1NYMGlObGo0RWpmVFJXcFBaWFZ1blhUcU5SSmw5UGQxdUc1RWdDSUMvMUk2eGZQZFdnN0d6ZQoyVkl0UjlqNzJFRVpFRTdjUFQwcFRUcExwdld3Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdFkyeHAKWlc1MExXTmhRREUzTlRZNU1qVTBPRFl3SGhjTk1qVXdPVEF6TVRnMU1USTJXaGNOTXpVd09UQXhNVGcxTVRJMgpXakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwWlc1MExXTmhRREUzTlRZNU1qVTBPRFl3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFUc2lmYnZWTEpGYjRHZmpKWHhMZ2gwbndCdW9KdUhUd3NRMWM4R09lQnMKSGtVbGtrOFc0clVVb1IybkMxdVdEeEF5S3N6bE1PRHF1VXVWOXBmT1hRcXFvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVTZJQmQ2dktzc2lIcTBwN0ZlYmR0Cll1UGRDUVF3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnU0RuNkh4UU12b2lLNkx3QkdpQW9kRFFQam1CYlUvaWkKQmVCVWFNazRyMXNDSVFEeFpYUTQxWEl3MTRJMWtsdW9wOHNRVVYzTEtWbEtqNTBiaXc5YXpia2xxUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSU9RdlF2RVpNZ0R3aHVSTUY1NUE4V2J4WUI0VDVQZTFIVW8xVlVnbkVvdDlvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFNldGTTVUQzk0V2YwSXpzZTg1VTdHWkxreTJXV3pwNEkzeXpHN3pyaDNjSFNENGY1K29ETQoraVlKVzBtMUdNeTBGb2dpWWc1eENXM2hpTWhJWmJ5UTFBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=

167
clusters/vntx/ansible/prepare-debian.yml
View file

@ -1,167 +0,0 @@
---
- name: Prepare Debian 13 nodes for K3s
hosts: k3s_cluster
become: yes
tasks:
- name: Wait for cloud-init to complete
ansible.builtin.wait_for:
path: /var/lib/cloud/instance/boot-finished
timeout: 300
- name: Wait for apt locks to be released
ansible.builtin.shell: |
while fuser /var/lib/apt/lists/lock >/dev/null 2>&1; do
echo "Waiting for apt lock..."
sleep 5
done
changed_when: false
- name: Create systemd-resolved config directory
ansible.builtin.file:
path: /etc/systemd/resolved.conf.d
state: directory
mode: '0755'
- name: Configure DNS for internet access
ansible.builtin.copy:
dest: /etc/systemd/resolved.conf.d/dns.conf
content: |
[Resolve]
DNS=9.9.9.9
FallbackDNS=149.112.112.112
Domains=~.
mode: '0644'
notify: restart systemd-resolved
- name: Restart systemd-resolved immediately
ansible.builtin.systemd:
name: systemd-resolved
state: restarted
daemon_reload: yes
- name: Update apt cache
ansible.builtin.apt:
update_cache: yes
- name: Install required packages
ansible.builtin.apt:
name:
- curl
- apt-transport-https
- ca-certificates
- gnupg
- lsb-release
- iptables
- open-iscsi
- nfs-common
state: present
- name: Load required kernel modules
community.general.modprobe:
name: "{{ item }}"
state: present
loop:
- br_netfilter
- overlay
- name: Configure sysctl for K3s
ansible.posix.sysctl:
name: "{{ item }}"
value: '1'
state: present
reload: yes
loop:
- net.bridge.bridge-nf-call-iptables
- net.ipv4.ip_forward
- net.bridge.bridge-nf-call-ip6tables
- name: Ensure kernel modules load on boot
ansible.builtin.copy:
dest: /etc/modules-load.d/k3s.conf
content: |
br_netfilter
overlay
mode: '0644'
- name: Format and mount additional storage disk (workers only)
when: inventory_hostname in groups['workers']
block:
- name: Find additional storage disk
ansible.builtin.shell: |
# Find the 200G disk which is our storage disk
for disk in /dev/sda /dev/sdb /dev/sdc; do
if [ -b "$disk" ]; then
size=$(lsblk -d -n -o SIZE "$disk" | tr -d ' ')
if [ "$size" = "200G" ]; then
echo "$disk"
exit 0
fi
fi
done
exit 1
register: storage_disk_path
failed_when: storage_disk_path.rc != 0
- name: Check if storage disk has a filesystem
ansible.builtin.command: "blkid {{ storage_disk_path.stdout }}"
register: blkid_result
failed_when: false
changed_when: false
- name: Create filesystem on storage disk
ansible.builtin.filesystem:
fstype: ext4
dev: "{{ storage_disk_path.stdout }}"
force: no
when: blkid_result.rc != 0
- name: Create mount point for container storage
ansible.builtin.file:
path: /var/lib/rancher
state: directory
mode: '0755'
- name: Get UUID of storage disk
ansible.builtin.command: "blkid -s UUID -o value {{ storage_disk_path.stdout }}"
register: storage_uuid
changed_when: false
- name: Mount storage disk by UUID
ansible.posix.mount:
path: /var/lib/rancher
src: "UUID={{ storage_uuid.stdout }}"
fstype: ext4
opts: defaults
state: mounted
- name: Configure systemd-resolved for cluster DNS
ansible.builtin.file:
path: /etc/systemd/resolved.conf.d
state: directory
mode: '0755'
- name: Configure cluster DNS
ansible.builtin.copy:
dest: /etc/systemd/resolved.conf.d/k3s.conf
content: |
[Resolve]
DNS=10.43.0.10
Domains=~cluster.local
mode: '0644'
notify: restart systemd-resolved
- name: Configure container runtime limits
ansible.builtin.copy:
dest: /etc/security/limits.d/k3s.conf
content: |
* soft nofile 65536
* hard nofile 65536
* soft nproc 65536
* hard nproc 65536
mode: '0644'
handlers:
- name: restart systemd-resolved
ansible.builtin.systemd:
name: systemd-resolved
state: restarted

66
clusters/vntx/create-vms-proxmox.sh
View file

@ -1,66 +0,0 @@
#!/bin/bash
# Script to create K3s VMs directly on Proxmox
# Run this on your Proxmox host (10.0.0.2)
echo "Creating K3s cluster VMs on Proxmox..."
# Your SSH public key
SSH_KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHLyVSEEAEbGZZqwPwEup0XrlTpu3x3iF9ExvvQPA4xN graham@mcintire.me"
# Create a temporary file with the SSH key
echo "$SSH_KEY" > /tmp/authorized_keys
# Start with higher VM IDs to avoid conflicts
BASE_ID=130
echo "Using VM IDs starting from $BASE_ID"
# Control plane nodes
for i in 1 2 3; do
VMID=$((BASE_ID + i - 1))
echo "Creating vntx-control-$i (VM $VMID)..."
qm clone 9000 $VMID --name vntx-control-$i
qm set $VMID \
--cores 4 \
--memory 8192 \
--net0 virtio,bridge=vmbr0 \
--ipconfig0 ip=10.0.0.$((100 + i))/24,gw=10.0.0.1 \
--sshkey /tmp/authorized_keys
qm resize $VMID scsi0 50G
qm start $VMID
sleep 2
done
# Worker nodes
for i in 1 2; do
VMID=$((BASE_ID + 3 + i - 1))
echo "Creating vntx-worker-$i (VM $VMID)..."
qm clone 9000 $VMID --name vntx-worker-$i
qm set $VMID \
--cores 8 \
--memory 16384 \
--net0 virtio,bridge=vmbr0 \
--ipconfig0 ip=10.0.0.$((110 + i))/24,gw=10.0.0.1 \
--sshkey /tmp/authorized_keys
qm resize $VMID scsi0 50G
# Add additional disk for storage
qm set $VMID --scsi1 local-lvm:200
qm start $VMID
sleep 2
done
# Clean up
rm -f /tmp/authorized_keys
echo
echo "All VMs created and started!"
echo
echo "VMs created:"
echo " vntx-control-1: 10.0.0.101 (VM ID: 130)"
echo " vntx-control-2: 10.0.0.102 (VM ID: 131)"
echo " vntx-control-3: 10.0.0.103 (VM ID: 132)"
echo " vntx-worker-1: 10.0.0.111 (VM ID: 133)"
echo " vntx-worker-2: 10.0.0.112 (VM ID: 134)"
echo
echo "Wait 60 seconds for cloud-init to complete, then run Ansible"

177
clusters/vntx/deploy.sh
View file

@ -1,177 +0,0 @@
#!/bin/bash
set -e
echo "=== VNTX K3s Cluster Deployment Script ==="
echo
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
# Check prerequisites
echo "Checking prerequisites..."
if ! command -v tofu &> /dev/null && ! command -v terraform &> /dev/null; then
echo -e "${RED}ERROR: OpenTofu or Terraform not found${NC}"
echo "Install OpenTofu: https://opentofu.org/docs/intro/install/"
exit 1
fi
if ! command -v ansible &> /dev/null; then
echo -e "${RED}ERROR: Ansible not found${NC}"
echo "Install Ansible: pip install ansible"
exit 1
fi
if ! command -v kubectl &> /dev/null; then
echo -e "${YELLOW}WARNING: kubectl not found${NC}"
echo "Install kubectl to manage the cluster after deployment"
fi
# Use tofu if available, otherwise terraform
TF_CMD="tofu"
if ! command -v tofu &> /dev/null; then
TF_CMD="terraform"
fi
echo -e "${GREEN}Prerequisites OK${NC}"
echo
# Step 1: Configure Terraform
echo "=== Step 1: Configure Terraform ==="
cd terraform
if [ ! -f terraform.tfvars ]; then
echo -e "${RED}ERROR: terraform.tfvars not found${NC}"
echo "Please edit terraform/terraform.tfvars with your settings"
exit 1
fi
# Check for SSH key
if grep -q "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC..." terraform.tfvars; then
echo -e "${RED}ERROR: SSH public key not configured${NC}"
echo "Please add your SSH public key to terraform.tfvars"
echo "You can find it in ~/.ssh/id_rsa.pub or ~/.ssh/id_ed25519.pub"
exit 1
fi
# Verify SSH key is present
if ! grep -E "ssh-(rsa|ed25519|ecdsa)" terraform.tfvars > /dev/null; then
echo -e "${RED}ERROR: No valid SSH key found in terraform.tfvars${NC}"
echo "Please add your SSH public key to terraform.tfvars"
exit 1
fi
# Get Proxmox password
if [ -z "$TF_VAR_proxmox_password" ]; then
echo -n "Enter Proxmox password: "
read -s TF_VAR_proxmox_password
export TF_VAR_proxmox_password
echo
fi
# Initialize Terraform
echo "Initializing Terraform..."
$TF_CMD init
# Plan deployment
echo
echo "Planning infrastructure..."
$TF_CMD plan -out=tfplan
echo
read -p "Deploy infrastructure? (y/n) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
exit 1
fi
# Deploy VMs
echo "Deploying Flatcar VMs on Proxmox..."
$TF_CMD apply tfplan
echo -e "${GREEN}Infrastructure deployed!${NC}"
# Step 2: Wait for VMs to be ready
echo
echo "=== Step 2: Waiting for VMs to be ready ==="
cd ../ansible
echo "Waiting 60 seconds for VMs to boot..."
sleep 60
# Test connectivity
echo "Testing connectivity to nodes..."
ansible -i inventory.yml all -m ping --one-line
# Step 3: Prepare Debian nodes
echo
echo "=== Step 3: Preparing Debian nodes ==="
ansible-playbook -i inventory.yml prepare-debian.yml
# Step 4: Install K3s
echo
echo "=== Step 4: Installing K3s cluster ==="
ansible-playbook -i inventory.yml k3s-install.yml
# Step 5: Setup kubeconfig
echo
echo "=== Step 5: Setting up kubectl access ==="
if [ -f kubeconfig ]; then
mkdir -p ~/.kube
cp kubeconfig ~/.kube/config-vntx
export KUBECONFIG=~/.kube/config-vntx
echo "Cluster nodes:"
kubectl get nodes
echo
echo -e "${GREEN}K3s cluster deployed successfully!${NC}"
echo
echo "To use this cluster:"
echo " export KUBECONFIG=~/.kube/config-vntx"
echo " kubectl get nodes"
else
echo -e "${RED}ERROR: kubeconfig not found${NC}"
exit 1
fi
# Step 6: Deploy core services
echo
read -p "Deploy core services (MetalLB, Longhorn)? (y/n) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
cd ..
echo "Creating namespaces..."
kubectl apply -f system/base/namespaces.yaml
echo "Installing MetalLB..."
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.12/config/manifests/metallb-native.yaml
echo "Waiting for MetalLB to be ready..."
kubectl wait --namespace metallb-system \
--for=condition=ready pod \
--selector=app=metallb \
--timeout=90s
echo "Configuring MetalLB..."
kubectl apply -f system/metallb/config.yaml
echo "Installing Longhorn..."
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.3/deploy/longhorn.yaml
echo -e "${GREEN}Core services deployed!${NC}"
fi
echo
echo "=== Deployment Complete ==="
echo
echo "Cluster endpoints:"
echo " Control plane: https://10.0.0.101:6443"
echo
echo "Next steps:"
echo "1. Deploy your services: kubectl apply -f infrastructure/"
echo "2. Check the deployment guide: cat DEPLOYMENT.md"

159
clusters/vntx/infrastructure/monitoring/icinga2/deployment.yaml
View file

@ -1,159 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: icinga2-config
namespace: monitoring
data:
icinga2.conf: |
include "constants.conf"
include "zones.conf"
include <itl>
include <plugins>
include <plugins-contrib>
include <manubulon>
include <windows-plugins>
include <nscp>
include "features-enabled/*.conf"
include_recursive "conf.d"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: icinga2-data
namespace: monitoring
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: icinga2
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: icinga2
template:
metadata:
labels:
app: icinga2
spec:
containers:
- name: icinga2
image: icinga/icinga2:2.14.0
ports:
- containerPort: 5665
name: api
env:
- name: ICINGA2_MASTER
value: "1"
- name: ICINGA2_CN
value: "icinga2.monitoring.svc.cluster.local"
- name: MYSQL_HOST
value: "mariadb.monitoring.svc.cluster.local"
- name: MYSQL_DATABASE
value: "icinga2"
- name: MYSQL_USER
value: "icinga2"
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: icinga2-mysql
key: password
volumeMounts:
- name: config
mountPath: /etc/icinga2/icinga2.conf
subPath: icinga2.conf
- name: data
mountPath: /var/lib/icinga2
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "2000m"
volumes:
- name: config
configMap:
name: icinga2-config
- name: data
persistentVolumeClaim:
claimName: icinga2-data
---
apiVersion: v1
kind: Service
metadata:
name: icinga2
namespace: monitoring
spec:
selector:
app: icinga2
ports:
- port: 5665
targetPort: 5665
name: api
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: icingaweb2
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: icingaweb2
template:
metadata:
labels:
app: icingaweb2
spec:
containers:
- name: icingaweb2
image: icinga/icingaweb2:2.12.1
ports:
- containerPort: 80
name: http
env:
- name: MYSQL_HOST
value: "mariadb.monitoring.svc.cluster.local"
- name: MYSQL_DATABASE
value: "icingaweb2"
- name: MYSQL_USER
value: "icingaweb2"
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: icingaweb2-mysql
key: password
- name: ICINGA2_API_HOST
value: "icinga2.monitoring.svc.cluster.local"
- name: ICINGA2_API_PORT
value: "5665"
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
name: icingaweb2
namespace: monitoring
spec:
selector:
app: icingaweb2
ports:
- port: 80
targetPort: 80
name: http

133
clusters/vntx/infrastructure/network-services/freeradius/deployment.yaml
View file

@ -1,133 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: freeradius-config
namespace: network-services
data:
radiusd.conf: |
prefix = /usr
exec_prefix = /usr
sysconfdir = /etc
localstatedir = /var
sbindir = ${exec_prefix}/sbin
logdir = /var/log/freeradius
raddbdir = /etc/freeradius
radacctdir = ${logdir}/radacct
name = radiusd
confdir = ${raddbdir}
modconfdir = ${confdir}/mods-config
certdir = ${confdir}/certs
cadir = ${confdir}/certs
run_dir = ${localstatedir}/run/${name}
db_dir = ${localstatedir}/lib/${name}
libdir = ${exec_prefix}/lib/freeradius
pidfile = ${run_dir}/${name}.pid
max_request_time = 30
cleanup_delay = 5
max_requests = 1024
listen {
type = auth
ipaddr = *
port = 1812
}
listen {
type = acct
ipaddr = *
port = 1813
}
log {
destination = stdout
auth = yes
auth_badpass = no
auth_goodpass = no
}
$INCLUDE mods-enabled/
$INCLUDE sites-enabled/
---
apiVersion: v1
kind: Secret
metadata:
name: freeradius-secrets
namespace: network-services
type: Opaque
data:
# Base64 encoded secrets
clients.conf: |
Y2xpZW50IGxvY2FsaG9zdCB7CiAgICBpcGFkZHIgPSAxMjcuMC4wLjEKICAgIHNlY3JldCA9IHRl
c3RpbmcxMjMKfQo=
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: freeradius
namespace: network-services
spec:
replicas: 2
selector:
matchLabels:
app: freeradius
template:
metadata:
labels:
app: freeradius
spec:
containers:
- name: freeradius
image: freeradius/freeradius-server:latest
ports:
- containerPort: 1812
protocol: UDP
name: auth
- containerPort: 1813
protocol: UDP
name: acct
volumeMounts:
- name: config
mountPath: /etc/freeradius/radiusd.conf
subPath: radiusd.conf
- name: secrets
mountPath: /etc/freeradius/clients.conf
subPath: clients.conf
- name: data
mountPath: /var/lib/freeradius
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
volumes:
- name: config
configMap:
name: freeradius-config
- name: secrets
secret:
secretName: freeradius-secrets
- name: data
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: freeradius
namespace: network-services
spec:
selector:
app: freeradius
type: LoadBalancer
ports:
- port: 1812
targetPort: 1812
protocol: UDP
name: auth
- port: 1813
targetPort: 1813
protocol: UDP
name: acct

117
clusters/vntx/infrastructure/network-services/unbound/deployment.yaml
View file

@ -1,117 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: unbound-config
namespace: network-services
data:
unbound.conf: |
server:
interface: 0.0.0.0
access-control: 10.0.0.0/8 allow
access-control: 172.16.0.0/12 allow
access-control: 192.168.0.0/16 allow
verbosity: 1
# Performance tuning
num-threads: 4
msg-cache-slabs: 8
rrset-cache-slabs: 8
infra-cache-slabs: 8
key-cache-slabs: 8
# Cache sizes
rrset-cache-size: 256m
msg-cache-size: 128m
# Security
hide-identity: yes
hide-version: yes
# Forward zones
include: /etc/unbound/forward-zones.conf
remote-control:
control-enable: yes
control-interface: 127.0.0.1
---
apiVersion: v1
kind: ConfigMap
metadata:
name: unbound-forward-zones
namespace: network-services
data:
forward-zones.conf: |
forward-zone:
name: "."
forward-addr: 8.8.8.8
forward-addr: 8.8.4.4
forward-addr: 1.1.1.1
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: unbound
namespace: network-services
spec:
selector:
matchLabels:
app: unbound
template:
metadata:
labels:
app: unbound
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: unbound
image: mvance/unbound:latest
ports:
- containerPort: 53
protocol: UDP
hostPort: 53
- containerPort: 53
protocol: TCP
hostPort: 53
volumeMounts:
- name: config
mountPath: /opt/unbound/etc/unbound/unbound.conf
subPath: unbound.conf
- name: forward-zones
mountPath: /etc/unbound/forward-zones.conf
subPath: forward-zones.conf
securityContext:
capabilities:
add: ["NET_BIND_SERVICE"]
resources:
requests:
memory: "512Mi"
cpu: "100m"
limits:
memory: "1Gi"
cpu: "500m"
volumes:
- name: config
configMap:
name: unbound-config
- name: forward-zones
configMap:
name: unbound-forward-zones
---
apiVersion: v1
kind: Service
metadata:
name: unbound
namespace: network-services
spec:
selector:
app: unbound
clusterIP: None
ports:
- port: 53
protocol: UDP
name: dns-udp
- port: 53
protocol: TCP
name: dns-tcp

40
clusters/vntx/setup-debian-template.sh
View file

@ -1,40 +0,0 @@
#!/bin/bash
# Script to create Debian 13 cloud-init template on Proxmox
echo "This script will create a Debian 13 (Bookworm) cloud-init template on your Proxmox host"
echo "Run this on your Proxmox host (10.0.0.1 or 10.0.0.2)"
echo
echo "Commands to run:"
echo
cat << 'EOF'
# Download Debian 13 cloud image
wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-13-genericcloud-amd64.qcow2
# Create a new VM
qm create 9000 --name debian13-cloud --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0
# Import the downloaded disk to the VM
qm importdisk 9000 debian-13-genericcloud-amd64.qcow2 local-lvm
# Attach the disk to the VM
qm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9000-disk-0
# Set boot disk
qm set 9000 --boot c --bootdisk scsi0
# Add cloud-init drive
qm set 9000 --ide2 local-lvm:cloudinit
# Configure serial console
qm set 9000 --serial0 socket --vga serial0
# Enable QEMU agent
qm set 9000 --agent enabled=1
# Convert to template
qm template 9000
echo "Template created! VM ID 9000 named 'debian13-cloud'"
EOF

40
clusters/vntx/setup-ubuntu-template.sh
View file

@ -1,40 +0,0 @@
#!/bin/bash
# Script to create Ubuntu cloud-init template on Proxmox
echo "This script will create an Ubuntu 22.04 cloud-init template on your Proxmox host"
echo "Run this on your Proxmox host (10.0.0.1)"
echo
echo "Commands to run:"
echo
cat << 'EOF'
# Download Ubuntu 22.04 cloud image
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
# Create a new VM
qm create 9000 --name ubuntu-cloud --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0
# Import the downloaded disk to the VM
qm importdisk 9000 jammy-server-cloudimg-amd64.img local-lvm
# Attach the disk to the VM
qm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9000-disk-0
# Set boot disk
qm set 9000 --boot c --bootdisk scsi0
# Add cloud-init drive
qm set 9000 --ide2 local-lvm:cloudinit
# Configure serial console
qm set 9000 --serial0 socket --vga serial0
# Enable QEMU agent
qm set 9000 --agent enabled=1
# Convert to template
qm template 9000
echo "Template created! VM ID 9000 named 'ubuntu-cloud'"
EOF

29
clusters/vntx/system/base/namespaces.yaml
View file

@ -1,29 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: metallb-system
---
apiVersion: v1
kind: Namespace
metadata:
name: longhorn-system
---
apiVersion: v1
kind: Namespace
metadata:
name: cert-manager
---
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
---
apiVersion: v1
kind: Namespace
metadata:
name: network-services
---
apiVersion: v1
kind: Namespace
metadata:
name: ingress-system

17
clusters/vntx/system/metallb/config.yaml
View file

@ -1,17 +0,0 @@
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: default-pool
namespace: metallb-system
spec:
addresses:
- 10.0.0.200-10.0.0.250 # Adjust based on your network
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: default-advertisement
namespace: metallb-system
spec:
ipAddressPools:
- default-pool

1
clusters/vntx/terraform.tfstate
View file

@ -1 +0,0 @@
{"version":4,"terraform_version":"1.10.5","serial":1,"lineage":"22c48096-8818-3e9f-61ec-3007015f4917","outputs":{},"resources":[],"check_results":null}

24
clusters/vntx/terraform/.terraform.lock.hcl generated
View file

@ -1,24 +0,0 @@
# This file is maintained automatically by "tofu init".
# Manual edits may be lost in future updates.
provider "registry.opentofu.org/telmate/proxmox" {
version = "2.9.14"
constraints = "~> 2.9"
hashes = [
"h1:asZa5VKbWeCpLNv1JAutt5CdD27HaGFjxxcr6mvn8Ps=",
"zh:0d049d33f705e5b814d30028770c084151218439424e99684ce31d7e26a720b5",
"zh:20b1c64ed56d81de95f3f37b82b45b4654c0de26670c0e87a474c5cce13cd015",
"zh:2946058abd1d8e50e475b9ec39781eb02576b40dbd80f4653fade4493a4514c6",
"zh:29e50a25c456f040ce072f23ac57b5b82ebd3b916ca5ae6688332b5ec62adc4a",
"zh:3612932306ce5f08db94868f526cbb8c56d0d3c6ebe1c11a83f92bbf94354296",
"zh:42d1699b0abebaac82ea5a19f4393541d8bb2741bde204a8ac1028cdc29d1b14",
"zh:5ffd5dc567262eb8aafdf2f6eac63f7f21361da9c5d75a3c36b479638a0001b0",
"zh:6692ef323e3b89de99934ad731f6a1850525bf8142916ae28ea4e4048d73a787",
"zh:a5afc98e9a4038516bb58e788cb77dea67a60dce780dfcd206d7373c5a56b776",
"zh:bf902cded709d84fa27fbf91b589c241f2238a6c4924e4e479eebd74320b93a5",
"zh:cab0e1e72c9cebcf669fc6f35ec28cb8ab2dffb0237afc8860aa40d23bf8a49f",
"zh:e523b99a48beec83d9bc04b2d336266044f9f53514cefb652fe6768611847196",
"zh:f593915e8a24829d322d2eaeedcb153328cf9042f0d84f66040dde1be70ede04",
"zh:fba1aff541133e2129dfda0160369635ab48503d5c44b8407ce5922ecc15d0bd",
]
}

108
clusters/vntx/terraform/main.tf
View file

@ -1,108 +0,0 @@
terraform {
required_providers {
proxmox = {
source = "Telmate/proxmox"
version = "2.9.11"
}
}
}
provider "proxmox" {
pm_api_url = var.proxmox_api_url
pm_user = var.proxmox_user
pm_password = var.proxmox_password
pm_tls_insecure = var.proxmox_tls_insecure
}
# Control plane nodes
resource "proxmox_vm_qemu" "k3s_control" {
count = var.control_plane_count
name = "vntx-control-${count.index + 1}"
target_node = var.proxmox_nodes[count.index % length(var.proxmox_nodes)]
# Use Debian 13 cloud image
clone = var.debian_template_name
cores = var.control_plane_specs.cores
memory = var.control_plane_specs.memory
boot = "order=scsi0"
agent = 1
disk {
size = var.control_plane_specs.disk_size
storage = var.proxmox_storage
type = "scsi"
format = "qcow2"
}
network {
model = "virtio"
bridge = var.network_bridge
}
# Cloud-init
os_type = "cloud-init"
ipconfig0 = "ip=${var.control_plane_ips[count.index]}/24,gw=${var.gateway}"
sshkeys = var.ssh_public_key
lifecycle {
ignore_changes = [
network,
disk,
]
}
}
# Worker nodes
resource "proxmox_vm_qemu" "k3s_worker" {
count = var.worker_count
name = "vntx-worker-${count.index + 1}"
target_node = var.proxmox_nodes[count.index % length(var.proxmox_nodes)]
# Use Debian 13 cloud image
clone = var.debian_template_name
cores = var.worker_specs.cores
memory = var.worker_specs.memory
boot = "order=scsi0"
agent = 1
disk {
size = var.worker_specs.disk_size
storage = var.proxmox_storage
type = "scsi"
format = "qcow2"
}
# Additional disk for container storage
disk {
size = var.worker_specs.storage_disk_size
storage = var.proxmox_storage
type = "scsi"
format = "qcow2"
slot = 1
}
network {
model = "virtio"
bridge = var.network_bridge
}
# Cloud-init
os_type = "cloud-init"
ipconfig0 = "ip=${var.worker_ips[count.index]}/24,gw=${var.gateway}"
sshkeys = var.ssh_public_key
lifecycle {
ignore_changes = [
network,
disk,
]
}
}

27
clusters/vntx/terraform/outputs.tf
View file

@ -1,27 +0,0 @@
output "control_plane_ips" {
value = proxmox_vm_qemu.k3s_control[*].default_ipv4_address
description = "IP addresses of control plane nodes"
}
output "worker_ips" {
value = proxmox_vm_qemu.k3s_worker[*].default_ipv4_address
description = "IP addresses of worker nodes"
}
output "cluster_config" {
value = {
control_nodes = [
for i, vm in proxmox_vm_qemu.k3s_control : {
name = vm.name
ip = var.control_plane_ips[i]
}
]
worker_nodes = [
for i, vm in proxmox_vm_qemu.k3s_worker : {
name = vm.name
ip = var.worker_ips[i]
}
]
}
description = "Cluster configuration for Ansible"
}

1
clusters/vntx/terraform/terraform.tfstate
View file

@ -1 +0,0 @@
{"version":4,"terraform_version":"1.10.5","serial":6,"lineage":"dec2afcf-a221-0f9b-d460-39a11719c258","outputs":{"cluster_config":{"value":{"control_nodes":[{"ip":"10.0.0.101","name":"vntx-control-1"},{"ip":"10.0.0.102","name":"vntx-control-2"},{"ip":"10.0.0.103","name":"vntx-control-3"}],"worker_nodes":[{"ip":"10.0.0.111","name":"vntx-worker-1"},{"ip":"10.0.0.112","name":"vntx-worker-2"}]},"type":["object",{"control_nodes":["tuple",[["object",{"ip":"string","name":"string"}],["object",{"ip":"string","name":"string"}],["object",{"ip":"string","name":"string"}]]],"worker_nodes":["tuple",[["object",{"ip":"string","name":"string"}],["object",{"ip":"string","name":"string"}]]]}]}},"resources":[],"check_results":null}

1
clusters/vntx/terraform/terraform.tfstate.backup
View file

@ -1 +0,0 @@
{"version":4,"terraform_version":"1.10.5","serial":5,"lineage":"dec2afcf-a221-0f9b-d460-39a11719c258","outputs":{"cluster_config":{"value":{"control_nodes":[{"ip":"10.0.0.101","name":"vntx-control-1"},{"ip":"10.0.0.102","name":"vntx-control-2"},{"ip":"10.0.0.103","name":"vntx-control-3"}],"worker_nodes":[{"ip":"10.0.0.111","name":"vntx-worker-1"},{"ip":"10.0.0.112","name":"vntx-worker-2"}]},"type":["object",{"control_nodes":["tuple",[["object",{"ip":"string","name":"string"}],["object",{"ip":"string","name":"string"}],["object",{"ip":"string","name":"string"}]]],"worker_nodes":["tuple",[["object",{"ip":"string","name":"string"}],["object",{"ip":"string","name":"string"}]]]}]}},"resources":[{"mode":"managed","type":"proxmox_vm_qemu","name":"k3s_worker","provider":"provider[\"registry.opentofu.org/telmate/proxmox\"]","instances":[]}],"check_results":null}

38
clusters/vntx/terraform/terraform.tfvars
View file

@ -1,38 +0,0 @@
# Proxmox connection
proxmox_api_url = "https://10.0.0.1:8006/api2/json"
proxmox_host = "10.0.0.1"
proxmox_user = "root@pam"
# proxmox_password = "set via TF_VAR_proxmox_password environment variable"
# Proxmox cluster nodes - deploying all on vm2-380
proxmox_nodes = ["vm2-380"]
# Debian template name (you'll need to create this)
debian_template_name = "debian13-cloud"
# Storage
proxmox_storage = "local-lvm" # Adjust to your storage pool name
# Network configuration
network_bridge = "vmbr0"
vlan_tag = -1 # Set to VLAN ID if needed, -1 for no VLAN
gateway = "10.0.0.1"
# Reduce node count for 2-host setup
control_plane_count = 3 # Can run on 2 hosts
worker_count = 2
# Node IPs - adjust for your network
control_plane_ips = [
"10.0.0.101",
"10.0.0.102",
"10.0.0.103"
]
worker_ips = [
"10.0.0.111",
"10.0.0.112"
]
# SSH key for node access
ssh_public_key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHLyVSEEAEbGZZqwPwEup0XrlTpu3x3iF9ExvvQPA4xN graham@mcintire.me"

31
clusters/vntx/terraform/terraform.tfvars.example
View file

@ -1,31 +0,0 @@
# Proxmox connection
proxmox_api_url = "https://pve1.vntx.net:8006/api2/json"
proxmox_host = "pve1.vntx.net"
proxmox_user = "root@pam"
# proxmox_password = "set via TF_VAR_proxmox_password"
# Proxmox cluster nodes
proxmox_nodes = ["pve1", "pve2", "pve3"]
# Storage
proxmox_storage = "local-lvm"
# Network configuration
network_bridge = "vmbr0"
vlan_tag = -1 # Set to VLAN ID if needed, -1 for no VLAN
gateway = "10.0.0.1"
# Node IPs
control_plane_ips = [
"10.0.0.101",
"10.0.0.102",
"10.0.0.103"
]
worker_ips = [
"10.0.0.111",
"10.0.0.112"
]
# SSH key for node access
ssh_public_key = "ssh-rsa AAAAB3..."

BIN
clusters/vntx/terraform/tfplan
View file

Binary file not shown.

129
clusters/vntx/terraform/variables.tf
View file

@ -1,129 +0,0 @@
variable "proxmox_api_url" {
description = "Proxmox API URL"
type = string
}
variable "proxmox_user" {
description = "Proxmox user"
type = string
default = "root@pam"
}
variable "proxmox_password" {
description = "Proxmox password"
type = string
sensitive = true
}
variable "proxmox_tls_insecure" {
description = "Allow insecure TLS for Proxmox API"
type = bool
default = true
}
variable "proxmox_host" {
description = "Proxmox host for SSH commands"
type = string
}
variable "proxmox_nodes" {
description = "List of Proxmox nodes to distribute VMs across"
type = list(string)
default = ["pve1", "pve2", "pve3"]
}
variable "proxmox_storage" {
description = "Proxmox storage pool"
type = string
default = "local-lvm"
}
variable "debian_template_name" {
description = "Name of Debian cloud-init template VM"
type = string
default = "debian13-cloud"
}
variable "network_bridge" {
description = "Network bridge"
type = string
default = "vmbr0"
}
variable "vlan_tag" {
description = "VLAN tag for cluster network"
type = number
default = -1
}
variable "gateway" {
description = "Network gateway"
type = string
default = "10.0.0.1"
}
variable "control_plane_count" {
description = "Number of control plane nodes"
type = number
default = 3
}
variable "worker_count" {
description = "Number of worker nodes"
type = number
default = 2
}
variable "control_plane_specs" {
description = "Specifications for control plane nodes"
type = object({
cores = number
memory = number
disk_size = string
})
default = {
cores = 4
memory = 8192
disk_size = "50G"
}
}
variable "worker_specs" {
description = "Specifications for worker nodes"
type = object({
cores = number
memory = number
disk_size = string
storage_disk_size = string
})
default = {
cores = 8
memory = 16384
disk_size = "50G"
storage_disk_size = "200G"
}
}
variable "control_plane_ips" {
description = "Static IPs for control plane nodes"
type = list(string)
default = [
"10.0.0.101",
"10.0.0.102",
"10.0.0.103"
]
}
variable "worker_ips" {
description = "Static IPs for worker nodes"
type = list(string)
default = [
"10.0.0.111",
"10.0.0.112"
]
}
variable "ssh_public_key" {
description = "SSH public key for node access"
type = string
}

82
home/ansible/create-k3s-vms.yml
View file

@ -1,82 +0,0 @@
---
- name: Create K3s VMs on Proxmox
hosts: lab02
gather_facts: no
vars:
template_vmid: 9000
cluster_name: "home-k3s"
network_gateway: "10.0.19.254"
ssh_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHLyVSEEAEbGZZqwPwEup0XrlTpu3x3iF9ExvvQPA4xN graham@mcintire.me"
control_nodes:
- { vmid: 1100, name: "{{ cluster_name }}-control-1", ip: "10.0.19.100/22", cores: 2, memory: 8192, disk: "32G", node: "lab02" }
- { vmid: 1101, name: "{{ cluster_name }}-control-2", ip: "10.0.19.101/22", cores: 2, memory: 8192, disk: "32G", node: "lab03" }
- { vmid: 1102, name: "{{ cluster_name }}-control-3", ip: "10.0.19.102/22", cores: 2, memory: 8192, disk: "32G", node: "lab04" }
worker_nodes:
- { vmid: 1200, name: "{{ cluster_name }}-worker-1", ip: "10.0.19.110/22", cores: 4, memory: 8192, disk: "64G", node: "lab02" }
- { vmid: 1201, name: "{{ cluster_name }}-worker-2", ip: "10.0.19.111/22", cores: 4, memory: 8192, disk: "64G", node: "lab03" }
- { vmid: 1202, name: "{{ cluster_name }}-worker-3", ip: "10.0.19.112/22", cores: 4, memory: 8192, disk: "64G", node: "lab04" }
all_nodes: "{{ control_nodes + worker_nodes }}"
tasks:
- name: Check which VMs already exist
command: qm status {{ item.vmid }}
loop: "{{ all_nodes }}"
register: vm_checks
failed_when: false
changed_when: false
- name: Create VMs that don't exist
command: |
qm clone {{ template_vmid }} {{ item.item.vmid }} \
--name {{ item.item.name }} \
--full true
loop: "{{ vm_checks.results }}"
when: item.rc != 0
loop_control:
label: "{{ item.item.name }}"
- name: Configure VMs
command: |
qm set {{ item.vmid }} \
--cores {{ item.cores }} \
--memory {{ item.memory }} \
--ipconfig0 ip={{ item.ip }},gw={{ network_gateway }} \
--sshkeys /root/.ssh/authorized_keys \
--onboot 1
loop: "{{ all_nodes }}"
loop_control:
label: "{{ item.name }}"
- name: Resize disks
command: qm resize {{ item.vmid }} scsi0 {{ item.disk }}
loop: "{{ all_nodes }}"
loop_control:
label: "{{ item.name }}"
register: resize_result
failed_when:
- resize_result.rc != 0
- "'shrinking disks is not supported' not in resize_result.stderr"
- name: Start VMs
command: qm start {{ item.vmid }}
loop: "{{ all_nodes }}"
loop_control:
label: "{{ item.name }}"
register: start_result
failed_when:
- start_result.rc != 0
- "'already running' not in start_result.stderr"
- name: Migrate VMs to their target nodes
command: qm migrate {{ item.vmid }} {{ item.node }} --online
loop: "{{ all_nodes }}"
when: item.node != 'lab02'
loop_control:
label: "{{ item.name }} -> {{ item.node }}"
- name: VM creation complete
debug:
msg: "All K3s VMs have been created and distributed across the cluster"

416
home/ansible/deploy-k3s.yml
View file

@ -1,416 +0,0 @@
---
- name: Prepare K3s nodes
hosts: k3s_cluster
become: true
tasks:
- name: Update apt cache
apt:
update_cache: yes
cache_valid_time: 3600
when: ansible_os_family == "Debian"
- name: Install curl
package:
name: curl
state: present
- name: Configure DNS
copy:
content: |
nameserver 9.9.9.9
nameserver 149.112.112.112
nameserver 1.1.1.1
dest: /etc/resolv.conf
backup: yes
- name: Test DNS resolution
command: nslookup github.com
changed_when: false
register: dns_test
failed_when: false
- name: Show DNS test result
debug:
var: dns_test.stdout_lines
when: dns_test.rc != 0
- name: Load required kernel modules
modprobe:
name: "{{ item }}"
state: present
loop:
- br_netfilter
- overlay
- wireguard
ignore_errors: yes # Wireguard module may not exist in Debian 13
- name: Persist kernel modules
copy:
content: |
br_netfilter
overlay
wireguard
dest: /etc/modules-load.d/k3s.conf
- name: Set sysctl parameters
sysctl:
name: "{{ item.name }}"
value: "{{ item.value }}"
state: present
sysctl_file: /etc/sysctl.d/k3s.conf
reload: yes
loop:
- { name: 'net.bridge.bridge-nf-call-iptables', value: '1' }
- { name: 'net.bridge.bridge-nf-call-ip6tables', value: '1' }
- { name: 'net.ipv4.ip_forward', value: '1' }
- { name: 'net.ipv6.conf.all.forwarding', value: '1' }
- name: Download k3s binary
hosts: k3s_cluster
become: true
tasks:
- name: Download k3s binary
get_url:
url: https://github.com/k3s-io/k3s/releases/download/{{ k3s_version }}/k3s
dest: /usr/local/bin/k3s
mode: '0755'
- name: Deploy first control plane node
hosts: home-k3s-control-1
become: true
tasks:
- name: Create k3s config directory
file:
path: /etc/rancher/k3s
state: directory
mode: '0755'
- name: Create k3s server config
copy:
content: |
cluster-init: true
disable:
- servicelb
- traefik
cluster-cidr: 10.42.0.0/16
service-cidr: 10.43.0.0/16
cluster-dns: 10.43.0.10
flannel-backend: vxlan
write-kubeconfig-mode: "0644"
tls-san:
- "{{ ansible_host }}"
- "{{ ansible_hostname }}"
dest: /etc/rancher/k3s/config.yaml
- name: Create k3s systemd service
copy:
content: |
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target
[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
KillMode=process
Delegate=yes
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server
[Install]
WantedBy=multi-user.target
dest: /etc/systemd/system/k3s.service
- name: Start k3s on first control plane
systemd:
name: k3s
daemon_reload: yes
enabled: yes
state: started
- name: Wait for node token
wait_for:
path: /var/lib/rancher/k3s/server/node-token
timeout: 120
- name: Read node token
slurp:
src: /var/lib/rancher/k3s/server/node-token
register: node_token
- name: Set token fact
set_fact:
k3s_token: "{{ node_token.content | b64decode | trim }}"
- name: Wait for k3s to be ready
command: /usr/local/bin/k3s kubectl get nodes
register: nodes_result
until: nodes_result.rc == 0
retries: 30
delay: 10
changed_when: false
environment:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
- name: Check k3s service status
command: systemctl status k3s
register: k3s_status
changed_when: false
failed_when: false
- name: Show k3s logs if not ready
command: journalctl -u k3s -n 50 --no-pager
register: k3s_logs
when: nodes_result.rc != 0
changed_when: false
- name: Display logs
debug:
var: k3s_logs.stdout_lines
when: k3s_logs is defined
- name: Deploy additional control plane nodes
hosts: k3s_control:!home-k3s-control-1
become: true
serial: 1
tasks:
- name: Create k3s config directory
file:
path: /etc/rancher/k3s
state: directory
mode: '0755'
- name: Create k3s server config for additional control planes
copy:
content: |
server: https://{{ hostvars['home-k3s-control-1']['ansible_host'] }}:6443
token: {{ hostvars['home-k3s-control-1']['k3s_token'] }}
disable:
- servicelb
- traefik
cluster-cidr: 10.42.0.0/16
service-cidr: 10.43.0.0/16
cluster-dns: 10.43.0.10
flannel-backend: vxlan
write-kubeconfig-mode: "0644"
tls-san:
- "{{ ansible_host }}"
- "{{ ansible_hostname }}"
dest: /etc/rancher/k3s/config.yaml
- name: Create k3s systemd service
copy:
content: |
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target
[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
KillMode=process
Delegate=yes
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server
[Install]
WantedBy=multi-user.target
dest: /etc/systemd/system/k3s.service
- name: Start k3s on additional control planes
systemd:
name: k3s
daemon_reload: yes
enabled: yes
state: started
- name: Deploy worker nodes
hosts: k3s_workers
become: true
tasks:
- name: Create k3s config directory
file:
path: /etc/rancher/k3s
state: directory
mode: '0755'
- name: Create k3s agent config
copy:
content: |
server: https://{{ hostvars['home-k3s-control-1']['ansible_host'] }}:6443
token: {{ hostvars['home-k3s-control-1']['k3s_token'] }}
dest: /etc/rancher/k3s/config.yaml
- name: Create k3s-agent systemd service
copy:
content: |
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target
[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
KillMode=process
Delegate=yes
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s agent
[Install]
WantedBy=multi-user.target
dest: /etc/systemd/system/k3s-agent.service
- name: Start k3s-agent on workers
systemd:
name: k3s-agent
daemon_reload: yes
enabled: yes
state: started
- name: Configure kubectl access
hosts: home-k3s-control-1
become: true
tasks:
- name: Create kubectl symlink
file:
src: /usr/local/bin/k3s
dest: /usr/local/bin/kubectl
state: link
- name: Create crictl symlink
file:
src: /usr/local/bin/k3s
dest: /usr/local/bin/crictl
state: link
- name: Create .kube directory for user
file:
path: "/home/{{ ansible_user }}/.kube"
state: directory
owner: "{{ ansible_user }}"
group: "{{ ansible_user }}"
mode: '0755'
- name: Copy kubeconfig to user
copy:
src: /etc/rancher/k3s/k3s.yaml
dest: "/home/{{ ansible_user }}/.kube/config"
owner: "{{ ansible_user }}"
group: "{{ ansible_user }}"
mode: '0600'
remote_src: yes
- name: Fetch kubeconfig
fetch:
src: /etc/rancher/k3s/k3s.yaml
dest: "{{ playbook_dir }}/kubeconfig"
flat: yes
- name: Update kubeconfig server URL
replace:
path: "{{ playbook_dir }}/kubeconfig"
regexp: 'server: https://127.0.0.1:6443'
replace: 'server: https://{{ ansible_host }}:6443'
delegate_to: localhost
become: false
- name: Apply MetalLB configuration
hosts: home-k3s-control-1
become: true
tasks:
- name: Wait for MetalLB namespace
command: /usr/local/bin/k3s kubectl get namespace metallb-system
register: metallb_ns
until: metallb_ns.rc == 0
retries: 60
delay: 5
changed_when: false
environment:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
- name: Wait for MetalLB controller deployment
command: /usr/local/bin/k3s kubectl -n metallb-system wait --for=condition=available --timeout=300s deployment/controller
changed_when: false
environment:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
- name: Create MetalLB IP pool manifest
copy:
content: |
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: home-pool
namespace: metallb-system
spec:
addresses:
- 10.0.19.1-10.0.19.99
dest: /tmp/metallb-pool.yaml
- name: Apply MetalLB IP pool
command: /usr/local/bin/k3s kubectl apply -f /tmp/metallb-pool.yaml
environment:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
- name: Create MetalLB L2 advertisement manifest
copy:
content: |
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: home-l2
namespace: metallb-system
spec:
ipAddressPools:
- home-pool
dest: /tmp/metallb-l2.yaml
- name: Apply MetalLB L2 advertisement
command: /usr/local/bin/k3s kubectl apply -f /tmp/metallb-l2.yaml
environment:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
- name: Clean up manifests
file:
path: "{{ item }}"
state: absent
loop:
- /tmp/metallb-pool.yaml
- /tmp/metallb-l2.yaml

35
home/ansible/fix-k3s-dns.yml
View file

@ -1,35 +0,0 @@
---
- name: Fix DNS on K3s Nodes
hosts: k3s_cluster
become: true
tasks:
- name: Update resolv.conf to use Quad9 DNS
copy:
content: |
nameserver 9.9.9.9
nameserver 149.112.112.112
dest: /etc/resolv.conf
backup: yes
- name: Ensure systemd-resolved is not managing resolv.conf
systemd:
name: systemd-resolved
state: stopped
enabled: no
ignore_errors: yes
- name: Test DNS resolution
command: nslookup registry-1.docker.io
register: dns_test
changed_when: false
- name: Show DNS test results
debug:
msg: "{{ dns_test.stdout_lines }}"
- name: Restart containerd to pick up DNS changes
systemd:
name: containerd
state: restarted
when: inventory_hostname in groups['k3s_cluster']

143
home/ansible/fix-k3s-workers.yml
View file

@ -1,143 +0,0 @@
---
- name: Fix K3s worker nodes
hosts: k3s_workers
become: true
tasks:
- name: Stop k3s-agent service
systemd:
name: k3s-agent
state: stopped
enabled: no
ignore_errors: yes
- name: Remove k3s node password
file:
path: /etc/rancher/node/password
state: absent
- name: Remove k3s data directory
file:
path: /var/lib/rancher/k3s
state: absent
- name: Remove k3s config
file:
path: /etc/rancher/k3s
state: absent
- name: Ensure k3s binary exists
stat:
path: /usr/local/bin/k3s
register: k3s_binary
- name: Download k3s binary if missing
get_url:
url: https://github.com/k3s-io/k3s/releases/download/{{ k3s_version | default('v1.28.5+k3s1') }}/k3s
dest: /usr/local/bin/k3s
mode: '0755'
when: not k3s_binary.stat.exists
- name: Create k3s config directory
file:
path: /etc/rancher/k3s
state: directory
mode: '0755'
- name: Get token from control node
delegate_to: home-k3s-control-1
slurp:
src: /var/lib/rancher/k3s/server/node-token
register: node_token
run_once: true
- name: Create k3s agent config
copy:
content: |
server: https://{{ hostvars['home-k3s-control-1']['ansible_host'] }}:6443
token: {{ node_token.content | b64decode | trim }}
with-node-id: true
dest: /etc/rancher/k3s/config.yaml
- name: Ensure k3s-agent service exists
copy:
content: |
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target
[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
KillMode=process
Delegate=yes
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s agent
[Install]
WantedBy=multi-user.target
dest: /etc/systemd/system/k3s-agent.service
- name: Start k3s-agent service
systemd:
name: k3s-agent
daemon_reload: yes
enabled: yes
state: started
- name: Remove stale nodes from cluster
hosts: home-k3s-control-1
become: true
tasks:
- name: Get list of nodes
command: /usr/local/bin/k3s kubectl get nodes -o json
register: nodes_json
environment:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
- name: Parse node list
set_fact:
nodes: "{{ nodes_json.stdout | from_json }}"
- name: Remove NotReady worker nodes
command: /usr/local/bin/k3s kubectl delete node {{ item.metadata.name }}
environment:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
loop: "{{ nodes['items'] }}"
when:
- item.metadata.name is match("home-k3s-worker-.*")
- item.status.conditions | selectattr('type', 'equalto', 'Ready') | selectattr('status', 'equalto', 'Unknown') | list | length > 0
- name: Verify worker nodes
hosts: home-k3s-control-1
become: true
tasks:
- name: Wait for worker nodes to join
shell: /usr/local/bin/k3s kubectl get nodes | grep -E "home-k3s-worker-[0-9]+.*Ready" | wc -l
register: ready_workers
until: ready_workers.stdout | int >= 3
retries: 30
delay: 10
environment:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
- name: Display cluster status
command: /usr/local/bin/k3s kubectl get nodes
register: final_status
environment:
KUBECONFIG: /etc/rancher/k3s/k3s.yaml
- name: Show cluster status
debug:
var: final_status.stdout_lines

4
home/ansible/kubeconfig
View file

@ -1,8 +1,8 @@
apiVersion: v1 apiVersion: v1
clusters: clusters:
- cluster: - cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUzTlRjd09EVTRORGd3SGhjTk1qVXdPVEExTVRVeU5EQTRXaGNOTXpVd09UQXpNVFV5TkRBNApXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUzTlRjd09EVTRORGd3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFUQm5YbGtOQlhZN0hrRTRyczlycEtCL0NhTnh1ekZUSUpTUjdpdWQyMEUKb2ZSNWZYVlFrTWp5R0tBZTBQaWFWT1ZFZXo0OEdVeUl1T21pMTZIZnp3czdvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVVF0d2RJaDlkWVlpTkZzcDBvRmZBCk5lYXNMem93Q2dZSUtvWkl6ajBFQXdJRFJ3QXdSQUlnVk5acVBQMDBOR09xYnF3SFkxdVM4YjRLeG9MWkpXekUKbmNmNUJHazNqQVlDSUNNMlZwd3E5VmZtTTRFMTZ0VEFqeStHOGprWW1PMEFCdHdST1NnM293cDgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= insecure-skip-tls-verify: true
server: https://10.0.19.100:6443 server: https://204.110.191.2:6443
name: default name: default
contexts: contexts:
- context: - context:

35
home/ansible/mount-cephfs-k8s.yml
View file

@ -1,35 +0,0 @@
---
- name: Configure CephFS access on K3s nodes
hosts: k3s_cluster
become: true
tasks:
- name: Install ceph-fuse package for CephFS
package:
name:
- ceph-fuse
- ceph-common
state: present
ignore_errors: yes
register: ceph_install
- name: Create mount point for testing
file:
path: /mnt/cephfs-test
state: directory
mode: '0755'
- name: Test CephFS mount
shell: |
timeout 5 ceph-fuse -n client.kubernetes --keyring=/etc/ceph/ceph.client.kubernetes.keyring -m 10.0.16.231:6789,10.0.16.232:6789,10.0.16.233:6789 /mnt/cephfs-test
register: mount_test
failed_when: false
changed_when: false
- name: Unmount test
shell: fusermount -u /mnt/cephfs-test
failed_when: false
changed_when: false
- name: Show mount test result
debug:
msg: "CephFS mount test: {{ 'SUCCESS' if mount_test.rc == 124 else 'FAILED' }}"

31
home/ansible/restart-k3s-services.yml
View file

@ -1,31 +0,0 @@
---
- name: Restart K3s Services After DNS Fix
hosts: k3s_cluster
become: true
tasks:
- name: Test DNS resolution using getent
command: getent hosts registry-1.docker.io
register: dns_test
changed_when: false
ignore_errors: yes
- name: Show DNS test results
debug:
msg: "{{ dns_test.stdout | default('DNS resolution failed') }}"
- name: Restart K3s service on control nodes
systemd:
name: k3s
state: restarted
when: inventory_hostname in groups['k3s_control_plane']
- name: Restart K3s agent on worker nodes
systemd:
name: k3s-agent
state: restarted
when: inventory_hostname in groups['k3s_workers']
- name: Wait for services to stabilize
pause:
seconds: 30

82
home/ansible/setup-ceph-clients-k3s.yml
View file

@ -1,82 +0,0 @@
---
- name: Setup Ceph client tools on K3s nodes
hosts: k3s_cluster
become: true
tasks:
- name: Install Ceph repository key
apt_key:
url: https://download.ceph.com/keys/release.asc
state: present
- name: Get Debian codename
command: lsb_release -cs
register: debian_codename
changed_when: false
- name: Add Ceph repository
apt_repository:
repo: "deb https://download.ceph.com/debian-reef/ {{ debian_codename.stdout }} main"
state: present
filename: ceph
- name: Update apt cache
apt:
update_cache: yes
- name: Install Ceph client packages
apt:
name:
- ceph-common
- python3-rbd
- python3-rados
state: present
- name: Create Ceph configuration directory
file:
path: /etc/ceph
state: directory
mode: '0755'
- name: Copy Ceph configuration
copy:
content: |
[global]
fsid = 6e5e8932-3613-471d-afa0-410fade4d4ff
mon_initial_members = lab02,lab03,lab04
mon_host = 10.0.16.231,10.0.16.232,10.0.16.233
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
dest: /etc/ceph/ceph.conf
mode: '0644'
- name: Copy Ceph kubernetes keyring
copy:
content: |
[client.kubernetes]
key = AQAq14Jo/g4ZORAAwxpPOmqj1PYgmih6s6b4vA==
dest: /etc/ceph/ceph.client.kubernetes.keyring
mode: '0600'
- name: Test Ceph connectivity
command: ceph -s --id kubernetes
register: ceph_status
changed_when: false
ignore_errors: yes
- name: Display Ceph status
debug:
var: ceph_status.stdout_lines
when: ceph_status.rc == 0
- name: Check RBD module
modprobe:
name: rbd
state: present
- name: Ensure RBD module loads on boot
lineinfile:
path: /etc/modules
line: rbd
create: yes

82
home/cluster-380/README.md
View file

@ -1,82 +0,0 @@
# Cluster-380 (w5isp.w5isp.com)
Single-node K3s cluster with Traefik ingress and automatic SSL.
## Architecture
- **Server**: w5isp.w5isp.com (204.110.191.200)
- **OS**: Debian 13
- **K3s**: Single-node cluster with Traefik enabled
- **Ingress**: Traefik with Let's Encrypt automatic SSL
- **Storage**: Longhorn distributed storage
## Setup
1. **Deploy the server** and install K3s using Ansible:
```bash
cd ansible
ansible-playbook -l w5isp.w5isp.com general.yml
```
2. **Deploy infrastructure components**:
```bash
cd home/cluster-380/infrastructure
./deploy-with-traefik.sh
```
3. **Deploy applications**:
```bash
kubectl apply -f ../apps/forgejo-traefik.yaml
```
## Services
### Traefik (Ingress Controller)
- Handles all HTTP/HTTPS traffic on port 80/443
- Automatic SSL certificates via Let's Encrypt
- Can handle TCP routing (e.g., SSH for Forgejo on port 2222)
### Example Application: Forgejo
- URL: https://git.w5isp.com
- SSH: git.w5isp.com:2222
- Automatic SSL certificate
## Adding New Services
Create an IngressRoute for automatic SSL:
```yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: myapp
namespace: myapp
spec:
entryPoints:
- websecure
routes:
- match: Host(`myapp.w5isp.com`)
kind: Rule
services:
- name: myapp
port: 80
tls:
certResolver: letsencrypt
```
For TCP services:
```yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: myapp-tcp
namespace: myapp
spec:
entryPoints:
- myapp-tcp # Define in traefik-config.yaml
routes:
- match: HostSNI(`*`)
services:
- name: myapp-tcp
port: 1234
```

148
home/cluster-380/apps/forgejo-traefik.yaml
View file

@ -1,148 +0,0 @@
# Forgejo deployment with Traefik ingress
---
# Namespace
apiVersion: v1
kind: Namespace
metadata:
name: forgejo
---
# PVC for Forgejo data using Longhorn
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: forgejo-data
namespace: forgejo
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 50Gi
---
# Main Forgejo Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: forgejo
namespace: forgejo
spec:
replicas: 1
selector:
matchLabels:
app: forgejo
template:
metadata:
labels:
app: forgejo
spec:
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
containers:
- name: forgejo
image: codeberg.org/forgejo/forgejo:1.21-rootless
ports:
- containerPort: 3000
name: http
- containerPort: 2222
name: ssh
env:
- name: USER_UID
value: "1000"
- name: USER_GID
value: "1000"
- name: FORGEJO__database__DB_TYPE
value: sqlite3
- name: FORGEJO__server__ROOT_URL
value: https://git.w5isp.com/
- name: FORGEJO__server__DOMAIN
value: git.w5isp.com
- name: FORGEJO__server__SSH_DOMAIN
value: git.w5isp.com
- name: FORGEJO__server__SSH_PORT
value: "2222"
- name: FORGEJO__DEFAULT__APP_NAME
value: "W5ISP Git"
- name: FORGEJO__server__OFFLINE_MODE
value: "false"
volumeMounts:
- name: data
mountPath: /data
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "1Gi"
cpu: "1000m"
volumes:
- name: data
persistentVolumeClaim:
claimName: forgejo-data
---
# Service for HTTP/HTTPS
apiVersion: v1
kind: Service
metadata:
name: forgejo-http
namespace: forgejo
spec:
type: ClusterIP
selector:
app: forgejo
ports:
- name: http
port: 80
targetPort: 3000
protocol: TCP
---
# Service for SSH (direct TCP routing)
apiVersion: v1
kind: Service
metadata:
name: forgejo-ssh
namespace: forgejo
spec:
type: ClusterIP
selector:
app: forgejo
ports:
- name: ssh
port: 2222
targetPort: 2222
protocol: TCP
---
# IngressRoute for HTTPS with automatic SSL
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: forgejo
namespace: forgejo
spec:
entryPoints:
- websecure
routes:
- match: Host(`git.w5isp.com`)
kind: Rule
services:
- name: forgejo-http
port: 80
tls:
certResolver: letsencrypt
---
# IngressRouteTCP for SSH
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: forgejo-ssh
namespace: forgejo
spec:
entryPoints:
- ssh-forgejo
routes:
- match: HostSNI(`*`)
services:
- name: forgejo-ssh
port: 2222

127
home/cluster-380/apps/forgejo.yaml
View file

@ -1,127 +0,0 @@
# Complete Forgejo deployment for cluster-380
---
# Namespace
apiVersion: v1
kind: Namespace
metadata:
name: forgejo
---
# PVC for Forgejo data using Longhorn
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: forgejo-data
namespace: forgejo
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 50Gi
---
# Main Forgejo Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: forgejo
namespace: forgejo
spec:
replicas: 1
selector:
matchLabels:
app: forgejo
template:
metadata:
labels:
app: forgejo
spec:
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
containers:
- name: forgejo
image: codeberg.org/forgejo/forgejo:1.21-rootless
ports:
- containerPort: 3000
name: http
- containerPort: 2222
name: ssh
env:
- name: USER_UID
value: "1000"
- name: USER_GID
value: "1000"
- name: FORGEJO__database__DB_TYPE
value: sqlite3
- name: FORGEJO__server__ROOT_URL
value: https://git.w5isp.com/
- name: FORGEJO__server__DOMAIN
value: git.w5isp.com
- name: FORGEJO__server__SSH_DOMAIN
value: git.w5isp.com
- name: FORGEJO__DEFAULT__APP_NAME
value: "W5ISP Git"
- name: FORGEJO__server__OFFLINE_MODE
value: "false"
volumeMounts:
- name: data
mountPath: /data
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "1Gi"
cpu: "1000m"
volumes:
- name: data
persistentVolumeClaim:
claimName: forgejo-data
---
# Service for LoadBalancer
apiVersion: v1
kind: Service
metadata:
name: forgejo
namespace: forgejo
spec:
type: LoadBalancer
selector:
app: forgejo
ports:
- name: http
port: 80
targetPort: 3000
protocol: TCP
- name: ssh
port: 22
targetPort: 2222
protocol: TCP
---
# Ingress for HTTPS access
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: forgejo
namespace: forgejo
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
ingressClassName: nginx
tls:
- hosts:
- git.w5isp.com
secretName: forgejo-tls
rules:
- host: git.w5isp.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: forgejo
port:
number: 80

24
home/cluster-380/infrastructure/deploy-with-traefik.sh
View file

@ -1,24 +0,0 @@
#!/bin/bash
# Deploy script for cluster-380 with Traefik ingress
set -e
echo "Deploying system namespaces..."
kubectl apply -f ../system/namespaces.yaml
echo "Configuring Traefik..."
kubectl apply -f ../system/traefik-config.yaml
echo "Waiting for Traefik to be ready..."
kubectl -n kube-system wait --for=condition=ready pod -l app.kubernetes.io/name=traefik --timeout=90s
echo "Installing Longhorn v1.5.3..."
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.3/deploy/longhorn.yaml
echo "Waiting for Longhorn to be ready..."
kubectl -n longhorn-system wait --for=condition=ready pod -l app=longhorn-manager --timeout=180s
echo "Infrastructure components deployed successfully!"
echo ""
echo "You can now deploy applications using Traefik IngressRoutes for automatic SSL"
echo "Example: kubectl apply -f ../apps/forgejo-traefik.yaml"

27
home/cluster-380/infrastructure/deploy.sh
View file

@ -1,27 +0,0 @@
#!/bin/bash
# Deploy script for cluster-380 infrastructure components
set -e
echo "Deploying system namespaces..."
kubectl apply -f ../system/namespaces.yaml
echo "Installing MetalLB v0.13.12..."
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.12/config/manifests/metallb-native.yaml
echo "Waiting for MetalLB to be ready..."
kubectl -n metallb-system wait --for=condition=ready pod -l app=metallb --timeout=90s
echo "Applying MetalLB configuration..."
kubectl apply -f ../system/metallb-config.yaml
echo "Installing Longhorn v1.5.3..."
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.3/deploy/longhorn.yaml
echo "Installing cert-manager v1.13.3..."
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.3/cert-manager.yaml
echo "Waiting for cert-manager to be ready..."
kubectl -n cert-manager wait --for=condition=ready pod -l app.kubernetes.io/instance=cert-manager --timeout=90s
echo "Infrastructure components deployed successfully!"

18
home/cluster-380/system/metallb-config.yaml
View file

@ -1,18 +0,0 @@
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: default-pool
namespace: metallb-system
spec:
addresses:
- 204.110.191.200/32 # Using the single server IP for LoadBalancer services
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: default-advertisement
namespace: metallb-system
spec:
ipAddressPools:
- default-pool

33
home/cluster-380/system/namespaces.yaml
View file

@ -1,33 +0,0 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: metallb-system
labels:
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/warn: privileged
pod-security.kubernetes.io/audit: privileged
---
apiVersion: v1
kind: Namespace
metadata:
name: longhorn-system
labels:
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/warn: privileged
pod-security.kubernetes.io/audit: privileged
---
apiVersion: v1
kind: Namespace
metadata:
name: cert-manager
---
apiVersion: v1
kind: Namespace
metadata:
name: ingress-system
---
apiVersion: v1
kind: Namespace
metadata:
name: monitoring

95
home/cluster-380/system/traefik-config.yaml
View file

@ -1,95 +0,0 @@
---
# Traefik configuration for automatic SSL
apiVersion: v1
kind: ConfigMap
metadata:
name: traefik-config
namespace: kube-system
data:
traefik.yaml: |
global:
checkNewVersion: false
sendAnonymousUsage: false
api:
dashboard: true
debug: false
entryPoints:
web:
address: ":80"
http:
redirections:
entrypoint:
to: websecure
scheme: https
websecure:
address: ":443"
ssh-forgejo:
address: ":2222/tcp"
providers:
kubernetesIngress:
allowCrossNamespace: true
publishedService:
enabled: true
name: "kube-system/traefik"
kubernetesCRD:
allowCrossNamespace: true
certificatesResolvers:
letsencrypt:
acme:
email: postmaster@w5isp.com
storage: /data/acme.json
httpChallenge:
entryPoint: web
log:
level: INFO
---
# HelmChartConfig to apply Traefik configuration
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
deployment:
kind: DaemonSet
service:
type: LoadBalancer
spec:
loadBalancerIP: 204.110.191.200
ports:
web:
port: 80
expose: true
exposedPort: 80
protocol: TCP
websecure:
port: 443
expose: true
exposedPort: 443
protocol: TCP
ssh-forgejo:
port: 2222
expose: true
exposedPort: 2222
protocol: TCP
persistence:
enabled: true
storageClass: longhorn
size: 1Gi
additionalArguments:
- "--configfile=/config/traefik.yaml"
volumes:
- name: traefik-config
mountPath: /config
type: configMap

57
home/cluster/aprs/README.md Normal file
View file

@ -0,0 +1,57 @@
# APRS.me Deployment
APRS.me amateur radio tracking application deployed on k3s cluster at 204.110.191.2.
## Architecture
- **Application**: 2-replica StatefulSet with Erlang clustering
- **Database**: PostgreSQL 17 with PostGIS extension (100GB Longhorn storage)
- **Cache**: Redis for job queuing and caching
- **Ingress**: Traefik with cert-manager for SSL (Let's Encrypt)
- **Access**: https://aprs.me (DNS points to 204.110.191.3)
## Prerequisites
1. GitHub container registry access for `ghcr.io/aprsme/aprs.me:latest`
- Create a GitHub personal access token with `read:packages` scope
- Run: `./create-image-pull-secret.sh <username> <token>`
## Deployment
```bash
# Deploy everything
./deploy.sh
# Check status
kubectl get pods -n aprs
kubectl get certificate -n aprs
kubectl get ingressroute -n aprs
```
## Files
- `namespace.yaml` - APRS namespace
- `postgres-secret-secure.yaml` - PostgreSQL credentials (secured)
- `postgres-deployment.yaml` - PostgreSQL 17 with PostGIS StatefulSet
- `redis.yaml` - Redis cache deployment
- `aprs-secret-secure.yaml` - Application secrets (secured)
- `aprs-configmap.yaml` - APRS configuration (callsign, server, etc.)
- `aprs-rbac.yaml` - Service account and permissions for clustering
- `aprs-statefulset-with-pullsecret.yaml` - APRS application StatefulSet
- `aprs-ingressroute.yaml` - Traefik ingress with SSL certificate
- `create-image-pull-secret.sh` - Helper to create GitHub image pull secret
- `deploy.sh` - Deployment script
## Configuration
- **APRS Callsign**: W5ISP-2
- **APRS Server**: dallas.aprs2.net:10152
- **Filter**: r/33/-96/1000000000000
- **Passcode**: 15748
- **LoadBalancer IP**: 10.0.101.32 (internal)
- **Public Access**: https://aprs.me via Traefik at 204.110.191.3
## Secrets
All passwords have been generated securely and stored in `*-secure.yaml` files.
Do not commit these files to version control.

18
home/cluster/aprs/aprs-configmap.yaml Normal file
View file

@ -0,0 +1,18 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: aprs-config
namespace: aprs
data:
CALLSIGN: "W5ISP-2"
APRS_SERVER: "dallas.aprs2.net"
APRS_PORT: "10152"
APRS_FILTER: "r/33/-96/1000000000000"
APRS_PASSCODE: "15748"
PHX_SERVER: "true"
MIX_ENV: "prod"
PORT: "4000"
POOL_SIZE: "10"
RELEASE_DISTRIBUTION: "name"
RELEASE_NODE: "aprs"

58
home/cluster/aprs/aprs-ingressroute.yaml Normal file
View file

@ -0,0 +1,58 @@
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: aprs-me-cert
namespace: aprs
spec:
secretName: aprs-me-tls
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
dnsNames:
- aprs.me
- www.aprs.me
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: aprs-https
namespace: aprs
spec:
entryPoints:
- websecure
routes:
- match: Host(`aprs.me`) || Host(`www.aprs.me`)
kind: Rule
services:
- name: aprs
port: 80
tls:
secretName: aprs-me-tls
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: aprs-http
namespace: aprs
spec:
entryPoints:
- web
routes:
- match: Host(`aprs.me`) || Host(`www.aprs.me`)
kind: Rule
services:
- name: aprs
port: 80
middlewares:
- name: redirect-to-https
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: redirect-to-https
namespace: aprs
spec:
redirectScheme:
scheme: https
permanent: true

11
home/cluster/aprs/aprs-pdb.yaml
View file

@ -1,11 +0,0 @@
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: aprs-pdb
namespace: aprs
spec:
minAvailable: 1
selector:
matchLabels:
app: aprs
unhealthyPodEvictionPolicy: AlwaysAllow

9
home/cluster/aprs/aprs-rbac.yaml
View file

@ -1,7 +1,8 @@
---
apiVersion: v1 apiVersion: v1
kind: ServiceAccount kind: ServiceAccount
metadata: metadata:
name: aprs-service-account name: aprs
namespace: aprs namespace: aprs
--- ---
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
@ -12,12 +13,12 @@ metadata:
rules: rules:
- apiGroups: [""] - apiGroups: [""]
resources: ["endpoints", "pods"] resources: ["endpoints", "pods"]
verbs: ["list", "get"] verbs: ["get", "list"]
--- ---
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding kind: RoleBinding
metadata: metadata:
name: aprs-endpoints-reader-binding name: aprs-endpoints-reader
namespace: aprs namespace: aprs
roleRef: roleRef:
apiGroup: rbac.authorization.k8s.io apiGroup: rbac.authorization.k8s.io
@ -25,5 +26,5 @@ roleRef:
name: aprs-endpoints-reader name: aprs-endpoints-reader
subjects: subjects:
- kind: ServiceAccount - kind: ServiceAccount
name: aprs-service-account name: aprs
namespace: aprs namespace: aprs

13
home/cluster/aprs/aprs-secret-secure.yaml Normal file
View file

@ -0,0 +1,13 @@
---
apiVersion: v1
kind: Secret
metadata:
name: aprs-secret
namespace: aprs
type: Opaque
stringData:
DATABASE_URL: "postgres://aprs:17fZZOsYeeBCf0GN8olKaKrYCsUca25V@postgres-aprs:5432/aprsdb"
REDIS_URL: "redis://redis:6379"
SECRET_KEY_BASE: "sz31WoKApm6ud9Aik5DPRnKDQ7fd7w7GaisbPY6spkybhfU0IiuwuQbp5hxrpQzWiZnAn6045RQlhXENgrOQ"
ERLANG_COOKIE: "3LOiBOvsjdCdJGFd2m2jmLYjh44"
PHX_HOST: "aprs.me"

14
home/cluster/aprs/aprs-secret.yaml
View file

@ -1,14 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: aprs-secret
namespace: aprs
type: Opaque
stringData:
database-password: "xK9mP2vQ7sR3nL5w"
# Direct connection to PostgreSQL VM
database-url: "ecto://aprs:xK9mP2vQ7sR3nL5w@10.0.19.5:5432/aprs"
# Connection via PgBouncer on PostgreSQL VM (if available)
database-url-pgbouncer: "ecto://aprs:xK9mP2vQ7sR3nL5w@10.0.19.5:6432/aprs"
erlang-cookie: "aprs-cluster-cookie-ksdf8934jfklsdjf983jfklsdf"
secret-key-base: "GJxzOUW3YXBAF7xB4mH5VcXY6BN1NYLnoJYadlZyiCxcRUGBw65w4Sco7+sNiW2t"

17
home/cluster/aprs/aprs-service.yaml
View file

@ -1,17 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: aprs
namespace: aprs
annotations:
metallb.universe.tf/address-pool: default-pool
spec:
selector:
app: aprs
ports:
- port: 80
targetPort: 4000
name: http
type: LoadBalancer
loadBalancerIP: 10.0.19.221
sessionAffinity: None

96
home/cluster/aprs/aprs-statefulset-with-pullsecret.yaml Normal file
View file

@ -0,0 +1,96 @@
---
apiVersion: v1
kind: Service
metadata:
name: aprs-headless
namespace: aprs
spec:
clusterIP: None
selector:
app: aprs
ports:
- name: epmd
port: 4369
targetPort: 4369
---
apiVersion: v1
kind: Service
metadata:
name: aprs
namespace: aprs
spec:
type: LoadBalancer
selector:
app: aprs
ports:
- name: http
port: 80
targetPort: 4000
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: aprs
namespace: aprs
spec:
serviceName: aprs-headless
replicas: 2
selector:
matchLabels:
app: aprs
template:
metadata:
labels:
app: aprs
spec:
serviceAccountName: aprs
imagePullSecrets:
- name: ghcr-secret
initContainers:
- name: migrate
image: ghcr.io/aprsme/aprs.me:latest
command: ["/app/bin/migrate"]
envFrom:
- configMapRef:
name: aprs-config
- secretRef:
name: aprs-secret
containers:
- name: aprs
image: ghcr.io/aprsme/aprs.me:latest
ports:
- containerPort: 4000
name: http
- containerPort: 4369
name: epmd
envFrom:
- configMapRef:
name: aprs-config
- secretRef:
name: aprs-secret
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: RELEASE_NAME
value: "aprs@$(POD_IP)"
livenessProbe:
httpGet:
path: /
port: 4000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: 4000
initialDelaySeconds: 10
periodSeconds: 5
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"

200
home/cluster/aprs/aprs-statefulset.yaml
View file

@ -1,200 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: aprs-headless
namespace: aprs
spec:
clusterIP: None
selector:
app: aprs
ports:
- name: epmd
port: 4369
targetPort: 4369
- name: erlang
port: 9100
targetPort: 9100
- name: gossip
port: 45892
targetPort: 45892
protocol: UDP
publishNotReadyAddresses: true
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: aprs
namespace: aprs
spec:
serviceName: aprs-headless
replicas: 2
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 0
podManagementPolicy: Parallel
selector:
matchLabels:
app: aprs
template:
metadata:
labels:
app: aprs
spec:
terminationGracePeriodSeconds: 60
serviceAccountName: aprs-service-account
imagePullSecrets:
- name: ghcr-pull-secret
initContainers:
- name: wait-for-db
image: busybox:1.36.1
command:
- sh
- -c
- |
echo "Waiting for PostgreSQL at 10.0.19.5:5432..."
until nc -z 10.0.19.5 5432; do
echo "Database not ready, waiting..."
sleep 2
done
echo "Database is ready!"
- name: wait-for-redis
image: busybox:1.36.1
command:
- sh
- -c
- until nc -z redis 6379; do echo waiting for redis; sleep 2; done
- name: migrate
image: ghcr.io/aprsme/aprs.me:latest
imagePullPolicy: Always
command: ["/app/bin/aprsme", "eval", "Aprsme.Release.migrate()"]
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: aprs-secret
key: database-url
- name: SECRET_KEY_BASE
valueFrom:
secretKeyRef:
name: aprs-secret
key: secret-key-base
- name: MIX_ENV
value: "prod"
- name: POOL_SIZE
value: "1"
- name: SKIP_DB_CREATE
value: "true"
- name: REDIS_URL
value: "redis://redis.aprs.svc.cluster.local:6379"
- name: PHX_HOST
value: "aprs.me"
- name: PORT
value: "4000"
- name: APRS_CALLSIGN
value: "W5ISP-1"
- name: APRS_FILTER
value: "r/33/-96/1000000000000"
- name: APRS_PASSCODE
value: "15748"
- name: APRS_PORT
value: "10152"
- name: APRS_SERVER
value: "dallas.aprs2.net"
containers:
- name: aprs
image: ghcr.io/aprsme/aprs.me:latest
imagePullPolicy: Always
ports:
- containerPort: 4000
- containerPort: 4369
name: epmd
- containerPort: 9100
name: erlang
- containerPort: 45892
name: gossip
protocol: UDP
env:
- name: PHX_HOST
value: "aprs.me"
- name: PORT
value: "4000"
- name: PHX_CHECK_ORIGIN
value: "false"
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: aprs-secret
key: database-url
- name: SECRET_KEY_BASE
valueFrom:
secretKeyRef:
name: aprs-secret
key: secret-key-base
- name: PHX_SERVER
value: "true"
- name: MIX_ENV
value: "prod"
- name: APRS_CALLSIGN
value: "W5ISP-1"
- name: APRS_FILTER
value: "r/33/-96/1000000000000"
- name: APRS_PASSCODE
value: "15748"
- name: APRS_PORT
value: "10152"
- name: APRS_SERVER
value: "dallas.aprs2.net"
- name: PACKET_RETENTION_DAYS
value: "7"
- name: POOL_SIZE
value: "5"
- name: CLUSTER_ENABLED
value: "true"
- name: RELEASE_COOKIE
valueFrom:
secretKeyRef:
name: aprs-secret
key: erlang-cookie
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: REDIS_URL
value: "redis://redis.aprs.svc.cluster.local:6379"
- name: DRAIN_TIMEOUT_MS
value: "45000"
- name: SKIP_DB_CREATE
value: "true" # Database already exists on external server
- name: AUTO_MIGRATE
value: "false" # Disabled - migrations now handled by init container
resources:
limits:
memory: "1Gi"
requests:
memory: "1Gi"
cpu: "500m"
livenessProbe:
httpGet:
path: /health
port: 4000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /health
port: 4000
initialDelaySeconds: 5
periodSeconds: 5
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]

19
home/cluster/aprs/aprs-tailscale-service.yaml
View file

@ -1,19 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: aprs-tailscale
namespace: aprs
annotations:
tailscale.com/expose: "true"
tailscale.com/hostname: "aprs"
tailscale.com/funnel: "true"
spec:
type: LoadBalancer
loadBalancerClass: tailscale
selector:
app: aprs
ports:
- name: http
port: 80
targetPort: 4000
protocol: TCP

42
home/cluster/aprs/ceph-csi/ceph-config.yaml
View file

@ -1,42 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: ceph-config
namespace: ceph-csi
data:
ceph.conf: |
[global]
fsid = 6e5e8932-3613-471d-afa0-410fade4d4ff
mon_initial_members = lab02,lab03,lab04
mon_host = 10.0.16.231,10.0.16.232,10.0.16.233
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
---
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret
namespace: ceph-csi
type: kubernetes.io/rbd
stringData:
userID: kubernetes
userKey: AQAq14Jo/g4ZORAAwxpPOmqj1PYgmih6s6b4vA==
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ceph-csi-config
namespace: ceph-csi
data:
config.json: |
[
{
"clusterID": "6e5e8932-3613-471d-afa0-410fade4d4ff",
"monitors": [
"10.0.16.231:6789",
"10.0.16.232:6789",
"10.0.16.233:6789"
]
}
]

42
home/cluster/aprs/ceph-csi/ceph-csi-configmap.yaml
View file

@ -1,42 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: ceph-csi-config
namespace: ceph-csi
data:
config.json: |
[
{
"clusterID": "6e5e8932-3613-471d-afa0-410fade4d4ff",
"monitors": [
"10.0.16.231:6789",
"10.0.16.232:6789",
"10.0.16.233:6789"
]
}
]
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ceph-csi-encryption-kms-config
namespace: ceph-csi
data:
config.json: |
{}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ceph-config
namespace: ceph-csi
data:
ceph.conf: |
[global]
fsid = 6e5e8932-3613-471d-afa0-410fade4d4ff
mon_initial_members = lab02,lab03,lab04
mon_host = 10.0.16.231,10.0.16.232,10.0.16.233
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
keyring: |

223
home/cluster/aprs/ceph-csi/ceph-csi-deployment.yaml
View file

@ -1,223 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: csi-rbdplugin-provisioner
namespace: ceph-csi
spec:
replicas: 1
selector:
matchLabels:
app: csi-rbdplugin-provisioner
template:
metadata:
labels:
app: csi-rbdplugin-provisioner
spec:
serviceAccountName: rbd-csi-provisioner
containers:
- name: csi-provisioner
image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0
args:
- "--csi-address=/csi/csi.sock"
- "--v=5"
- "--timeout=150s"
- "--retry-interval-start=500ms"
- "--feature-gates=Topology=false"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-snapshotter
image: registry.k8s.io/sig-storage/csi-snapshotter:v6.3.0
args:
- "--csi-address=/csi/csi.sock"
- "--v=5"
- "--timeout=150s"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-attacher
image: registry.k8s.io/sig-storage/csi-attacher:v4.4.0
args:
- "--csi-address=/csi/csi.sock"
- "--v=5"
- "--timeout=150s"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-resizer
image: registry.k8s.io/sig-storage/csi-resizer:v1.9.0
args:
- "--csi-address=/csi/csi.sock"
- "--v=5"
- "--timeout=150s"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-rbdplugin
image: quay.io/cephcsi/cephcsi:v3.9.0
args:
- "--nodeid=$(NODE_ID)"
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=5"
- "--type=rbd"
- "--drivername=rbd.csi.ceph.com"
- "--configmap=ceph-csi-config"
env:
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CEPH_CONF
value: /etc/ceph/ceph.conf
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: host-sys
mountPath: /sys
- name: lib-modules
mountPath: /lib/modules
readOnly: true
- name: host-dev
mountPath: /dev
- name: ceph-config
mountPath: /etc/ceph/
- name: ceph-config-dir
mountPath: /etc/ceph-csi
volumes:
- name: socket-dir
emptyDir: {}
- name: host-sys
hostPath:
path: /sys
- name: lib-modules
hostPath:
path: /lib/modules
- name: host-dev
hostPath:
path: /dev
- name: ceph-config
configMap:
name: ceph-config
- name: ceph-config-dir
emptyDir: {}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: csi-rbdplugin
namespace: ceph-csi
spec:
selector:
matchLabels:
app: csi-rbdplugin
template:
metadata:
labels:
app: csi-rbdplugin
spec:
serviceAccountName: rbd-csi-nodeplugin
hostNetwork: true
hostPID: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: driver-registrar
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0
args:
- "--csi-address=/csi/csi.sock"
- "--kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: registration-dir
mountPath: /registration
- name: csi-rbdplugin
image: quay.io/cephcsi/cephcsi:v3.9.0
args:
- "--nodeid=$(NODE_ID)"
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=5"
- "--type=rbd"
- "--drivername=rbd.csi.ceph.com"
- "--configmap=ceph-csi-config"
env:
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CEPH_CONF
value: /etc/ceph/ceph.conf
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: plugin-dir
mountPath: /var/lib/kubelet/plugins
mountPropagation: "Bidirectional"
- name: mountpoint-dir
mountPath: /var/lib/kubelet/pods
mountPropagation: "Bidirectional"
- name: host-sys
mountPath: /sys
- name: lib-modules
mountPath: /lib/modules
readOnly: true
- name: host-dev
mountPath: /dev
- name: ceph-config
mountPath: /etc/ceph/
- name: ceph-config-dir
mountPath: /etc/ceph-csi
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/rbd.csi.ceph.com
type: DirectoryOrCreate
- name: plugin-dir
hostPath:
path: /var/lib/kubelet/plugins
type: Directory
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry
type: Directory
- name: mountpoint-dir
hostPath:
path: /var/lib/kubelet/pods
type: DirectoryOrCreate
- name: host-sys
hostPath:
path: /sys
- name: lib-modules
hostPath:
path: /lib/modules
- name: host-dev
hostPath:
path: /dev
- name: ceph-config
configMap:
name: ceph-config
- name: ceph-config-dir
emptyDir: {}

23
home/cluster/aprs/ceph-csi/ceph-csi-rbac-fix.yaml
View file

@ -1,23 +0,0 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-csi-provisioner-leader-election
namespace: ceph-csi
rules:
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-csi-provisioner-leader-election
namespace: ceph-csi
subjects:
- kind: ServiceAccount
name: rbd-csi-provisioner
namespace: ceph-csi
roleRef:
kind: Role
name: rbd-csi-provisioner-leader-election
apiGroup: rbac.authorization.k8s.io

99
home/cluster/aprs/ceph-csi/ceph-csi-rbd-driver.yaml
View file

@ -1,99 +0,0 @@
# This deploys the Ceph CSI RBD driver
# Source: https://github.com/ceph/ceph-csi
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-csi-provisioner
namespace: ceph-csi
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-csi-nodeplugin
namespace: ceph-csi
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-csi-provisioner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments/status"]
verbs: ["patch"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents"]
verbs: ["get", "list", "watch", "update", "create", "delete"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents/status"]
verbs: ["update"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-csi-nodeplugin
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-csi-provisioner
subjects:
- kind: ServiceAccount
name: rbd-csi-provisioner
namespace: ceph-csi
roleRef:
kind: ClusterRole
name: rbd-csi-provisioner
apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-csi-nodeplugin
subjects:
- kind: ServiceAccount
name: rbd-csi-nodeplugin
namespace: ceph-csi
roleRef:
kind: ClusterRole
name: rbd-csi-nodeplugin
apiGroup: rbac.authorization.k8s.io

9
home/cluster/aprs/ceph-csi/ceph-csi-secret.yaml
View file

@ -1,9 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: csi-rbd-secret
namespace: ceph-csi
type: kubernetes.io/rbd
stringData:
userID: kubernetes
userKey: AQAq14Jo/g4ZORAAwxpPOmqj1PYgmih6s6b4vA==

31
home/cluster/aprs/ceph-csi/ceph-storageclass.yaml
View file

@ -1,31 +0,0 @@
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-rbd
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: rbd.csi.ceph.com
parameters:
# Ceph cluster ID
clusterID: "6e5e8932-3613-471d-afa0-410fade4d4ff"
# Ceph pool name - usually 'rbd' or create a dedicated pool for K8s
pool: "kubernetes"
# Image features - using only layering for compatibility
imageFeatures: "layering"
# CSI secrets for provisioner
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi
csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi
# File system type
csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

15
home/cluster/aprs/ceph-csi/csi-config-map-template.yaml
View file

@ -1,15 +0,0 @@
#
# /!\ DO NOT MODIFY THIS FILE
#
# This file has been automatically generated by Ceph-CSI yamlgen.
# The source for the contents can be found in the api/deploy directory, make
# your modifications there.
#
---
apiVersion: v1
kind: ConfigMap
metadata:
name: "ceph-csi-config"
data:
config.json: |-
[]

9
home/cluster/aprs/ceph-csi/csi-driver.yaml
View file

@ -1,9 +0,0 @@
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
name: rbd.csi.ceph.com
spec:
attachRequired: true
podInfoOnMount: false
volumeLifecycleModes:
- Persistent

248
home/cluster/aprs/ceph-csi/csi-rbdplugin-provisioner.yaml
View file

@ -1,248 +0,0 @@
---
kind: Service
apiVersion: v1
metadata:
name: csi-rbdplugin-provisioner
# replace with non-default namespace name
namespace: default
labels:
app: csi-metrics
spec:
selector:
app: csi-rbdplugin-provisioner
ports:
- name: http-metrics
port: 8080
protocol: TCP
targetPort: 8680
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: csi-rbdplugin-provisioner
# replace with non-default namespace name
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: csi-rbdplugin-provisioner
template:
metadata:
labels:
app: csi-rbdplugin-provisioner
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- csi-rbdplugin-provisioner
topologyKey: "kubernetes.io/hostname"
serviceAccountName: rbd-csi-provisioner
priorityClassName: system-cluster-critical
containers:
- name: csi-provisioner
image: registry.k8s.io/sig-storage/csi-provisioner:v3.5.0
args:
- "--csi-address=$(ADDRESS)"
- "--v=1"
- "--timeout=150s"
- "--retry-interval-start=500ms"
- "--leader-election=true"
# set it to true to use topology based provisioning
- "--feature-gates=Topology=false"
- "--feature-gates=HonorPVReclaimPolicy=true"
- "--prevent-volume-mode-conversion=true"
# if fstype is not specified in storageclass, ext4 is default
- "--default-fstype=ext4"
- "--extra-create-metadata=true"
env:
- name: ADDRESS
value: unix:///csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-snapshotter
image: registry.k8s.io/sig-storage/csi-snapshotter:v6.2.2
args:
- "--csi-address=$(ADDRESS)"
- "--v=1"
- "--timeout=150s"
- "--leader-election=true"
- "--extra-create-metadata=true"
env:
- name: ADDRESS
value: unix:///csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-attacher
image: registry.k8s.io/sig-storage/csi-attacher:v4.3.0
args:
- "--v=1"
- "--csi-address=$(ADDRESS)"
- "--leader-election=true"
- "--retry-interval-start=500ms"
- "--default-fstype=ext4"
env:
- name: ADDRESS
value: /csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-resizer
image: registry.k8s.io/sig-storage/csi-resizer:v1.8.0
args:
- "--csi-address=$(ADDRESS)"
- "--v=1"
- "--timeout=150s"
- "--leader-election"
- "--retry-interval-start=500ms"
- "--handle-volume-inuse-error=false"
- "--feature-gates=RecoverVolumeExpansionFailure=true"
env:
- name: ADDRESS
value: unix:///csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-rbdplugin
image: quay.io/cephcsi/cephcsi:v3.9.0
args:
- "--nodeid=$(NODE_ID)"
- "--type=rbd"
- "--controllerserver=true"
- "--endpoint=$(CSI_ENDPOINT)"
- "--csi-addons-endpoint=$(CSI_ADDONS_ENDPOINT)"
- "--v=5"
- "--drivername=rbd.csi.ceph.com"
- "--pidlimit=-1"
- "--rbdhardmaxclonedepth=8"
- "--rbdsoftmaxclonedepth=4"
- "--enableprofiling=false"
- "--setmetadata=true"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# - name: KMS_CONFIGMAP_NAME
# value: encryptionConfig
- name: CSI_ENDPOINT
value: unix:///csi/csi-provisioner.sock
- name: CSI_ADDONS_ENDPOINT
value: unix:///csi/csi-addons.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- mountPath: /dev
name: host-dev
- mountPath: /sys
name: host-sys
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- name: ceph-csi-config
mountPath: /etc/ceph-csi-config/
- name: ceph-csi-encryption-kms-config
mountPath: /etc/ceph-csi-encryption-kms-config/
- name: keys-tmp-dir
mountPath: /tmp/csi/keys
- name: ceph-config
mountPath: /etc/ceph/
- name: oidc-token
mountPath: /run/secrets/tokens
readOnly: true
- name: csi-rbdplugin-controller
image: quay.io/cephcsi/cephcsi:v3.9.0
args:
- "--type=controller"
- "--v=5"
- "--drivername=rbd.csi.ceph.com"
- "--drivernamespace=$(DRIVER_NAMESPACE)"
- "--setmetadata=true"
env:
- name: DRIVER_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: ceph-csi-config
mountPath: /etc/ceph-csi-config/
- name: keys-tmp-dir
mountPath: /tmp/csi/keys
- name: ceph-config
mountPath: /etc/ceph/
- name: liveness-prometheus
image: quay.io/cephcsi/cephcsi:v3.9.0
args:
- "--type=liveness"
- "--endpoint=$(CSI_ENDPOINT)"
- "--metricsport=8680"
- "--metricspath=/metrics"
- "--polltime=60s"
- "--timeout=3s"
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi-provisioner.sock
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- name: socket-dir
mountPath: /csi
imagePullPolicy: "IfNotPresent"
volumes:
- name: host-dev
hostPath:
path: /dev
- name: host-sys
hostPath:
path: /sys
- name: lib-modules
hostPath:
path: /lib/modules
- name: socket-dir
emptyDir: {
medium: "Memory"
}
- name: ceph-config
configMap:
name: ceph-config
- name: ceph-csi-config
configMap:
name: ceph-csi-config
- name: ceph-csi-encryption-kms-config
configMap:
name: ceph-csi-encryption-kms-config
- name: keys-tmp-dir
emptyDir: {
medium: "Memory"
}
- name: oidc-token
projected:
sources:
- serviceAccountToken:
path: oidc-token
expirationSeconds: 3600
audience: ceph-csi-kms

229
home/cluster/aprs/ceph-csi/csi-rbdplugin.yaml
View file

@ -1,229 +0,0 @@
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: csi-rbdplugin
# replace with non-default namespace name
namespace: default
spec:
selector:
matchLabels:
app: csi-rbdplugin
template:
metadata:
labels:
app: csi-rbdplugin
spec:
serviceAccountName: rbd-csi-nodeplugin
hostNetwork: true
hostPID: true
priorityClassName: system-node-critical
# to use e.g. Rook orchestrated cluster, and mons' FQDN is
# resolved through k8s service, set dns policy to cluster first
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: driver-registrar
# This is necessary only for systems with SELinux, where
# non-privileged sidecar containers cannot access unix domain socket
# created by privileged CSI driver container.
securityContext:
privileged: true
allowPrivilegeEscalation: true
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0
args:
- "--v=1"
- "--csi-address=/csi/csi.sock"
- "--kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock"
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: registration-dir
mountPath: /registration
- name: csi-rbdplugin
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
image: quay.io/cephcsi/cephcsi:v3.9.0
args:
- "--nodeid=$(NODE_ID)"
- "--pluginpath=/var/lib/kubelet/plugins"
- "--stagingpath=/var/lib/kubelet/plugins/kubernetes.io/csi/"
- "--type=rbd"
- "--nodeserver=true"
- "--endpoint=$(CSI_ENDPOINT)"
- "--csi-addons-endpoint=$(CSI_ADDONS_ENDPOINT)"
- "--v=5"
- "--drivername=rbd.csi.ceph.com"
- "--enableprofiling=false"
# If topology based provisioning is desired, configure required
# node labels representing the nodes topology domain
# and pass the label names below, for CSI to consume and advertise
# its equivalent topology domain
# - "--domainlabels=failure-domain/region,failure-domain/zone"
#
# Options to enable read affinity.
# If enabled Ceph CSI will fetch labels from kubernetes node and
# pass `read_from_replica=localize,crush_location=type:value` during
# rbd map command. refer:
# https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options
# for more details.
# - "--enable-read-affinity=true"
# - "--crush-location-labels=topology.io/zone,topology.io/rack"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# - name: KMS_CONFIGMAP_NAME
# value: encryptionConfig
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: CSI_ADDONS_ENDPOINT
value: unix:///csi/csi-addons.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- mountPath: /dev
name: host-dev
- mountPath: /sys
name: host-sys
- mountPath: /run/mount
name: host-mount
- mountPath: /etc/selinux
name: etc-selinux
readOnly: true
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- name: ceph-csi-config
mountPath: /etc/ceph-csi-config/
- name: ceph-csi-encryption-kms-config
mountPath: /etc/ceph-csi-encryption-kms-config/
- name: plugin-dir
mountPath: /var/lib/kubelet/plugins
mountPropagation: "Bidirectional"
- name: mountpoint-dir
mountPath: /var/lib/kubelet/pods
mountPropagation: "Bidirectional"
- name: keys-tmp-dir
mountPath: /tmp/csi/keys
- name: ceph-logdir
mountPath: /var/log/ceph
- name: ceph-config
mountPath: /etc/ceph/
- name: oidc-token
mountPath: /run/secrets/tokens
readOnly: true
- name: liveness-prometheus
securityContext:
privileged: true
allowPrivilegeEscalation: true
image: quay.io/cephcsi/cephcsi:v3.9.0
args:
- "--type=liveness"
- "--endpoint=$(CSI_ENDPOINT)"
- "--metricsport=8680"
- "--metricspath=/metrics"
- "--polltime=60s"
- "--timeout=3s"
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- name: socket-dir
mountPath: /csi
imagePullPolicy: "IfNotPresent"
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/rbd.csi.ceph.com
type: DirectoryOrCreate
- name: plugin-dir
hostPath:
path: /var/lib/kubelet/plugins
type: Directory
- name: mountpoint-dir
hostPath:
path: /var/lib/kubelet/pods
type: DirectoryOrCreate
- name: ceph-logdir
hostPath:
path: /var/log/ceph
type: DirectoryOrCreate
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry/
type: Directory
- name: host-dev
hostPath:
path: /dev
- name: host-sys
hostPath:
path: /sys
- name: etc-selinux
hostPath:
path: /etc/selinux
- name: host-mount
hostPath:
path: /run/mount
- name: lib-modules
hostPath:
path: /lib/modules
- name: ceph-config
configMap:
name: ceph-config
- name: ceph-csi-config
configMap:
name: ceph-csi-config
- name: ceph-csi-encryption-kms-config
configMap:
name: ceph-csi-encryption-kms-config
- name: keys-tmp-dir
emptyDir: {
medium: "Memory"
}
- name: oidc-token
projected:
sources:
- serviceAccountToken:
path: oidc-token
expirationSeconds: 3600
audience: ceph-csi-kms
---
# This is a service to expose the liveness metrics
apiVersion: v1
kind: Service
metadata:
name: csi-metrics-rbdplugin
# replace with non-default namespace name
namespace: default
labels:
app: csi-metrics
spec:
ports:
- name: http-metrics
port: 8080
protocol: TCP
targetPort: 8680
selector:
app: csi-rbdplugin

14
home/cluster/aprs/ceph-csi/kustomization.yaml
View file

@ -1,14 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ceph-csi
resources:
- namespace.yaml
- ceph-csi-configmap.yaml
- ceph-csi-secret.yaml
- ceph-csi-rbd-driver.yaml
- csi-driver.yaml
- csi-rbdplugin-provisioner.yaml
- csi-rbdplugin.yaml
- ceph-storageclass.yaml

25
home/cluster/aprs/create-image-pull-secret.sh Executable file
View file

@ -0,0 +1,25 @@
#!/bin/bash
# Create a GitHub personal access token with read:packages scope
# Then run this script with your GitHub username and token
if [ "$#" -ne 2 ]; then
echo "Usage: $0 <github-username> <github-token>"
echo "Create a token at: https://github.com/settings/tokens/new"
echo "Required scope: read:packages"
exit 1
fi
GITHUB_USER=$1
GITHUB_TOKEN=$2
kubectl create secret docker-registry ghcr-secret \
--docker-server=ghcr.io \
--docker-username=$GITHUB_USER \
--docker-password=$GITHUB_TOKEN \
--namespace=aprs
echo "Secret created! Now update the StatefulSet to use it:"
echo "Add this to the pod spec:"
echo " imagePullSecrets:"
echo " - name: ghcr-secret"

52
home/cluster/aprs/deploy.sh Executable file
View file

@ -0,0 +1,52 @@
#!/bin/bash
set -e
echo "Deploying APRS.me to k3s cluster at 204.110.191.2"
# Use homelab context
kubectl config use-context homelab
# Create namespace
echo "Creating namespace..."
kubectl apply -f namespace.yaml --validate=false
# Deploy PostgreSQL with PostGIS
echo "Deploying PostgreSQL 17 with PostGIS..."
kubectl apply -f postgres-secret-secure.yaml --validate=false
kubectl apply -f postgres-deployment.yaml --validate=false
# Wait for PostgreSQL to be ready
echo "Waiting for PostgreSQL to be ready..."
kubectl wait --for=condition=ready pod -l app=postgres-aprs -n aprs --timeout=300s
# Deploy Redis
echo "Deploying Redis..."
kubectl apply -f redis.yaml --validate=false
# Wait for Redis to be ready
echo "Waiting for Redis to be ready..."
kubectl wait --for=condition=ready pod -l app=redis -n aprs --timeout=120s
# Deploy APRS.me
echo "Deploying APRS.me application..."
kubectl apply -f aprs-secret-secure.yaml --validate=false
kubectl apply -f aprs-configmap.yaml --validate=false
kubectl apply -f aprs-rbac.yaml --validate=false
kubectl apply -f aprs-statefulset-with-pullsecret.yaml --validate=false
# Deploy Traefik ingress
echo "Deploying Traefik ingress..."
kubectl apply -f aprs-ingressroute.yaml --validate=false
echo "Deployment complete!"
echo ""
echo "To check status:"
echo "kubectl get pods -n aprs"
echo ""
echo "To get the LoadBalancer IP:"
echo "kubectl get svc aprs -n aprs"
echo ""
echo "Remember to:"
echo "1. Create GitHub image pull secret: ./create-image-pull-secret.sh <username> <token>"
echo "2. Access the application at: https://aprs.me"
echo "3. DNS points to 204.110.191.3 (Traefik ingress)"

13
home/cluster/aprs/ghcr-pull-secret-template.yaml
View file

@ -1,13 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: ghcr-pull-secret
namespace: aprs
type: kubernetes.io/dockerconfigjson
data:
# This needs to be base64 encoded JSON in the format:
# {"auths":{"ghcr.io":{"username":"YOUR_GITHUB_USERNAME","password":"YOUR_GITHUB_PAT","auth":"BASE64_ENCODED_USERNAME:PAT"}}}
#
# You can generate this by running:
# echo -n '{"auths":{"ghcr.io":{"username":"YOUR_GITHUB_USERNAME","password":"YOUR_GITHUB_PAT","auth":"'$(echo -n "YOUR_GITHUB_USERNAME:YOUR_GITHUB_PAT" | base64)'"}}}' | base64
.dockerconfigjson: REPLACE_THIS_WITH_BASE64_ENCODED_DOCKER_CONFIG

Some files were not shown because too many files have changed in this diff Show more