infra/clusters/aprs/ceph-setup-guide.md

Setting up Ceph Storage for PostgreSQL in K3s

Prerequisites

Before applying these manifests, you need to gather information from your Proxmox Ceph cluster:

1. Ceph Monitor IPs: Get the monitor IPs from Proxmox

   # On Proxmox node:
   ceph mon dump
   

2. Ceph Cluster ID: Get your Ceph cluster ID

   # On Proxmox node:
   ceph fsid
   

3. Create a Ceph pool for Kubernetes (if not already exists):

   # On Proxmox node:
   ceph osd pool create kubernetes 128
   ceph osd pool application enable kubernetes rbd
   

4. Create a Ceph user for Kubernetes:

   # On Proxmox node:
   ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
   

Configuration Steps

1. Update the Ceph configuration in ceph-csi/ceph-config.yaml:

   • Replace mon_host with your actual Ceph monitor IPs
   • Replace the userKey with the key from step 4 above

2. Update the StorageClass in ceph-csi/ceph-storageclass.yaml:

   • Replace clusterID with your Ceph cluster ID from step 2
   • Ensure the pool name matches what you created in step 3

3. Apply the Ceph CSI driver:

   kubectl apply -k ceph-csi/
   

4. Verify the installation:

   # Check if CSI pods are running
   kubectl get pods -n ceph-csi
   
   # Check if StorageClass is created
   kubectl get storageclass ceph-rbd
   

Migrating PostgreSQL to Ceph Storage

1. Backup existing data (if any):

   kubectl exec -n aprs deployment/postgis -- pg_dump -U aprs aprs_prod > backup.sql
   

2. Scale down the current deployment:

   kubectl scale deployment postgis -n aprs --replicas=0
   

3. Create the new PVC with Ceph:

   kubectl apply -f postgis-pvc-ceph.yaml
   

4. Update the deployment to use Ceph PVC:

   kubectl apply -f postgis-deployment-ceph.yaml
   

5. Restore data (if needed):

   kubectl exec -i -n aprs deployment/postgis -- psql -U aprs aprs_prod < backup.sql
   

Troubleshooting

If pods fail to mount Ceph volumes:

1. Check CSI driver logs:

   kubectl logs -n ceph-csi deployment/csi-rbdplugin-provisioner
   kubectl logs -n ceph-csi daemonset/csi-rbdplugin
   

2. Verify Ceph connectivity from K3s nodes:

   # Install ceph-common on K3s nodes
   apt-get install ceph-common
   
   # Test connection
   ceph -c /etc/ceph/ceph.conf -n client.kubernetes --keyring /etc/ceph/keyring status
   

3. Check PVC events:

   kubectl describe pvc postgis-pvc-ceph -n aprs