# Setting up Ceph Storage for PostgreSQL in K3s

## Prerequisites

Before applying these manifests, you need to gather information from your Proxmox Ceph cluster:

1. **Ceph Monitor IPs**: Get the monitor IPs from Proxmox
   ```bash
   # On Proxmox node:
   ceph mon dump
   ```

2. **Ceph Cluster ID**: Get your Ceph cluster ID
   ```bash
   # On Proxmox node:
   ceph fsid
   ```

3. **Create a Ceph pool for Kubernetes** (if not already exists):
   ```bash
   # On Proxmox node:
   ceph osd pool create kubernetes 128
   ceph osd pool application enable kubernetes rbd
   ```

4. **Create a Ceph user for Kubernetes**:
   ```bash
   # On Proxmox node:
   ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
   ```

## Configuration Steps

1. **Update the Ceph configuration** in `ceph-csi/ceph-config.yaml`:
   - Replace `mon_host` with your actual Ceph monitor IPs
   - Replace the `userKey` with the key from step 4 above

2. **Update the StorageClass** in `ceph-csi/ceph-storageclass.yaml`:
   - Replace `clusterID` with your Ceph cluster ID from step 2
   - Ensure the `pool` name matches what you created in step 3

3. **Apply the Ceph CSI driver**:
   ```bash
   kubectl apply -k ceph-csi/
   ```

4. **Verify the installation**:
   ```bash
   # Check if CSI pods are running
   kubectl get pods -n ceph-csi

   # Check if StorageClass is created
   kubectl get storageclass ceph-rbd
   ```

## Migrating PostgreSQL to Ceph Storage

1. **Backup existing data** (if any):
   ```bash
   kubectl exec -n aprs deployment/postgis -- pg_dump -U aprs aprs_prod > backup.sql
   ```

2. **Scale down the current deployment**:
   ```bash
   kubectl scale deployment postgis -n aprs --replicas=0
   ```

3. **Create the new PVC with Ceph**:
   ```bash
   kubectl apply -f postgis-pvc-ceph.yaml
   ```

4. **Update the deployment to use Ceph PVC**:
   ```bash
   kubectl apply -f postgis-deployment-ceph.yaml
   ```

5. **Restore data** (if needed):
   ```bash
   kubectl exec -i -n aprs deployment/postgis -- psql -U aprs aprs_prod < backup.sql
   ```

## Troubleshooting

If pods fail to mount Ceph volumes:

1. Check CSI driver logs:
   ```bash
   kubectl logs -n ceph-csi deployment/csi-rbdplugin-provisioner
   kubectl logs -n ceph-csi daemonset/csi-rbdplugin
   ```

2. Verify Ceph connectivity from K3s nodes:
   ```bash
   # Install ceph-common on K3s nodes
   apt-get install ceph-common
   
   # Test connection
   ceph -c /etc/ceph/ceph.conf -n client.kubernetes --keyring /etc/ceph/keyring status
   ```

3. Check PVC events:
   ```bash
   kubectl describe pvc postgis-pvc-ceph -n aprs
   ```