# Ceph CSI RBD Driver for Talos Cluster

This directory contains manifests for deploying the Ceph CSI RBD driver to use Proxmox Ceph storage directly in the Talos Kubernetes cluster.

## Configuration

- **Ceph Cluster FSID**: `7e968d44-61b4-4342-8318-2e8e48808e68`
- **Ceph Monitors**: 10.0.19.101:6789, 10.0.19.102:6789, 10.0.19.103:6789
- **Ceph Pool**: kubernetes
- **Ceph User**: client.kubernetes
- **StorageClass**: ceph-rbd (default)

## Deployment

All manifests have been applied to the cluster:

```bash
cd /Users/graham/dev/infra/cluster/ceph-csi
./deploy.sh
```

## Deployed Components

1. **Namespace**: ceph-csi-rbd (with privileged pod security)
2. **ConfigMaps**: Ceph cluster configuration and encryption KMS config
3. **Secret**: Ceph credentials for client.kubernetes user
4. **RBAC**: ServiceAccounts, ClusterRoles, and bindings for provisioner and plugin
5. **Provisioner**: 3-replica deployment for volume provisioning
6. **Plugin**: DaemonSet running on all nodes (including control plane)
7. **CSIDriver**: CSI driver registration
8. **StorageClass**: ceph-rbd (default, WaitForFirstConsumer binding mode)

## Status

```bash
# Check all pods
kubectl get pods -n ceph-csi-rbd

# Check StorageClass
kubectl get storageclass ceph-rbd

# Check CSI nodes
kubectl get csinodes
```

## Topology Configuration

The Ceph CSI driver requires topology configuration for proper volume scheduling. This has been configured as follows:

**Solution Applied**:
1. Added `--domainlabels=topology.kubernetes.io/zone` to CSI node plugin
2. Labeled all nodes with `topology.kubernetes.io/zone=default`
3. CSI driver automatically creates `topology.rbd.csi.ceph.com/zone=default` labels
4. Restarted provisioner pods to pick up updated CSINode topology information

**Result**: PVCs now provision successfully and volumes are created in the Ceph kubernetes pool.

**Verification**:
```bash
# Check topology labels on nodes
kubectl get nodes --show-labels | grep topology

# Check CSINode topology keys
kubectl get csinode -o yaml | grep topologyKeys -A 2

# List RBD images in Ceph
ssh root@10.0.19.101 "rbd ls -p kubernetes"
```

## Testing

A test PVC and pod are provided:

```bash
kubectl apply -f test-pvc.yaml

# Check PVC status
kubectl get pvc ceph-rbd-test

# Check pod status
kubectl get pod ceph-rbd-test-pod

# Clean up test
kubectl delete -f test-pvc.yaml
```

## Ceph Pool Management

```bash
# SSH to any Proxmox node
ssh root@10.0.19.101

# List pools
ceph osd pool ls

# Check pool usage
ceph df

# List RBD images in kubernetes pool
rbd ls -p kubernetes

# Get image info
rbd info -p kubernetes <image-name>
```

## Troubleshooting

### Check Provisioner Logs
```bash
kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner -c csi-provisioner --tail=50
kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner -c csi-rbdplugin --tail=50
```

### Check Plugin Logs
```bash
kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin -c csi-rbdplugin --tail=50
```

### Restart Provisioner
```bash
kubectl delete pods -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner
```

### Check Ceph Health
```bash
ssh root@10.0.19.101 "ceph health detail"
```

## Files

- `00-namespace.yaml` - Namespace definition
- `01-configmap.yaml` - Ceph cluster and encryption configuration
- `02-secret.yaml` - Ceph client credentials
- `03-rbac-provisioner.yaml` - RBAC for provisioner
- `04-rbac-plugin.yaml` - RBAC for node plugin
- `05-provisioner.yaml` - Provisioner deployment
- `06-plugin.yaml` - Plugin DaemonSet (with control plane tolerations)
- `07-csidriver.yaml` - CSI driver registration
- `08-storageclass.yaml` - StorageClass definition
- `deploy.sh` - Deployment script
- `test-pvc.yaml` - Test PVC and pod

## Alternative: Longhorn

If Ceph CSI continues to have issues, consider deploying Longhorn instead:

```bash
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml
```

Longhorn is simpler to set up and doesn't require external Ceph storage.