infra/cluster/ceph-csi/README.md

Ceph CSI RBD Driver for Talos Cluster

This directory contains manifests for deploying the Ceph CSI RBD driver to use Proxmox Ceph storage directly in the Talos Kubernetes cluster.

Configuration

• Ceph Cluster FSID: 7e968d44-61b4-4342-8318-2e8e48808e68
• Ceph Monitors: 10.0.19.101:6789, 10.0.19.102:6789, 10.0.19.103:6789
• Ceph Pool: kubernetes
• Ceph User: client.kubernetes
• StorageClass: ceph-rbd (default)

Deployment

All manifests have been applied to the cluster:

cd /Users/graham/dev/infra/cluster/ceph-csi
./deploy.sh

Deployed Components

1. Namespace: ceph-csi-rbd (with privileged pod security)
2. ConfigMaps: Ceph cluster configuration and encryption KMS config
3. Secret: Ceph credentials for client.kubernetes user
4. RBAC: ServiceAccounts, ClusterRoles, and bindings for provisioner and plugin
5. Provisioner: 3-replica deployment for volume provisioning
6. Plugin: DaemonSet running on all nodes (including control plane)
7. CSIDriver: CSI driver registration
8. StorageClass: ceph-rbd (default, WaitForFirstConsumer binding mode)

Status

# Check all pods
kubectl get pods -n ceph-csi-rbd

# Check StorageClass
kubectl get storageclass ceph-rbd

# Check CSI nodes
kubectl get csinodes

Topology Configuration

The Ceph CSI driver requires topology configuration for proper volume scheduling. This has been configured as follows:

Solution Applied:

1. Added --domainlabels=topology.kubernetes.io/zone to CSI node plugin
2. Labeled all nodes with topology.kubernetes.io/zone=default
3. CSI driver automatically creates topology.rbd.csi.ceph.com/zone=default labels
4. Restarted provisioner pods to pick up updated CSINode topology information

Result: PVCs now provision successfully and volumes are created in the Ceph kubernetes pool.

Verification:

# Check topology labels on nodes
kubectl get nodes --show-labels | grep topology

# Check CSINode topology keys
kubectl get csinode -o yaml | grep topologyKeys -A 2

# List RBD images in Ceph
ssh root@10.0.19.101 "rbd ls -p kubernetes"

Testing

A test PVC and pod are provided:

kubectl apply -f test-pvc.yaml

# Check PVC status
kubectl get pvc ceph-rbd-test

# Check pod status
kubectl get pod ceph-rbd-test-pod

# Clean up test
kubectl delete -f test-pvc.yaml

Ceph Pool Management

# SSH to any Proxmox node
ssh root@10.0.19.101

# List pools
ceph osd pool ls

# Check pool usage
ceph df

# List RBD images in kubernetes pool
rbd ls -p kubernetes

# Get image info
rbd info -p kubernetes <image-name>

Troubleshooting

Check Provisioner Logs

kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner -c csi-provisioner --tail=50
kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner -c csi-rbdplugin --tail=50

Check Plugin Logs

kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin -c csi-rbdplugin --tail=50

Restart Provisioner

kubectl delete pods -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner

Check Ceph Health

ssh root@10.0.19.101 "ceph health detail"

Files

• 00-namespace.yaml - Namespace definition
• 01-configmap.yaml - Ceph cluster and encryption configuration
• 02-secret.yaml - Ceph client credentials
• 03-rbac-provisioner.yaml - RBAC for provisioner
• 04-rbac-plugin.yaml - RBAC for node plugin
• 05-provisioner.yaml - Provisioner deployment
• 06-plugin.yaml - Plugin DaemonSet (with control plane tolerations)
• 07-csidriver.yaml - CSI driver registration
• 08-storageclass.yaml - StorageClass definition
• deploy.sh - Deployment script
• test-pvc.yaml - Test PVC and pod

Alternative: Longhorn

If Ceph CSI continues to have issues, consider deploying Longhorn instead:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml

Longhorn is simpler to set up and doesn't require external Ceph storage.