infra/cluster/ceph-csi

Ceph CSI RBD Driver for Talos Cluster

This directory contains manifests for deploying the Ceph CSI RBD driver to use Proxmox Ceph storage directly in the Talos Kubernetes cluster.

Configuration

• Ceph Cluster FSID: 7e968d44-61b4-4342-8318-2e8e48808e68
• Ceph Monitors: 10.0.15.101:6789, 10.0.15.102:6789, 10.0.15.103:6789
• Ceph Pool: kubernetes
• Ceph User: client.kubernetes
• StorageClass: ceph-rbd (default)

Deployment

All manifests have been applied to the cluster:

cd /Users/graham/dev/infra/cluster/ceph-csi
./deploy.sh

Deployed Components

1. Namespace: ceph-csi-rbd (with privileged pod security)
2. ConfigMaps: Ceph cluster configuration and encryption KMS config
3. Secret: Ceph credentials for client.kubernetes user
4. RBAC: ServiceAccounts, ClusterRoles, and bindings for provisioner and plugin
5. Provisioner: 3-replica deployment for volume provisioning
6. Plugin: DaemonSet running on all nodes (including control plane)
7. CSIDriver: CSI driver registration
8. StorageClass: ceph-rbd (default, WaitForFirstConsumer binding mode)

Status

# Check all pods
kubectl get pods -n ceph-csi-rbd

# Check StorageClass
kubectl get storageclass ceph-rbd

# Check CSI nodes
kubectl get csinodes

Topology Configuration

The Ceph CSI driver requires topology configuration for proper volume scheduling. This has been configured as follows:

Solution Applied:

1. Added --domainlabels=topology.kubernetes.io/zone to CSI node plugin
2. Labeled all nodes with topology.kubernetes.io/zone=default
3. CSI driver automatically creates topology.rbd.csi.ceph.com/zone=default labels
4. Restarted provisioner pods to pick up updated CSINode topology information

Result: PVCs now provision successfully and volumes are created in the Ceph kubernetes pool.

Verification:

# Check topology labels on nodes
kubectl get nodes --show-labels | grep topology

# Check CSINode topology keys
kubectl get csinode -o yaml | grep topologyKeys -A 2

# List RBD images in Ceph
ssh root@10.0.15.101 "rbd ls -p kubernetes"

Testing

A test PVC and pod are provided:

kubectl apply -f test-pvc.yaml

# Check PVC status
kubectl get pvc ceph-rbd-test

# Check pod status
kubectl get pod ceph-rbd-test-pod

# Clean up test
kubectl delete -f test-pvc.yaml

Ceph Pool Management

# SSH to any Proxmox node
ssh root@10.0.15.101

# List pools
ceph osd pool ls

# Check pool usage
ceph df

# List RBD images in kubernetes pool
rbd ls -p kubernetes

# Get image info
rbd info -p kubernetes <image-name>

Troubleshooting

Check Provisioner Logs

kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner -c csi-provisioner --tail=50
kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner -c csi-rbdplugin --tail=50

Check Plugin Logs

kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin -c csi-rbdplugin --tail=50

Restart Provisioner

kubectl delete pods -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner

Check Ceph Health

ssh root@10.0.15.101 "ceph health detail"

Files

• 00-namespace.yaml - Namespace definition
• 01-configmap.yaml - Ceph cluster and encryption configuration
• 02-secret.yaml - Ceph client credentials
• 03-rbac-provisioner.yaml - RBAC for provisioner
• 04-rbac-plugin.yaml - RBAC for node plugin
• 05-provisioner.yaml - Provisioner deployment
• 06-plugin.yaml - Plugin DaemonSet (with control plane tolerations)
• 07-csidriver.yaml - CSI driver registration
• 08-storageclass.yaml - StorageClass definition
• deploy.sh - Deployment script
• test-pvc.yaml - Test PVC and pod

Alternative: Longhorn

If Ceph CSI continues to have issues, consider deploying Longhorn instead:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml

Longhorn is simpler to set up and doesn't require external Ceph storage.