Graham McIntire
39bedb08d1
Proxmox: node1-3 → 10.0.19.101-103 Talos cp1-3 → 10.0.19.1-3, workers → 10.0.19.4-6 K8s endpoint → VIP https://10.0.19.10:6443 Ansible: prom → 10.0.19.31, db → 10.0.19.30 Talos: added VIP block to controlplane.yaml base config Promtail: Loki URL → 10.0.19.31 Docs: all references updated, talos4 removed
151 lines
3.9 KiB
Markdown
151 lines
3.9 KiB
Markdown
# Ceph CSI RBD Driver for Talos Cluster
|
|
|
|
This directory contains manifests for deploying the Ceph CSI RBD driver to use Proxmox Ceph storage directly in the Talos Kubernetes cluster.
|
|
|
|
## Configuration
|
|
|
|
- **Ceph Cluster FSID**: `7e968d44-61b4-4342-8318-2e8e48808e68`
|
|
- **Ceph Monitors**: 10.0.19.101:6789, 10.0.19.102:6789, 10.0.19.103:6789
|
|
- **Ceph Pool**: kubernetes
|
|
- **Ceph User**: client.kubernetes
|
|
- **StorageClass**: ceph-rbd (default)
|
|
|
|
## Deployment
|
|
|
|
All manifests have been applied to the cluster:
|
|
|
|
```bash
|
|
cd /Users/graham/dev/infra/cluster/ceph-csi
|
|
./deploy.sh
|
|
```
|
|
|
|
## Deployed Components
|
|
|
|
1. **Namespace**: ceph-csi-rbd (with privileged pod security)
|
|
2. **ConfigMaps**: Ceph cluster configuration and encryption KMS config
|
|
3. **Secret**: Ceph credentials for client.kubernetes user
|
|
4. **RBAC**: ServiceAccounts, ClusterRoles, and bindings for provisioner and plugin
|
|
5. **Provisioner**: 3-replica deployment for volume provisioning
|
|
6. **Plugin**: DaemonSet running on all nodes (including control plane)
|
|
7. **CSIDriver**: CSI driver registration
|
|
8. **StorageClass**: ceph-rbd (default, WaitForFirstConsumer binding mode)
|
|
|
|
## Status
|
|
|
|
```bash
|
|
# Check all pods
|
|
kubectl get pods -n ceph-csi-rbd
|
|
|
|
# Check StorageClass
|
|
kubectl get storageclass ceph-rbd
|
|
|
|
# Check CSI nodes
|
|
kubectl get csinodes
|
|
```
|
|
|
|
## Topology Configuration
|
|
|
|
The Ceph CSI driver requires topology configuration for proper volume scheduling. This has been configured as follows:
|
|
|
|
**Solution Applied**:
|
|
1. Added `--domainlabels=topology.kubernetes.io/zone` to CSI node plugin
|
|
2. Labeled all nodes with `topology.kubernetes.io/zone=default`
|
|
3. CSI driver automatically creates `topology.rbd.csi.ceph.com/zone=default` labels
|
|
4. Restarted provisioner pods to pick up updated CSINode topology information
|
|
|
|
**Result**: PVCs now provision successfully and volumes are created in the Ceph kubernetes pool.
|
|
|
|
**Verification**:
|
|
```bash
|
|
# Check topology labels on nodes
|
|
kubectl get nodes --show-labels | grep topology
|
|
|
|
# Check CSINode topology keys
|
|
kubectl get csinode -o yaml | grep topologyKeys -A 2
|
|
|
|
# List RBD images in Ceph
|
|
ssh root@10.0.19.101 "rbd ls -p kubernetes"
|
|
```
|
|
|
|
## Testing
|
|
|
|
A test PVC and pod are provided:
|
|
|
|
```bash
|
|
kubectl apply -f test-pvc.yaml
|
|
|
|
# Check PVC status
|
|
kubectl get pvc ceph-rbd-test
|
|
|
|
# Check pod status
|
|
kubectl get pod ceph-rbd-test-pod
|
|
|
|
# Clean up test
|
|
kubectl delete -f test-pvc.yaml
|
|
```
|
|
|
|
## Ceph Pool Management
|
|
|
|
```bash
|
|
# SSH to any Proxmox node
|
|
ssh root@10.0.19.101
|
|
|
|
# List pools
|
|
ceph osd pool ls
|
|
|
|
# Check pool usage
|
|
ceph df
|
|
|
|
# List RBD images in kubernetes pool
|
|
rbd ls -p kubernetes
|
|
|
|
# Get image info
|
|
rbd info -p kubernetes <image-name>
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Check Provisioner Logs
|
|
```bash
|
|
kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner -c csi-provisioner --tail=50
|
|
kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner -c csi-rbdplugin --tail=50
|
|
```
|
|
|
|
### Check Plugin Logs
|
|
```bash
|
|
kubectl logs -n ceph-csi-rbd -l app=csi-rbdplugin -c csi-rbdplugin --tail=50
|
|
```
|
|
|
|
### Restart Provisioner
|
|
```bash
|
|
kubectl delete pods -n ceph-csi-rbd -l app=csi-rbdplugin-provisioner
|
|
```
|
|
|
|
### Check Ceph Health
|
|
```bash
|
|
ssh root@10.0.19.101 "ceph health detail"
|
|
```
|
|
|
|
## Files
|
|
|
|
- `00-namespace.yaml` - Namespace definition
|
|
- `01-configmap.yaml` - Ceph cluster and encryption configuration
|
|
- `02-secret.yaml` - Ceph client credentials
|
|
- `03-rbac-provisioner.yaml` - RBAC for provisioner
|
|
- `04-rbac-plugin.yaml` - RBAC for node plugin
|
|
- `05-provisioner.yaml` - Provisioner deployment
|
|
- `06-plugin.yaml` - Plugin DaemonSet (with control plane tolerations)
|
|
- `07-csidriver.yaml` - CSI driver registration
|
|
- `08-storageclass.yaml` - StorageClass definition
|
|
- `deploy.sh` - Deployment script
|
|
- `test-pvc.yaml` - Test PVC and pod
|
|
|
|
## Alternative: Longhorn
|
|
|
|
If Ceph CSI continues to have issues, consider deploying Longhorn instead:
|
|
|
|
```bash
|
|
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn.yaml
|
|
```
|
|
|
|
Longhorn is simpler to set up and doesn't require external Ceph storage.
|