Skip to content

📚 Complete Ceph CSI Deployment & Troubleshooting Guide

This guide details the preparation and configuration necessary for successful dynamic provisioning of Ceph RBD (RWO) and CephFS (RWX) volumes in a Kubernetes cluster running on MicroK8s, backed by a Proxmox VE (PVE) Ceph cluster.

1. ⚙️ Ceph Cluster Preparation (Proxmox VE)

These steps ensure the Ceph backend has the necessary pools and structure.

  • Create Dedicated Pools: Create OSD Pools for data, e.g., k8s_rbd (for RWO), and k8s_data and k8s_metadata (for CephFS).
  • Create CephFS Metadata Servers (MDS): Deploy at least two Metadata Server (MDS) instances.
  • Create CephFS Filesystem: Create the Ceph Filesystem (e.g., named k8s), linking the metadata and data pools.
  • Create Subvolume Group (Mandatory Fix): Create the dedicated Subvolume Group csi inside your CephFS. This is required by the CSI driver's default configuration and fixes the "No such file or directory" error during provisioning.
    • CLI Command: ceph fs subvolumegroup create k8s csi

2. 🔑 Ceph User and Authorization (The Permission Fix)

This addresses the persistent "Permission denied" errors during provisioning.

  • Create and Configure Ceph User: Create the user (client.kubernetes) and set permissions for all services. The wildcard MGR cap (mgr "allow *") is critical for volume creation.
    • Final Correct Caps Command: bash sudo ceph auth caps client.kubernetes \ mon 'allow r' \ mgr "allow *" \ mds 'allow rw' \ osd 'allow class-read object_prefix rbd_children, allow pool k8s_rbd rwx, allow pool k8s_metadata rwx, allow pool k8s_data rwx'
  • Export Key to Kubernetes Secrets: Create and place two Secrets with the user key in the correct CSI provisioner namespaces:
    • RBD Secret: csi-rbd-secret (in the RBD Provisioner namespace).
    • CephFS Secret: csi-cephfs-secret (in the CephFS Provisioner namespace).

The secrets should contain the keys: userID & userKey. userID should omit the 'client.' from the ceph output.

3. 🌐 Network Configuration and Bi-Directional Routing

These steps ensure stable, bidirectional communication for volume staging and mounting.

A. PVE Host Firewall Configuration

The PVE firewall must explicitly allow inbound traffic from the entire Kubernetes Pod Network to the Ceph service ports.

Protocol Port(s) Source Purpose
TCP 6789 K8s Pod Network CIDR (e.g., 10.1.0.0/16) Monitor connection.
TCP 6800-7300 K8s Pod Network CIDR OSD/MDS/MGR data transfer.

Alternatively, you may find a 'ceph' macro you can use on the PVE Firewall, if so use the Macro instead of additional rules.

B. PVE Host Static Routing (Ceph > K8s)

Add persistent static routes on all PVE Ceph hosts to allow Ceph to send responses back to the Pod Network.

  • Action: Edit /etc/network/interfaces on each PVE host: ini # Example: post-up ip route add <POD_NETWORK_CIDR> via <K8S_NODE_IP> dev <PVE_INTERFACE> # e.g., post-up ip route add 10.1.0.0/16 via 172.35.100.40 dev vmbr0

C. K8s Node IP Forwarding (Gateway Function)

Enable IP forwarding on all Kubernetes nodes so they can route incoming Ceph traffic to the correct Pods.

  • Action: Run on all K8s nodes: bash sudo sysctl net.ipv4.ip_forward=1 sudo sh -c 'echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/99-sysctl.conf'

D. K8s Static Routing (K8s > Ceph) - Conditional/Advanced ⚠️

This routing is only required if the Ceph Public Network (the network Ceph Monitors/OSDs listen on) is not reachable by your Kubernetes Node's default gateway.

  • Action: This is implemented via a Netplan configuration on the Kubernetes nodes, using multiple routes with different metrics to provide load balancing and automatic failover.
  • Example Netplan Configuration (/etc/netplan/99-ceph-routes.yaml): ```yaml network: version: 2 renderer: networkd ethernets: eth0: # Replace with your primary K8s network interface routes: # Route 1: Directs traffic destined for the first Ceph Monitor IP (10.11.12.1) # through three different PVE hosts (172.35.100.x) as gateways. # The lowest metric (10) is preferred. - to: 10.11.12.1/32 via: 172.35.100.10 metric: 10 - to: 10.11.12.1/32 via: 172.35.100.20 metric: 100 - to: 10.11.12.1/32 via: 172.35.100.30 metric: 100
      # Route 2: Directs traffic destined for the second Ceph Monitor IP (10.11.12.2)
  # with a similar failover strategy.
  - to: 10.11.12.2/32
    via: 172.35.100.20
    metric: 10
  - to: 10.11.12.2/32
    via: 172.35.100.10
    metric: 100
  - to: 10.11.12.2/32
    via: 172.35.100.30
    metric: 100

    ```

Use route priorities (Lower is Higher) to prefer the most direct path, while still offering alternative gateways into the Ceph network where needed.

4. 🧩 MicroK8s CSI Driver Configuration (The Path Fix) - Conditional/Advanced ⚠️

This adjustment is only required if MicroK8s is running your Kuberneted deployment. Alternative changes may be needed for other Kubetnetes distributions.

This resolves the staging path does not exist on node error for the Node Plugin.

  • Update kubeletDir: When deploying the CSI driver (via Helm or YAML), the kubeletDir parameter must be set to the MicroK8s-specific path. yaml # Correct path for MicroK8s Kubelet root directory kubeletDir: /var/snap/microk8s/common/var/lib/kubelet