kubernetes存储之ceph-csi

scofield 菜鸟运维杂谈

因为kubernetes默认的provisioner: kubernetes.io/rbd 无法正常使用，改用ceph官网提供的provisioner: rbd.csi.ceph.com，需要部署相关的csi插件以及配置ceph连接授权，所以本文记录如下。

git clone https://github.com/ceph/ceph-csi.gitcd ceph-csi/deploy/rbd/kubernetes[root@qd01-stop-k8s-master001 kubernetes]# ls -ltotal 36-rw-r--r-- 1 root root  304 Feb 23 16:24 csi-config-map.yaml-rw-r--r-- 1 root root 1674 Feb 23 16:20 csi-nodeplugin-psp.yaml-rw-r--r-- 1 root root  747 Feb 23 16:20 csi-nodeplugin-rbac.yaml-rw-r--r-- 1 root root 1300 Feb 23 16:20 csi-provisioner-psp.yaml-rw-r--r-- 1 root root 2915 Feb 23 16:20 csi-provisioner-rbac.yaml-rw-r--r-- 1 root root 7123 Feb 23 16:34 csi-rbdplugin-provisioner.yaml-rw-r--r-- 1 root root 5841 Feb 23 16:34 csi-rbdplugin.yaml

如下是需要的镜像

k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.2k8s.gcr.io/sig-storage/csi-attacher:v3.0.2k8s.gcr.io/sig-storage/csi-resizer:v1.0.1k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1quay.io/cephcsi/cephcsi:canary如果您的网络不能从k8s.gcr.io下载镜像，可以改成如下scofield/csi-provisioner:v2.0.4scofield/csi-snapshotter:v3.0.2scofield/csi-attacher:v3.0.2scofield/csi-resizer:v1.0.1scofield/csi-node-driver-registrar:v2.0.1scofield/cephcsi:canary

将kms所在行注释掉，因为没有这个配置，否则会部署失败

root@qd01-stop-k8s-master001 kubernetes]# vim csi-rbdplugin-provisioner.yaml[root@qd01-stop-k8s-master001 kubernetes]# vim csi-rbdplugin.yaml        #- name: ceph-csi-encryption-kms-config        #  mountPath: /etc/ceph-csi-encryption-kms-config/        #- name: ceph-csi-encryption-kms-config        #  configMap:        #    name: ceph-csi-encryption-kms-config

[root@qd01-stop-k8s-master001 kubernetes]# kubectl apply -f . -n csiconfigmap/ceph-csi-config createdpodsecuritypolicy.policy/rbd-csi-nodeplugin-psp createdrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin-psp createdrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-psp createdserviceaccount/rbd-csi-nodeplugin createdclusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin createdclusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin createdpodsecuritypolicy.policy/rbd-csi-provisioner-psp createdrole.rbac.authorization.k8s.io/rbd-csi-provisioner-psp createdrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-psp createdserviceaccount/rbd-csi-provisioner createdclusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner createdclusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role createdrole.rbac.authorization.k8s.io/rbd-external-provisioner-cfg createdrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg createdservice/csi-rbdplugin-provisioner createddeployment.apps/csi-rbdplugin-provisioner createddaemonset.apps/csi-rbdplugin createdservice/csi-metrics-rbdplugin created等待部署完成即可[root@qd01-stop-k8s-master001 UseRBD]# kubectl get po -n csiNAME                                        READY   STATUS    RESTARTS   AGEcsi-rbdplugin-5xtbz                         3/3     Running   0          29mcsi-rbdplugin-hwrsr                         3/3     Running   0          29mcsi-rbdplugin-mtscj                         3/3     Running   0          29mcsi-rbdplugin-pmqjv                         3/3     Running   0          29mcsi-rbdplugin-provisioner-b96dc4989-fd7kt   7/7     Running   0          29mcsi-rbdplugin-provisioner-b96dc4989-tk9bv   7/7     Running   0          29mcsi-rbdplugin-provisioner-b96dc4989-xrxgz   7/7     Running   0          29mcsi-rbdplugin-qzsjr                         3/3     Running   0          29mcsi-rbdplugin-tt4b9                         3/3     Running   0          29mcsi-rbdplugin-w429q                         3/3     Running   0          29mcsi-rbdplugin-w6xp7                         3/3     Running   0          29mcsi-rbdplugin-wxc94                         3/3     Running   0          29m

1、创建需要的secret
创建csi-rbd-secret.yaml

---apiVersion: v1kind: Secretmetadata:  name: csi-rbd-secret  namespace: csistringData:  userID: admin  userKey: AQALpatf81ZmNhAAz6xt03v4boTYj7o5MOa0iQ==[root@qd01-stop-k8s-master001 UseRBD]# kubectl apply -f csi-rbd-secret.yamlsecret/csi-rbd-secret created

2、创建存储类sc
创建storageclass.yaml

---apiVersion: storage.k8s.io/v1kind: StorageClassmetadata:   name: rbdprovisioner: rbd.csi.ceph.comparameters:   clusterID: ec7ee19a-f7c6-4ed0-93a7-f48af473352c   pool: k8s   imageFeatures: layering   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret   csi.storage.k8s.io/provisioner-secret-namespace: csi   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret   csi.storage.k8s.io/controller-expand-secret-namespace: csi   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret   csi.storage.k8s.io/node-stage-secret-namespace: csi   csi.storage.k8s.io/fstype: xfsreclaimPolicy: DeleteallowVolumeExpansion: truemountOptions:   - discard[root@qd01-stop-k8s-master001 UseRBD]# kubectl apply -f storageclass.yamlstorageclass.storage.k8s.io/rbd created[root@qd01-stop-k8s-master001 UseRBD]# kubectl get scNAME   PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGErbd    rbd.csi.ceph.com   Delete          Immediate           true                   20m

3、创建PVC验证sc是否可用
创建raw-block-pvc.yaml

---apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: raw-block-pvcspec:  accessModes:    - ReadWriteOnce  volumeMode: Block  resources:    requests:      storage: 1Gi  storageClassName: rbd[root@qd01-stop-k8s-master001 UseRBD]# kubectl apply -f raw-block-pvc.yamlpersistentvolumeclaim/raw-block-pvc created[root@qd01-stop-k8s-master001 UseRBD]# kubectl get pvcNAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGEraw-block-pvc   Bound    pvc-84bf2ffb-7aee-41bd-9e6d-614c9f29eab4   1Gi        RWO            rbd            39s

4、测试动态分配PVC
创建demo-statefulset-csi.yaml

apiVersion: apps/v1kind: StatefulSetmetadata:  name: demo-nginx  namespace: default  labels:    app: demo-nginxspec:  serviceName: demo-nginx  replicas: 2  selector:    matchLabels:      app: demo-nginx  template:    metadata:      labels:        app: demo-nginx    spec:      terminationGracePeriodSeconds: 180      initContainers:        - name: init          image: busybox          command: ["chmod","777","-R","/data"]          imagePullPolicy: Always          volumeMounts:          - name: volume            mountPath: /data      containers:      - name: demo-nginx        image: nginx        ports:        - containerPort: 80          name: port        volumeMounts:        - name: volume          mountPath: /data  volumeClaimTemplates:  - metadata:      name: volume    spec:      accessModes: ["ReadWriteOnce"]      storageClassName: rbd      resources:        requests:          storage: 5Gi[root@qd01-stop-k8s-master001 UseRBD]# kubectl apply -f demo-statefulset-csi.yamlstatefulset.apps/demo-nginx created

5、验证可以看到,pvs自动创建好，并且正常挂载到pod中

[root@qd01-stop-k8s-master001 UseRBD]# kubectl get pvcNAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGEvolume-demo-nginx-0   Bound    pvc-b0e3c919-10ad-49f7-a225-4337c07133ea   5Gi        RWO            rbd            6m37svolume-demo-nginx-1   Bound    pvc-cb526baa-62ae-43ee-a544-eb1655c9c8c6   5Gi        RWO            rbd            2m24s[root@qd01-stop-k8s-master001 UseRBD]# kubectl get po NAME           READY   STATUS    RESTARTS   AGEdemo-nginx-0   1/1     Running   0          5m5sdemo-nginx-1   1/1     Running   0          2m31s进入到其中一个pod中可以看到，挂载了一个/dev/rbd2块存储，大小为制定的5G[root@qd01-stop-k8s-master001 UseRBD]# kubectl exec -ti demo-nginx-0 /bin/shkubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.# df -hFilesystem      Size  Used Avail Use% Mounted on/dev/rbd2       5.0G   38M  5.0G   1% /data

PS：文章会同步到dev.kubeops.net