操作系統(tǒng)	k8s	mysql	minio	juicefs	內(nèi)核
centos8.2	1.19.1	8.0.39	RELEASE.2023-12-20T01-00-02Z	v0.19.0	4.18.0-193.el8.x86_64

本文k8s較老采用老版本的juicefs，中間件也都是部署在k8s上。測(cè)試是否能成功創(chuàng)建動(dòng)態(tài)pvc掛在到測(cè)試pod當(dāng)中并查看到數(shù)據(jù)信息。一些偏理論知識(shí)就不多說了，直接開始操作

官方演示視頻：??????JuiceFS CSI Driver 簡介 - Office Hours 20220629_嗶哩嗶哩_bilibili?

官方文檔：安裝 | JuiceFS Document Center

?k8s部署mysql-CSDN博客

k8s部署minio集群-CSDN博客

參考以上兩篇文章將中間件依賴環(huán)境先搭建起來

準(zhǔn)備juicefs部署文件

k8s.yaml（juicefs）

# DO NOT EDIT: generated by 'kustomize build'
apiVersion: v1
kind: ServiceAccount
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-controller-sanamespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-node-sanamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-external-node-service-role
rules:
- apiGroups:- ""resources:- podsverbs:- get- list- create- update- delete- patch- watch
- apiGroups:- ""resources:- pods/logverbs:- get
- apiGroups:- ""resources:- secretsverbs:- get- create- update- delete- patch
- apiGroups:- batchresources:- jobsverbs:- get- create- update- delete- patch
- apiGroups:- ""resources:- nodes/proxyverbs:- '*'
- apiGroups:- ""resources:- persistentvolumesverbs:- get- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-external-provisioner-role
rules:
- apiGroups:- ""resources:- persistentvolumesverbs:- get- list- watch- create- delete
- apiGroups:- ""resources:- persistentvolumeclaimsverbs:- get- list- watch- update
- apiGroups:- storage.k8s.ioresources:- storageclassesverbs:- get- list- watch
- apiGroups:- ""resources:- eventsverbs:- list- watch- create- update- patch
- apiGroups:- storage.k8s.ioresources:- csinodesverbs:- get- list- watch
- apiGroups:- ""resources:- nodesverbs:- get- list- watch
- apiGroups:- ""resources:- secretsverbs:- get- create- update- patch- delete
- apiGroups:- ""resources:- pods- pods/logverbs:- get- list- watch- create- update- patch- delete
- apiGroups:- batchresources:- jobsverbs:- get- create- update- patch- delete
- apiGroups:- ""resources:- endpointsverbs:- get- list- watch- create- update- patch
- apiGroups:- appsresources:- daemonsetsverbs:- get- list
- apiGroups:- coordination.k8s.ioresources:- leasesverbs:- get- watch- list- delete- update- create
- apiGroups:- ""resources:- configmapsverbs:- get- watch- list- delete- update- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-node-service-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: juicefs-csi-external-node-service-role
subjects:
- kind: ServiceAccountname: juicefs-csi-node-sanamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-provisioner-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: juicefs-external-provisioner-role
subjects:
- kind: ServiceAccountname: juicefs-csi-controller-sanamespace: kube-system
---
apiVersion: apps/v1
kind: StatefulSet
metadata:labels:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-controllernamespace: kube-system
spec:replicas: 1selector:matchLabels:app: juicefs-csi-controllerapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: masterserviceName: juicefs-csi-controllertemplate:metadata:labels:app: juicefs-csi-controllerapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: masterspec:containers:- args:- --endpoint=$(CSI_ENDPOINT)- --logtostderr- --nodeid=$(NODE_NAME)- --leader-election- --v=5env:- name: CSI_ENDPOINTvalue: unix:///var/lib/csi/sockets/pluginproxy/csi.sock- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName- name: JUICEFS_MOUNT_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: JUICEFS_MOUNT_PATHvalue: /var/lib/juicefs/volume- name: JUICEFS_CONFIG_PATHvalue: /var/lib/juicefs/configimage: juicedata/juicefs-csi-driver:v0.19.0livenessProbe:failureThreshold: 5httpGet:path: /healthzport: healthzinitialDelaySeconds: 10periodSeconds: 10timeoutSeconds: 3name: juicefs-pluginports:- containerPort: 9909name: healthzprotocol: TCPresources:limits:cpu: 1000mmemory: 1Girequests:cpu: 100mmemory: 512MisecurityContext:capabilities:add:- SYS_ADMINprivileged: truevolumeMounts:- mountPath: /var/lib/csi/sockets/pluginproxy/name: socket-dir- mountPath: /jfsmountPropagation: Bidirectionalname: jfs-dir- mountPath: /root/.juicefsmountPropagation: Bidirectionalname: jfs-root-dir- args:- --csi-address=$(ADDRESS)- --timeout=60s- --enable-leader-election- --v=5env:- name: ADDRESSvalue: /var/lib/csi/sockets/pluginproxy/csi.sockimage: quay.io/k8scsi/csi-provisioner:v1.6.0name: csi-provisionervolumeMounts:- mountPath: /var/lib/csi/sockets/pluginproxy/name: socket-dir- args:- --csi-address=$(ADDRESS)- --health-port=$(HEALTH_PORT)env:- name: ADDRESSvalue: /csi/csi.sock- name: HEALTH_PORTvalue: "9909"image: quay.io/k8scsi/livenessprobe:v1.1.0name: liveness-probevolumeMounts:- mountPath: /csiname: socket-dirpriorityClassName: system-cluster-criticalserviceAccount: juicefs-csi-controller-satolerations:- key: CriticalAddonsOnlyoperator: Existsvolumes:- emptyDir: {}name: socket-dir- hostPath:path: /var/lib/juicefs/volumetype: DirectoryOrCreatename: jfs-dir- hostPath:path: /var/lib/juicefs/configtype: DirectoryOrCreatename: jfs-root-dirvolumeClaimTemplates: []
---
apiVersion: apps/v1
kind: DaemonSet
metadata:labels:app.kubernetes.io/component: nodeapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-nodenamespace: kube-system
spec:selector:matchLabels:app: juicefs-csi-nodeapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastertemplate:metadata:labels:app: juicefs-csi-nodeapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: masterspec:containers:- args:- --endpoint=$(CSI_ENDPOINT)- --logtostderr- --nodeid=$(NODE_NAME)- --v=5- --enable-manager=trueenv:- name: CSI_ENDPOINTvalue: unix:/csi/csi.sock- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName- name: JUICEFS_MOUNT_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: HOST_IPvalueFrom:fieldRef:fieldPath: status.hostIP- name: KUBELET_PORTvalue: "10250"- name: JUICEFS_MOUNT_PATHvalue: /var/lib/juicefs/volume- name: JUICEFS_CONFIG_PATHvalue: /var/lib/juicefs/configimage: juicedata/juicefs-csi-driver:v0.19.0lifecycle:preStop:exec:command:- /bin/sh- -c- rm /csi/csi.socklivenessProbe:failureThreshold: 5httpGet:path: /healthzport: healthzinitialDelaySeconds: 10periodSeconds: 10timeoutSeconds: 3name: juicefs-pluginports:- containerPort: 9909name: healthzprotocol: TCPresources:limits:cpu: 1000mmemory: 1Girequests:cpu: 100mmemory: 512MisecurityContext:privileged: truevolumeMounts:- mountPath: /var/lib/kubeletmountPropagation: Bidirectionalname: kubelet-dir- mountPath: /csiname: plugin-dir- mountPath: /devname: device-dir- mountPath: /jfsmountPropagation: Bidirectionalname: jfs-dir- mountPath: /root/.juicefsmountPropagation: Bidirectionalname: jfs-root-dir- args:- --csi-address=$(ADDRESS)- --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)- --v=5env:- name: ADDRESSvalue: /csi/csi.sock- name: DRIVER_REG_SOCK_PATHvalue: /var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sockimage: quay.io/k8scsi/csi-node-driver-registrar:v2.1.0name: node-driver-registrarvolumeMounts:- mountPath: /csiname: plugin-dir- mountPath: /registrationname: registration-dir- args:- --csi-address=$(ADDRESS)- --health-port=$(HEALTH_PORT)env:- name: ADDRESSvalue: /csi/csi.sock- name: HEALTH_PORTvalue: "9909"image: quay.io/k8scsi/livenessprobe:v1.1.0name: liveness-probevolumeMounts:- mountPath: /csiname: plugin-dirdnsPolicy: ClusterFirstWithHostNetpriorityClassName: system-node-criticalserviceAccount: juicefs-csi-node-satolerations:- key: CriticalAddonsOnlyoperator: Existsvolumes:- hostPath:path: /var/lib/kubelettype: Directoryname: kubelet-dir- hostPath:path: /var/lib/kubelet/csi-plugins/csi.juicefs.com/type: DirectoryOrCreatename: plugin-dir- hostPath:path: /var/lib/kubelet/plugins_registry/type: Directoryname: registration-dir- hostPath:path: /devtype: Directoryname: device-dir- hostPath:path: /var/lib/juicefs/volumetype: DirectoryOrCreatename: jfs-dir- hostPath:path: /var/lib/juicefs/configtype: DirectoryOrCreatename: jfs-root-dir
---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: csi.juicefs.com
spec:attachRequired: falsepodInfoOnMount: false

secret.yaml（存儲(chǔ)數(shù)據(jù)秘鑰信息）

apiVersion: v1
kind: Secret
metadata:name: juicefs-secretnamespace: defaultlabels:# 增加該標(biāo)簽以啟用認(rèn)證信息校驗(yàn)juicefs.com/validate-secret: "true"
type: Opaque
stringData:name: data metaurl: mysql://root:PR38Ke2sP@(mysql.juicefs.svc.cluster.local:3306)/juicefs storage: minio bucket: http://minio-service.juicefs.svc.cluster.local:9000/juicefsaccess-key: adminsecret-key: 0oO1iIlLq9g#MINIO_ACCESS_KEY: admin #MINIO_SECRET_KEY: 0oO1iIlLq9g 

#這里minio用來做數(shù)據(jù)存儲(chǔ)，需要注意的是9000/juicefs這里是你的桶實(shí)現(xiàn)創(chuàng)建好，name: data會(huì)自行拼接到j(luò)uicefs后面。也需要提前創(chuàng)建data出來

?storageclasses.yaml（根據(jù)秘鑰信息創(chuàng)建的sc）

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: juicefs-sc
provisioner: csi.juicefs.com
parameters:csi.storage.k8s.io/provisioner-secret-name: juicefs-secretcsi.storage.k8s.io/provisioner-secret-namespace: defaultcsi.storage.k8s.io/node-publish-secret-name: juicefs-secretcsi.storage.k8s.io/node-publish-secret-namespace: default
reclaimPolicy: Retain

pvc-test.yaml（測(cè)試pvc掛載pod）

apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: juicefs-pvc#namespace: juicefs 
spec:accessModes:- ReadWriteManyresources:requests:# 從 StorageClass 中申請(qǐng) 10GiB 存儲(chǔ)容量storage: 3GistorageClassName: juicefs-sc
---
apiVersion: v1
kind: Pod
metadata:name: busybox#namespace: juicefs
spec:containers:- name: busyboximage: gcr.io/google-containers/busybox:latestimagePullPolicy: IfNotPresentcommand: ["sh", "-c", "tail -f /dev/null"]tty: true  # 啟用TTY以便可以連接到容器volumeMounts:  # 確保此行與 'containers' 下的其他字段對(duì)齊- mountPath: /optname: juicefs-pvmountPropagation: HostToContainerrestartPolicy: Never  # 設(shè)置為 Never 以便測(cè)試完成后不會(huì)自動(dòng)重啟volumes:  # 確保此行與 'containers' 對(duì)齊- name: juicefs-pvpersistentVolumeClaim:claimName: juicefs-pvc

測(cè)試步驟?

apply k8s.yaml 后會(huì)得到以下pod

當(dāng)juicefs-csi相關(guān)pod成功拉起以后需要生成基于mysql與minio的secret。這個(gè)secret是用來給storageclasses使用的，讓它能夠調(diào)用csi去mount文件創(chuàng)建掛載點(diǎn)。

以上配置創(chuàng)建好以后apply pvc-test來測(cè)試動(dòng)態(tài)創(chuàng)建pvc效果，發(fā)現(xiàn)會(huì)啟用一個(gè)這種pod。是一個(gè)掛載點(diǎn)pod。

發(fā)現(xiàn)測(cè)試pod也成功創(chuàng)建了pvc也創(chuàng)建了并且綁定了busyboxpod

感興趣可以看一下minio跟mysql數(shù)據(jù)存儲(chǔ)的狀態(tài)。這里就不截圖演示了。

官方文檔給的樣例模板：創(chuàng)建和使用 PV | JuiceFS Document Center

#juicefs相關(guān)鏡像
juicedata/juicefs-csi-driver:v0.19.0
quay.io/k8scsi/csi-node-driver-registrar:v2.1.0
quay.io/k8scsi/csi-provisioner:v1.6.0
quay.io/k8scsi/livenessprobe:v1.1.0
juicedata/mount:ce-v1.0.4