乐趣区

关于kubernetes:Kubernetes-NFS作为动态存储创建pvcpv

jiezi

NFS 因为本身的问题，不罕用于生产环境，这里仅作为 demo 展现动静存储的应用。生产环境能够应用 ceph，rook-ceph 来治理 ceph 存储。

假如已部署好 NFS Server，这里演示如何在集群中部署动静存储与创立 storageclass/pvc/pv。

1. 创立独立的 namespace

# kubectl create ns storage

2. 创立 rbac 给 serviceAccount 赋权

创立一个 serviceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  namespace: storage

为 serviceAccount 赋权：

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: storage
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

3. 部署 Provisioner

privisioner 能够了解为底层存储的驱动，由 privisioner 治理底层存储。
privisioner 以 deploy 形式部署了 1 个 pod，pod 内 container 指定了 nfs 的环境信息 (包含 name/ip/path 等)，serviceAccountName= 上一步创立的 serviceAccount 名称；

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nfs-provisioner
  name: nfs-provisioner
  namespace: storage
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-provisioner
  template:
    metadata:
      labels:
        app: nfs-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-volume
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 178.104.163.63
            - name: NFS_PATH
              value: /var/nfs
      volumes:
        - name: nfs-volume
          nfs:
            server: 178.104.163.63
            path: /var/nfs

查看部署的 pod:

# kubectl get pod -n storage
NAME                               READY   STATUS    RESTARTS   AGE
nfs-provisioner-778c655cbd-4twcz   1/1     Running   0          19s

provisioner 部署结束后，察看 pod 的 log；若有报错，需及时排查：

# kubectl logs nfs-provisioner-778c655cbd-4twcz -n storage
I0330 08:08:27.981652       1 leaderelection.go:185] attempting to acquire leader lease  storage/fuseim.pri-ifs...
I0330 08:08:27.992969       1 leaderelection.go:194] successfully acquired lease storage/fuseim.pri-ifs
I0330 08:08:27.993028       1 controller.go:631] Starting provisioner controller fuseim.pri/ifs_nfs-provisioner-778c655cbd-4twcz_1bbd49e5-912f-11eb-a36b-fef168fb5776!
I0330 08:08:27.993504       1 event.go:221] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"storage", Name:"fuseim.pri-ifs", UID:"35414cb4-5e1c-45e4-9d0e-9b031e0c3df2", APIVersion:"v1", ResourceVersion:"2323405", FieldPath:""}): type:'Normal'reason:'LeaderElection' nfs-provisioner-778c655cbd-4twcz_1bbd49e5-912f-11eb-a36b-fef168fb5776 became leader
I0330 08:08:28.093343       1 controller.go:680] Started provisioner controller fuseim.pri/ifs_nfs-provisioner-778c655cbd-4twcz_1bbd49e5-912f-11eb-a36b-fef168fb5776!

4. 创立 storageclass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-sc
  namespace: storage
provisioner: fuseim.pri/ifs    # 这里的 provisioner== 下面 env.PROVISIONER_NAME

查看 storageclass:

# kubectl get sc -n storage
NAME     PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-sc   fuseim.pri/ifs   Delete          Immediate           false                  19s

5.storageclass 创立 pvc 给 pod 应用

创立 pvc：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
  namespace: storage
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs-sc    // 应用 storageClass

查看主动创立的 pv:

# kubectl get sc,pv,pvc -n storage
NAME                                 PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/nfs-sc   fuseim.pri/ifs   Delete          Immediate           false                  13m

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   REASON   AGE
persistentvolume/pvc-2fa56db1-fe02-4722-a9e9-d0dfad565934   1Gi        RWX            Delete           Bound    storage/myclaim   nfs-sc                  3m12s

NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/myclaim   Bound    pvc-2fa56db1-fe02-4722-a9e9-d0dfad565934   1Gi        RWX            nfs-sc         3m12s

将 pvc 提供给 pod 应用：

kind: Pod
apiVersion: v1
metadata:
  name: test-pod
  namespace: storage
spec:
  containers:
  - name: test-pod
    image: nginx:1.15.2
    volumeMounts:
      - name: nfs-pvc
        mountPath: /mnt/nginx
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: myclaim        ## claimName== 下面创立的 pvc 的名称

pod 创立结束，到容器中查看挂载的目录：

# kubectl exec -it test-pod -n storage -- bash
# df -h
Filesystem                                                                        Size  Used Avail Use% Mounted on
overlay                                                                           100G  6.2G   94G   7% /
tmpfs                                                                              64M     0   64M   0% /dev
tmpfs                                                                             3.0G     0  3.0G   0% /sys/fs/cgroup
/dev/vda1                                                                         100G  6.2G   94G   7% /etc/hosts
178.104.163.63:/var/nfs/storage-myclaim-pvc-2fa56db1-fe02-4722-a9e9-d0dfad565934  100G  1.4G   99G   2% /mnt/nginx

6.storageclass 为 statefulset 创立 pvc/pv

statefulset.spec 指定 storageClass，部署时会主动为其创立 pvc、pv：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: storage
spec:
  serviceName: "nginx"        # 等于 headless service 的名称
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.15.2
        imagePullPolicy: IfNotPresent
        name: nginx
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /mnt/nginx
          name: nfs-sc-volume
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
  volumeClaimTemplates:                ## 指定连贯到 sc 去申请 pvc
  - metadata:
      name: nfs-sc-volume
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        limits:
          storage: 2Gi
        requests:
          storage: 1Gi
      storageClassName: nfs-sc
      volumeMode: Filesystem

这里的 statefulset 创立了 2 个 pod，为每个 pod 都绑定了一个 pvc，2 个 pod 应用独立的存储卷:

# kubectl get sc,pvc,pv -n storage
NAME                                 PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/nfs-sc   fuseim.pri/ifs   Delete          Immediate           false                  4m58s

NAME                                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/nfs-sc-volume-nginx-0   Bound    pvc-63389a42-a00d-4b34-bd51-4542cebb42aa   1Gi        RWO            nfs-sc         75s
persistentvolumeclaim/nfs-sc-volume-nginx-1   Bound    pvc-e01d4f5c-004b-4618-94d6-9556612cd198   1Gi        RWO            nfs-sc         70s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS   REASON   AGE
persistentvolume/pvc-63389a42-a00d-4b34-bd51-4542cebb42aa   1Gi        RWO            Delete           Bound    storage/nfs-sc-volume-nginx-0   nfs-sc                  75s
persistentvolume/pvc-e01d4f5c-004b-4618-94d6-9556612cd198   1Gi        RWO            Delete           Bound    storage/nfs-sc-volume-nginx-1   nfs-sc                  70s
退出移动版