kubernetes 集群三步安装
安装
git clone https://github.com/rook/rook
cd cluster/examples/kubernetes/ceph
kubectl create -f operator.yaml
查看 operator 是否成功：
[root@dev-86-201 ~]# kubectl get pod -n rook-ceph-system
NAME READY STATUS RESTARTS AGE
rook-ceph-agent-5z6p7 1/1 Running 0 88m
rook-ceph-agent-6rj7l 1/1 Running 0 88m
rook-ceph-agent-8qfpj 1/1 Running 0 88m
rook-ceph-agent-xbhzh 1/1 Running 0 88m
rook-ceph-operator-67f4b8f67d-tsnf2 1/1 Running 0 88m
rook-discover-5wghx 1/1 Running 0 88m
rook-discover-lhwvf 1/1 Running 0 88m
rook-discover-nl5m2 1/1 Running 0 88m
rook-discover-qmbx7 1/1 Running 0 88m
然后创建 ceph 集群：
kubectl create -f cluster.yaml
查看 ceph 集群：
[root@dev-86-201 ~]# kubectl get pod -n rook-ceph
NAME READY STATUS RESTARTS AGE
rook-ceph-mgr-a-8649f78d9b-jklbv 1/1 Running 0 64m
rook-ceph-mon-a-5d7fcfb6ff-2wq9l 1/1 Running 0 81m
rook-ceph-mon-b-7cfcd567d8-lkqff 1/1 Running 0 80m
rook-ceph-mon-d-65cd79df44-66rgz 1/1 Running 0 79m
rook-ceph-osd-0-56bd7545bd-5k9xk 1/1 Running 0 63m
rook-ceph-osd-1-77f56cd549-7rm4l 1/1 Running 0 63m
rook-ceph-osd-2-6cf58ddb6f-wkwp6 1/1 Running 0 63m
rook-ceph-osd-3-6f8b78c647-8xjzv 1/1 Running 0 63m
参数说明：
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
# For the latest ceph images, see https://hub.docker.com/r/ceph/ceph/tags
image: ceph/ceph:v13.2.2-20181023
dataDirHostPath: /var/lib/rook # 数据盘目录
mon:
count: 3
allowMultiplePerNode: true
dashboard:
enabled: true
storage:
useAllNodes: true
useAllDevices: false
config:
databaseSizeMB: “1024”
journalSizeMB: “1024”
访问 ceph dashboard:
[root@dev-86-201 ~]# kubectl get svc -n rook-ceph
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mgr ClusterIP 10.98.183.33 <none> 9283/TCP 66m
rook-ceph-mgr-dashboard NodePort 10.103.84.48 <none> 8443:31631/TCP 66m # 把这个改成 NodePort 模式
rook-ceph-mon-a ClusterIP 10.99.71.227 <none> 6790/TCP 83m
rook-ceph-mon-b ClusterIP 10.110.245.119 <none> 6790/TCP 82m
rook-ceph-mon-d ClusterIP 10.101.79.159 <none> 6790/TCP 81m
然后访问 https://10.1.86.201:31631 即可
管理账户 admin, 获取登录密码：
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o yaml | grep “password:” | awk ‘{print $2}’ | base64 –decode
使用
创建 pool
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: replicapool # operator 会监听并创建一个 pool，执行完后界面上也能看到对应的 pool
namespace: rook-ceph
spec:
failureDomain: host
replicated:
size: 3
—
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-block # 这里创建一个 storage class, 在 pvc 中指定这个 storage class 即可实现动态创建 PV
provisioner: ceph.rook.io/block
parameters:
blockPool: replicapool
# The value of “clusterNamespace” MUST be the same as the one in which your rook cluster exist
clusterNamespace: rook-ceph
# Specify the filesystem type of the volume. If not specified, it will use `ext4`.
fstype: xfs
# Optional, default reclaimPolicy is “Delete”. Other options are: “Retain”, “Recycle” as documented in https://kubernetes.io/docs/concepts/storage/storage-classes/
reclaimPolicy: Retain
创建 pvc
在 cluster/examples/kubernetes 目录下，官方给了个 worldpress 的例子，可以直接运行一下：
kubectl create -f mysql.yaml
kubectl create -f wordpress.yaml
查看 PV PVC：
[root@dev-86-201 ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-a910f8c2-1ee9-11e9-84fc-becbfc415cde 20Gi RWO rook-ceph-block 144m
wp-pv-claim Bound pvc-af2dfbd4-1ee9-11e9-84fc-becbfc415cde 20Gi RWO rook-ceph-block 144m
[root@dev-86-201 ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-a910f8c2-1ee9-11e9-84fc-becbfc415cde 20Gi RWO Retain Bound default/mysql-pv-claim rook-ceph-block 145m
pvc-af2dfbd4-1ee9-11e9-84fc-becbfc415cde 20Gi RWO Retain Bound default/wp-pv-claim rook-ceph-block 145m
看下 yaml 文件：
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: wordpress
spec:
storageClassName: rook-ceph-block # 指定 storage class
accessModes:
– ReadWriteOnce
resources:
requests:
storage: 20Gi # 需要一个 20G 的盘
…
volumeMounts:
– name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
– name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim # 指定上面定义的 PVC
是不是非常简单。
要访问 wordpress 的话请把 service 改成 NodePort 类型，官方给的是 loadbalance 类型：
kubectl edit svc wordpress
[root@dev-86-201 kubernetes]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
wordpress NodePort 10.109.30.99 <none> 80:30130/TCP 148m
总结
分布式存储在容器集群中充当非常重要的角色，使用容器集群一个非常重要的理念就是把集群当成一个整体使用，如果你在使用中还关心单个主机，比如调度到某个节点，
挂载某个节点目录等，必然会导致不能把云的威力百分之百发挥出来。一旦计算存储分离后，就可真正实现随意漂移，对集群维护来说是个极大的福音。
比如集群机器过保了需要下架，那么我们云化的架构因为所有东西无单点，所以只需要简单驱逐改节点，然后下架即可，不用关心上面跑的是什么业务，不管是有状态还是无
状态的都可以自动修复。不过目前面临最大的挑战可能还是分布式存储的性能问题。在性能要求不苛刻的场景下我是极推荐这种计算存储分离架构的。
探讨可加 QQ 群：98488045
公众号：
微信群：