疾速部署Ceph分布式高可用集群
Ceph简介
Ceph是一个PB，EB级别的分布式存储系统，能够提供文件存储，对象存储、和块存储，它可靠性高，易扩大，治理简便，其中对象存储和块存储能够和其余云平台集成。一个Ceph集群中有Monitor节点、MDS节点（用于文件存储）、OSD守护过程。
Ceph根底概念
ceph-deploy
一个集群自动化部署工具，应用较久，成熟稳固，被很多自动化工具所集成，可用于生产部署；
cephadm
从Octopus开始提供的新集群部署工具，反对通过图形界面或者命令行界面增加节点，目前不倡议用于生产环境，有趣味能够尝试；
manual
手动部署，一步步部署Ceph集群，反对较多定制化和理解部署细节，装置难度较大，但能够清晰把握装置部署的细节。
admin-node：
须要一个装置治理节点，装置节点负责集群整体部署，这里咱们用CephNode01为admin-node和Ceph-Mon节点；
mon：
monitor节点，即是Ceph的监督治理节点，承当Ceph集群重要的治理工作，个别须要3或5个节点，此处部署简略的一个Monitor节点；
osd：
OSD即Object Storage Daemon，理论负责数据存储的节点，3个节点上别离有2块100G的磁盘充当OSD角色。
Ceph零碎初始化
配置主机信息
# 设置主机名#node1hostnamectl set-hostname node1#node2hostnamectl set-hostname node2#node3hostnamectl set-hostname node3# 写入hostscat >> /etc/hosts <<EOF192.168.1.156 node1192.168.1.157 node2192.168.1.159 node3EOFcat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.1.156 node1192.168.1.157 node2192.168.1.159 node3配置免密
# 配置免密 （二选一）ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'.Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa.Your public key has been saved in /root/.ssh/id_rsa.pub.The key fingerprint is:SHA256:nK3CqSGRBGZfrE5rncPEQ2eU/Gq6dttYMLIiesXHyO8 root@ceph-01The key's randomart image is:+---[RSA 3072]----+|.o ..o.. ||o.. .o = || ..+ o . || . + + . + || =o=+ooS . || ==*=+o. || .oo.+B .. ||. o..=.o+ ||.. ooEo.. |+----[SHA256]-----+# 将免密传输到各个主机上ssh-copy-id root@node1ssh-copy-id root@node2ssh-copy-id root@node3# 应用懒人形式配置免密 （二选一）yum install -y sshpassssh-keygen -f /root/.ssh/id_rsa -P ''export IP="node1 node2 node3"export SSHPASS=123123for HOST in $IP;do sshpass -e ssh-copy-id -o StrictHostKeyChecking=no $HOSTdone配置根底环境
# 敞开防火墙systemctl stop firewalldsystemctl disable firewalldRemoved /etc/systemd/system/multi-user.target.wants/firewalld.service.Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.# 敞开swapswapoff -ased -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab# 敞开selinuxsetenforce 0sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config配置YUM源
# 配置yum源sudo sed -e 's|^mirrorlist=|#mirrorlist=|g' \ -e 's|^#baseurl=http://mirror.centos.org/$contentdir|baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos|g' \ -i.bak \ /etc/yum.repos.d/CentOS-*.repo# 配置ceph源cat > /etc/yum.repos.d/ceph.repo <<EOF[noarch] name=Ceph noarch baseurl=https://mirrors.ustc.edu.cn/ceph/rpm-17.2.0/el8/noarch/ enabled=1 gpgcheck=0 [x86_64] name=Ceph x86_64 baseurl=https://mirrors.ustc.edu.cn/ceph/rpm-17.2.0/el8/x86_64/ enabled=1 gpgcheck=0EOF装置根底环境
# 更新yum源yum update -y# 装置工具包、python-setuptools肯定要装置、不然会报错的yum install -y chrony conntrack ipset jq iptables curl sysstat libseccomp wget socat git vim epel-release epel-next-release调整时区\间
# 配置零碎时区timedatectl set-timezone Asia/Shanghai# 配置时钟同步timedatectl status# 注：System clock synchronized: yes，示意时钟已同步；NTP service: active，示意开启了时钟同步服务# 写入硬件时钟# 将以后的 UTC 工夫写入硬件时钟timedatectl set-local-rtc 0# 重启依赖于零碎工夫的服务systemctl restart rsyslog systemctl restart crond杂项
# 敞开无关服务systemctl stop postfix && systemctl disable postfix# 重启rebootCeph零碎装置
初始化monitor节点
yum install ceph -y# 初始化monitor节点# 在node1节点生成uuid，并在所有节点导入uuid环境变量[root@node1 ~]# uuidgen8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb#node1export cephuid=8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb#node2export cephuid=8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb#node3export cephuid=8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb# 所有节点创立Ceph配置文件：cat > /etc/ceph/ceph.conf <<EOF[global]fsid = 8d2cfd33-9132-48a7-8c00-3ef10cb5ddebmon initial members = node1, node2, node3mon host = 192.168.1.156, 192.168.1.157, 192.168.1.159public network = 192.168.1.0/24auth cluster required = cephxauth service required = cephxauth client required = cephxosd journal size = 1024osd pool default size = 3osd pool default min size = 2osd pool default pg num = 333osd pool default pgp num = 333osd crush chooseleaf type = 1EOF# 以下操作在node1节点执行# 为集群创立一个keyring，并生成一个monitor密钥。#node1ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'# 生成administrator keyring，生成client.admin用户并将用户增加到keyring。#node1ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'# 生成bootstrap-osd keyring，生成client.bootstrap-osd用户并将用户增加到keyring。#node1ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'# 将生成的密钥增加到中ceph.mon.keyring。#node1ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyringceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring# 将所有者更改为ceph.mon.keyring。#node1chown ceph:ceph /tmp/ceph.mon.keyring# 应用主机名，主机IP地址和FSID生成monitor map。另存为/tmp/monmap：#node1monmaptool --create --add node1 192.168.1.156 --add node2 192.168.1.157 --add node3 192.168.1.159 --fsid $cephuid /tmp/monmap# 复制monitor map到另外2个节点#node1scp /tmp/monmap root@node2:/tmpscp /tmp/monmap root@node3:/tmp# 复制ceph.client.admin.keyring到另外2个节点#node1scp /etc/ceph/ceph.client.admin.keyring root@node2:/etc/ceph/scp /etc/ceph/ceph.client.admin.keyring root@node3:/etc/ceph/# 复制ceph.mon.keyring到另外2个节点#node1scp /tmp/ceph.mon.keyring root@node2:/tmp/scp /tmp/ceph.mon.keyring root@node3:/tmp/#留神批改文件权限#node2chown ceph:ceph /tmp/ceph.mon.keyring#node3chown ceph:ceph /tmp/ceph.mon.keyring# 创立monitor数据目录#node1sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node1#node2sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node2#node3sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node3# 用monitor map和keyring填充monitor守护程序。#node1sudo -u ceph ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring#node2sudo -u ceph ceph-mon --mkfs -i node2 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring#node3sudo -u ceph ceph-mon --mkfs -i node3 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring# 查看生成的文件#node1ls /var/lib/ceph/mon/ceph-node1/keyring kv_backend store.db# 启动monitor服务#node1systemctl restart ceph-mon@node1systemctl enable ceph-mon@node1#node2systemctl restart ceph-mon@node2systemctl enable ceph-mon@node2#node3systemctl restart ceph-mon@node3systemctl enable ceph-mon@node3# 查看以后集群状态ceph -s cluster: id: 8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb health: HEALTH_OK services: mon: 3 daemons, quorum node1,node2,node3 (age 0.35737s) mgr: no daemons active osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: # 若异样则启用msgr2# ceph mon enable-msgr2初始化manager节点
#node1ceph auth get-or-create mgr.node1 mon 'allow profile mgr' osd 'allow *' mds 'allow *'sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node1sudo -u ceph vim /var/lib/ceph/mgr/ceph-node1/keyring[mgr.node1] key = AQBk7aZiZD1NDRAAfXyfT2ovmsJwADzkbioHzQ== #node2ceph auth get-or-create mgr.node2 mon 'allow profile mgr' osd 'allow *' mds 'allow *'sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node2sudo -u ceph vim /var/lib/ceph/mgr/ceph-node2/keyring[mgr.node2] key = AQB67aZicvq7DhAAKEUipQSIDZEUZVv740mEuA==#node3ceph auth get-or-create mgr.node3 mon 'allow profile mgr' osd 'allow *' mds 'allow *'sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node3sudo -u ceph vim /var/lib/ceph/mgr/ceph-node3/keyring[mgr.node3] key = AQCS7aZiC75UIhAA2aue7yr1XGiBs4cRt8ru3A==# 启动ceph-mgr守护程序：#node1systemctl restart ceph-mgr@node1systemctl enable ceph-mgr@node1#node2systemctl restart ceph-mgr@node2systemctl enable ceph-mgr@node2#node3systemctl restart ceph-mgr@node3systemctl enable ceph-mgr@node3# 通过ceph status查看输入来查看mgr是否呈现ceph status cluster: id: 8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb health: HEALTH_WARN mons are allowing insecure global_id reclaim clock skew detected on mon.node2, mon.node3 OSD count 0 < osd_pool_default_size 3 services: mon: 3 daemons, quorum node1,node2,node3 (age 29s) mgr: node3(active, since 19s), standbys: node1, node2 osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 增加OSD
# 复制keyring到其余2个节点#node1scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@node2:/var/lib/ceph/bootstrap-osd/scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@node3:/var/lib/ceph/bootstrap-osd/# 创立OSD[root@node1 ~]# lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTsda 8:0 0 100G 0 disk ├─sda1 8:1 0 1G 0 part /boot└─sda2 8:2 0 99G 0 part ├─cs-root 253:0 0 61.2G 0 lvm / ├─cs-swap 253:1 0 7.9G 0 lvm └─cs-home 253:2 0 29.9G 0 lvm /homesdb 8:16 0 10G 0 disk # 3个节点上执行yum install ceph-volumeceph-volume lvm create --data /dev/sdb# 启动各个节点osd过程#node1systemctl restart ceph-osd@0systemctl enable ceph-osd@0#node2systemctl restart ceph-osd@1systemctl enable ceph-osd@1#node3systemctl restart ceph-osd@2systemctl enable ceph-osd@2# 查看集群状态ceph -s cluster: id: 8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb health: HEALTH_WARN mons are allowing insecure global_id reclaim services: mon: 3 daemons, quorum node1,node2,node3 (age 5m) mgr: node3(active, since 4m), standbys: node1, node2 osd: 3 osds: 3 up (since 7s), 3 in (since 62s) data: pools: 1 pools, 1 pgs objects: 2 objects, 577 KiB usage: 18 MiB used, 30 GiB / 30 GiB avail pgs: 1 active+clean io: client: 1.2 KiB/s rd, 36 KiB/s wr, 1 op/s rd, 1 op/s wr recovery: 27 KiB/s, 0 objects/s增加MDS
# 创立mds数据目录。#node1sudo -u ceph mkdir -p /var/lib/ceph/mds/ceph-node1#node2sudo -u ceph mkdir -p /var/lib/ceph/mds/ceph-node2#node3sudo -u ceph mkdir -p /var/lib/ceph/mds/ceph-node3# 创立keyring：#node1ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node1/keyring --gen-key -n mds.node1#node2ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node2/keyring --gen-key -n mds.node2#node3ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node3/keyring --gen-key -n mds.node3# 导入keyring并设置权限：#node1ceph auth add mds.node1 osd "allow rwx" mds "allow" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-node1/keyringchown ceph:ceph /var/lib/ceph/mds/ceph-node1/keyring#node2ceph auth add mds.node2 osd "allow rwx" mds "allow" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-node2/keyringchown ceph:ceph /var/lib/ceph/mds/ceph-node2/keyring#node3ceph auth add mds.node3 osd "allow rwx" mds "allow" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-node3/keyringchown ceph:ceph /var/lib/ceph/mds/ceph-node3/keyring收尾
所有节点批改ceph.conf配置文件，追加以下内容cat >> /etc/ceph/ceph.conf <<EOF[mds.node1]host = node1[mds.node2]host = node2[mds.node3]host = node3EOF重新启动所有服务#node1systemctl restart ceph-mon@node1systemctl restart ceph-mgr@node1systemctl restart ceph-mds@node1systemctl enable ceph-mds@node1systemctl restart ceph-osd@0#node2systemctl restart ceph-mon@node2systemctl restart ceph-mgr@node2systemctl restart ceph-mds@node2systemctl enable ceph-mds@node2systemctl restart ceph-osd@1#node3systemctl restart ceph-mon@node3systemctl restart ceph-mgr@node3systemctl restart ceph-mds@node3systemctl enable ceph-mds@node3systemctl restart ceph-osd@2查看集群状态ceph -s cluster: id: 8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb health: HEALTH_WARN mons are allowing insecure global_id reclaim services: mon: 3 daemons, quorum node1,node2,node3 (age 9s) mgr: node3(active, since 4s), standbys: node1, node2 osd: 3 osds: 3 up (since 4s), 3 in (since 2m) data: pools: 1 pools, 1 pgs objects: 2 objects, 577 KiB usage: 18 MiB used, 30 GiB / 30 GiB avail pgs: 1 active+clean查看osd状态[root@node1 ~]# ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-1 0.02939 root default -3 0.00980 host node1 0 hdd 0.00980 osd.0 up 1.00000 1.00000-5 0.00980 host node2 1 hdd 0.00980 osd.1 up 1.00000 1.00000-7 0.00980 host node3 2 hdd 0.00980 osd.2 up 1.00000 1.00000对于
https://www.oiox.cn/
https://www.oiox.cn/index.php...
CSDN、GitHub、知乎、微信公众号、开源中国、思否、掘金、简书、华为云、阿里云、腾讯云、哔哩哔哩、今日头条、新浪微博、集体博客、全网可搜《小陈运维》
文章次要公布于微信公众号