共计 10781 个字符,预计需要花费 27 分钟才能阅读完成。
疾速部署 Ceph 分布式高可用集群
Ceph 简介
Ceph 是一个 PB,EB 级别的分布式存储系统,能够提供文件存储,对象存储、和块存储,它可靠性高,易扩大,治理简便,其中对象存储和块存储能够和其余云平台集成。一个 Ceph 集群中有 Monitor 节点、MDS 节点(用于文件存储)、OSD 守护过程。
Ceph 根底概念
-
ceph-deploy
一个集群自动化部署工具,应用较久,成熟稳固,被很多自动化工具所集成,可用于生产部署;
-
cephadm
从 Octopus 开始提供的新集群部署工具,反对通过图形界面或者命令行界面增加节点,目前不倡议用于生产环境,有趣味能够尝试;
-
manual
手动部署,一步步部署 Ceph 集群,反对较多定制化和理解部署细节,装置难度较大,但能够清晰把握装置部署的细节。
-
admin-node:
须要一个装置治理节点,装置节点负责集群整体部署,这里咱们用 CephNode01 为 admin-node 和 Ceph-Mon 节点;
-
mon:
monitor 节点,即是 Ceph 的监督治理节点,承当 Ceph 集群重要的治理工作,个别须要 3 或 5 个节点,此处部署简略的一个 Monitor 节点;
-
osd:
OSD 即 Object Storage Daemon,理论负责数据存储的节点,3 个节点上别离有 2 块 100G 的磁盘充当 OSD 角色。
Ceph 零碎初始化
配置主机信息
# 设置主机名 | |
#node1 | |
hostnamectl set-hostname node1 | |
#node2 | |
hostnamectl set-hostname node2 | |
#node3 | |
hostnamectl set-hostname node3 | |
# 写入 hosts | |
cat >> /etc/hosts <<EOF | |
192.168.1.156 node1 | |
192.168.1.157 node2 | |
192.168.1.159 node3 | |
EOF | |
cat /etc/hosts | |
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 | |
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 | |
192.168.1.156 node1 | |
192.168.1.157 node2 | |
192.168.1.159 node3 |
配置免密
# 配置免密(二选一)ssh-keygen -t rsa | |
Generating public/private rsa key pair. | |
Enter file in which to save the key (/root/.ssh/id_rsa): | |
Created directory '/root/.ssh'. | |
Enter passphrase (empty for no passphrase): | |
Enter same passphrase again: | |
Your identification has been saved in /root/.ssh/id_rsa. | |
Your public key has been saved in /root/.ssh/id_rsa.pub. | |
The key fingerprint is: | |
SHA256:nK3CqSGRBGZfrE5rncPEQ2eU/Gq6dttYMLIiesXHyO8 root@ceph-01 | |
The key's randomart image is: | |
+---[RSA 3072]----+ | |
|.o ..o.. | | |
|o.. .o = | | |
| ..+ o . | | |
| . + + . + | | |
| =o=+ooS . | | |
| ==*=+o. | | |
| .oo.+B .. | | |
|. o..=.o+ | | |
|.. ooEo.. | | |
+----[SHA256]-----+ | |
# 将免密传输到各个主机上 | |
ssh-copy-id root@node1 | |
ssh-copy-id root@node2 | |
ssh-copy-id root@node3 | |
# 应用懒人形式配置免密(二选一)yum install -y sshpass | |
ssh-keygen -f /root/.ssh/id_rsa -P ''export IP="node1 node2 node3" | |
export SSHPASS=123123 | |
for HOST in $IP;do | |
sshpass -e ssh-copy-id -o StrictHostKeyChecking=no $HOST | |
done |
配置根底环境
# 敞开防火墙 | |
systemctl stop firewalld | |
systemctl disable firewalld | |
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service. | |
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. | |
# 敞开 swap | |
swapoff -a | |
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab | |
# 敞开 selinux | |
setenforce 0 | |
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config |
配置 YUM 源
# 配置 yum 源 | |
sudo sed -e 's|^mirrorlist=|#mirrorlist=|g' \ | |
-e 's|^#baseurl=http://mirror.centos.org/$contentdir|baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos|g' \ | |
-i.bak \ | |
/etc/yum.repos.d/CentOS-*.repo | |
# 配置 ceph 源 | |
cat > /etc/yum.repos.d/ceph.repo <<EOF | |
[noarch] | |
name=Ceph noarch | |
baseurl=https://mirrors.ustc.edu.cn/ceph/rpm-17.2.0/el8/noarch/ | |
enabled=1 | |
gpgcheck=0 | |
[x86_64] | |
name=Ceph x86_64 | |
baseurl=https://mirrors.ustc.edu.cn/ceph/rpm-17.2.0/el8/x86_64/ | |
enabled=1 | |
gpgcheck=0 | |
EOF |
装置根底环境
# 更新 yum 源 | |
yum update -y | |
# 装置工具包、python-setuptools 肯定要装置、不然会报错的 | |
yum install -y chrony conntrack ipset jq iptables curl sysstat libseccomp wget socat git vim epel-release epel-next-release |
调整时区 \ 间
# 配置零碎时区 | |
timedatectl set-timezone Asia/Shanghai | |
# 配置时钟同步 | |
timedatectl status | |
# 注:System clock synchronized: yes,示意时钟已同步;NTP service: active,示意开启了时钟同步服务 | |
# 写入硬件时钟 | |
# 将以后的 UTC 工夫写入硬件时钟 | |
timedatectl set-local-rtc 0 | |
# 重启依赖于零碎工夫的服务 | |
systemctl restart rsyslog | |
systemctl restart crond |
杂项
# 敞开无关服务 | |
systemctl stop postfix && systemctl disable postfix | |
# 重启 | |
reboot |
Ceph 零碎装置
初始化 monitor 节点
yum install ceph -y | |
# 初始化 monitor 节点 | |
# 在 node1 节点生成 uuid,并在所有节点导入 uuid 环境变量 | |
[root@node1 ~]# uuidgen | |
8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb | |
#node1 | |
export cephuid=8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb | |
#node2 | |
export cephuid=8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb | |
#node3 | |
export cephuid=8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb | |
# 所有节点创立 Ceph 配置文件:cat > /etc/ceph/ceph.conf <<EOF | |
[global] | |
fsid = 8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb | |
mon initial members = node1, node2, node3 | |
mon host = 192.168.1.156, 192.168.1.157, 192.168.1.159 | |
public network = 192.168.1.0/24 | |
auth cluster required = cephx | |
auth service required = cephx | |
auth client required = cephx | |
osd journal size = 1024 | |
osd pool default size = 3 | |
osd pool default min size = 2 | |
osd pool default pg num = 333 | |
osd pool default pgp num = 333 | |
osd crush chooseleaf type = 1 | |
EOF | |
# 以下操作在 node1 节点执行 | |
# 为集群创立一个 keyring,并生成一个 monitor 密钥。#node1 | |
ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *' | |
# 生成 administrator keyring,生成 client.admin 用户并将用户增加到 keyring。#node1 | |
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' | |
# 生成 bootstrap-osd keyring,生成 client.bootstrap-osd 用户并将用户增加到 keyring。#node1 | |
ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r' | |
# 将生成的密钥增加到中 ceph.mon.keyring。#node1 | |
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring | |
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring | |
# 将所有者更改为 ceph.mon.keyring。#node1 | |
chown ceph:ceph /tmp/ceph.mon.keyring | |
# 应用主机名,主机 IP 地址和 FSID 生成 monitor map。另存为 /tmp/monmap:#node1 | |
monmaptool --create --add node1 192.168.1.156 --add node2 192.168.1.157 --add node3 192.168.1.159 --fsid $cephuid /tmp/monmap | |
# 复制 monitor map 到另外 2 个节点 | |
#node1 | |
scp /tmp/monmap root@node2:/tmp | |
scp /tmp/monmap root@node3:/tmp | |
# 复制 ceph.client.admin.keyring 到另外 2 个节点 | |
#node1 | |
scp /etc/ceph/ceph.client.admin.keyring root@node2:/etc/ceph/ | |
scp /etc/ceph/ceph.client.admin.keyring root@node3:/etc/ceph/ | |
# 复制 ceph.mon.keyring 到另外 2 个节点 | |
#node1 | |
scp /tmp/ceph.mon.keyring root@node2:/tmp/ | |
scp /tmp/ceph.mon.keyring root@node3:/tmp/ | |
#留神批改文件权限 | |
#node2 | |
chown ceph:ceph /tmp/ceph.mon.keyring | |
#node3 | |
chown ceph:ceph /tmp/ceph.mon.keyring | |
# 创立 monitor 数据目录 | |
#node1 | |
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node1 | |
#node2 | |
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node2 | |
#node3 | |
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node3 | |
# 用 monitor map 和 keyring 填充 monitor 守护程序。#node1 | |
sudo -u ceph ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring | |
#node2 | |
sudo -u ceph ceph-mon --mkfs -i node2 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring | |
#node3 | |
sudo -u ceph ceph-mon --mkfs -i node3 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring | |
# 查看生成的文件 | |
#node1 | |
ls /var/lib/ceph/mon/ceph-node1/ | |
keyring kv_backend store.db | |
# 启动 monitor 服务 | |
#node1 | |
systemctl restart ceph-mon@node1 | |
systemctl enable ceph-mon@node1 | |
#node2 | |
systemctl restart ceph-mon@node2 | |
systemctl enable ceph-mon@node2 | |
#node3 | |
systemctl restart ceph-mon@node3 | |
systemctl enable ceph-mon@node3 | |
# 查看以后集群状态 | |
ceph -s | |
cluster: | |
id: 8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb | |
health: HEALTH_OK | |
services: | |
mon: 3 daemons, quorum node1,node2,node3 (age 0.35737s) | |
mgr: no daemons active | |
osd: 0 osds: 0 up, 0 in | |
data: | |
pools: 0 pools, 0 pgs | |
objects: 0 objects, 0 B | |
usage: 0 B used, 0 B / 0 B avail | |
pgs: | |
# 若异样则启用 msgr2 | |
# ceph mon enable-msgr2 |
初始化 manager 节点
#node1 | |
ceph auth get-or-create mgr.node1 mon 'allow profile mgr' osd 'allow *' mds 'allow *' | |
sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node1 | |
sudo -u ceph vim /var/lib/ceph/mgr/ceph-node1/keyring | |
[mgr.node1] | |
key = AQBk7aZiZD1NDRAAfXyfT2ovmsJwADzkbioHzQ== | |
#node2 | |
ceph auth get-or-create mgr.node2 mon 'allow profile mgr' osd 'allow *' mds 'allow *' | |
sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node2 | |
sudo -u ceph vim /var/lib/ceph/mgr/ceph-node2/keyring | |
[mgr.node2] | |
key = AQB67aZicvq7DhAAKEUipQSIDZEUZVv740mEuA== | |
#node3 | |
ceph auth get-or-create mgr.node3 mon 'allow profile mgr' osd 'allow *' mds 'allow *' | |
sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node3 | |
sudo -u ceph vim /var/lib/ceph/mgr/ceph-node3/keyring | |
[mgr.node3] | |
key = AQCS7aZiC75UIhAA2aue7yr1XGiBs4cRt8ru3A== | |
# 启动 ceph-mgr 守护程序:#node1 | |
systemctl restart ceph-mgr@node1 | |
systemctl enable ceph-mgr@node1 | |
#node2 | |
systemctl restart ceph-mgr@node2 | |
systemctl enable ceph-mgr@node2 | |
#node3 | |
systemctl restart ceph-mgr@node3 | |
systemctl enable ceph-mgr@node3 | |
# 通过 ceph status 查看输入来查看 mgr 是否呈现 | |
ceph status | |
cluster: | |
id: 8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb | |
health: HEALTH_WARN | |
mons are allowing insecure global_id reclaim | |
clock skew detected on mon.node2, mon.node3 | |
OSD count 0 < osd_pool_default_size 3 | |
services: | |
mon: 3 daemons, quorum node1,node2,node3 (age 29s) | |
mgr: node3(active, since 19s), standbys: node1, node2 | |
osd: 0 osds: 0 up, 0 in | |
data: | |
pools: 0 pools, 0 pgs | |
objects: 0 objects, 0 B | |
usage: 0 B used, 0 B / 0 B avail | |
pgs: |
增加 OSD
# 复制 keyring 到其余 2 个节点 | |
#node1 | |
scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@node2:/var/lib/ceph/bootstrap-osd/ | |
scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@node3:/var/lib/ceph/bootstrap-osd/ | |
# 创立 OSD | |
[root@node1 ~]# lsblk | |
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT | |
sda 8:0 0 100G 0 disk | |
├─sda1 8:1 0 1G 0 part /boot | |
└─sda2 8:2 0 99G 0 part | |
├─cs-root 253:0 0 61.2G 0 lvm / | |
├─cs-swap 253:1 0 7.9G 0 lvm | |
└─cs-home 253:2 0 29.9G 0 lvm /home | |
sdb 8:16 0 10G 0 disk | |
# 3 个节点上执行 | |
yum install ceph-volume | |
ceph-volume lvm create --data /dev/sdb | |
# 启动各个节点 osd 过程 | |
#node1 | |
systemctl restart ceph-osd@0 | |
systemctl enable ceph-osd@0 | |
#node2 | |
systemctl restart ceph-osd@1 | |
systemctl enable ceph-osd@1 | |
#node3 | |
systemctl restart ceph-osd@2 | |
systemctl enable ceph-osd@2 | |
# 查看集群状态 | |
ceph -s | |
cluster: | |
id: 8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb | |
health: HEALTH_WARN | |
mons are allowing insecure global_id reclaim | |
services: | |
mon: 3 daemons, quorum node1,node2,node3 (age 5m) | |
mgr: node3(active, since 4m), standbys: node1, node2 | |
osd: 3 osds: 3 up (since 7s), 3 in (since 62s) | |
data: | |
pools: 1 pools, 1 pgs | |
objects: 2 objects, 577 KiB | |
usage: 18 MiB used, 30 GiB / 30 GiB avail | |
pgs: 1 active+clean | |
io: | |
client: 1.2 KiB/s rd, 36 KiB/s wr, 1 op/s rd, 1 op/s wr | |
recovery: 27 KiB/s, 0 objects/s |
增加 MDS
# 创立 mds 数据目录。#node1 | |
sudo -u ceph mkdir -p /var/lib/ceph/mds/ceph-node1 | |
#node2 | |
sudo -u ceph mkdir -p /var/lib/ceph/mds/ceph-node2 | |
#node3 | |
sudo -u ceph mkdir -p /var/lib/ceph/mds/ceph-node3 | |
# 创立 keyring:#node1 | |
ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node1/keyring --gen-key -n mds.node1 | |
#node2 | |
ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node2/keyring --gen-key -n mds.node2 | |
#node3 | |
ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node3/keyring --gen-key -n mds.node3 | |
# 导入 keyring 并设置权限:#node1 | |
ceph auth add mds.node1 osd "allow rwx" mds "allow" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-node1/keyring | |
chown ceph:ceph /var/lib/ceph/mds/ceph-node1/keyring | |
#node2 | |
ceph auth add mds.node2 osd "allow rwx" mds "allow" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-node2/keyring | |
chown ceph:ceph /var/lib/ceph/mds/ceph-node2/keyring | |
#node3 | |
ceph auth add mds.node3 osd "allow rwx" mds "allow" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-node3/keyring | |
chown ceph:ceph /var/lib/ceph/mds/ceph-node3/keyring |
收尾
所有节点批改 ceph.conf 配置文件,追加以下内容 | |
cat >> /etc/ceph/ceph.conf <<EOF | |
[mds.node1] | |
host = node1 | |
[mds.node2] | |
host = node2 | |
[mds.node3] | |
host = node3 | |
EOF | |
重新启动所有服务 | |
#node1 | |
systemctl restart ceph-mon@node1 | |
systemctl restart ceph-mgr@node1 | |
systemctl restart ceph-mds@node1 | |
systemctl enable ceph-mds@node1 | |
systemctl restart ceph-osd@0 | |
#node2 | |
systemctl restart ceph-mon@node2 | |
systemctl restart ceph-mgr@node2 | |
systemctl restart ceph-mds@node2 | |
systemctl enable ceph-mds@node2 | |
systemctl restart ceph-osd@1 | |
#node3 | |
systemctl restart ceph-mon@node3 | |
systemctl restart ceph-mgr@node3 | |
systemctl restart ceph-mds@node3 | |
systemctl enable ceph-mds@node3 | |
systemctl restart ceph-osd@2 | |
查看集群状态 | |
ceph -s | |
cluster: | |
id: 8d2cfd33-9132-48a7-8c00-3ef10cb5ddeb | |
health: HEALTH_WARN | |
mons are allowing insecure global_id reclaim | |
services: | |
mon: 3 daemons, quorum node1,node2,node3 (age 9s) | |
mgr: node3(active, since 4s), standbys: node1, node2 | |
osd: 3 osds: 3 up (since 4s), 3 in (since 2m) | |
data: | |
pools: 1 pools, 1 pgs | |
objects: 2 objects, 577 KiB | |
usage: 18 MiB used, 30 GiB / 30 GiB avail | |
pgs: 1 active+clean | |
查看 osd 状态 | |
[root@node1 ~]# ceph osd tree | |
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF | |
-1 0.02939 root default | |
-3 0.00980 host node1 | |
0 hdd 0.00980 osd.0 up 1.00000 1.00000 | |
-5 0.00980 host node2 | |
1 hdd 0.00980 osd.1 up 1.00000 1.00000 | |
-7 0.00980 host node3 | |
2 hdd 0.00980 osd.2 up 1.00000 1.00000 |
对于
https://www.oiox.cn/
https://www.oiox.cn/index.php…
CSDN、GitHub、知乎、微信公众号、开源中国、思否、掘金、简书、华为云、阿里云、腾讯云、哔哩哔哩、今日头条、新浪微博、集体博客、全网可搜《小陈运维》
文章次要公布于微信公众号
正文完