一、前言
为了便于学习容器编排，实际容器编排，本篇文章记录在本地虚拟机搭建一个准生产级别的Kubernetes集群。在此k8s集群的根底上，咱们能够尝试着容器化工作或者学习场景中的各种中间件集群，以及微服务利用。
二、集群搭建资源筹备
从学习的目标登程，筹备搭建一个管制节点和两个工作节点的集群，为了节约老本，须要反对pod被调度到管制节点上。
2.1、机器筹备
| 虚拟机 | 节点角色 | 系统配置 | 装置组件 |
|---|---|---|---|
| k8s-master01 | 主节点 | centos7，2核6G | kube-apiserver、kube-scheduler、kube-controller-manager、etcd、kube-proxy、kubeadm、kubelet、kubectl、docker |
| k8s-workder01 | 工作节点 | centos7，2核6G | kubeadm、kubelet、kubectl、kube-proxy、docker |
| k8s-workder02 | 工作节点 | centos7，2核6G | kubeadm、kubelet、kubectl、kube-proxy、docker |
2.2、组件介绍
| 组件名称 | 组件类型 | 部署形式 | 组件形容 |
|---|---|---|---|
| kube-apiserver | 管制立体组件 | 通过kubeadm容器化部署 | API 服务器是 Kubernetes 管制面的组件， 该组件公开了 Kubernetes API。 API 服务器是 Kubernetes 管制面的前端 |
| kube-scheduler | 管制立体组件 | 通过kubeadm容器化部署 | 负责监督新创建的、未指定运行节点（node）的 Pods，抉择节点让 Pod 在下面运行 |
| kube-controller-manager | 管制立体组件 | 通过kubeadm容器化部署 | 从逻辑上讲，每个控制器都是一个独自的过程， 然而为了升高复杂性，它们都被编译到同一个可执行文件，并在一个过程中运行 |
| etcd | 管制立体组件 | 通过kubeadm容器化部署 | etcd是兼具一致性和高可用性的键值数据库，作为保留 Kubernetes所有集群数据（资源对象等）的后盾数据库 |
| kubelet | 节点组件 | yum装置，无奈容器化部署 | kubelet 接管一组通过各类机制提供给它的 PodSpecs，确保这些 PodSpecs 中形容的容器处于运行状态且衰弱。 kubelet 不会治理不是由 Kubernetes 创立的容器 |
| kube-proxy | 节点组件 | 通过kubeadm容器化部署 | kube-proxy 是集群中每个节点上运行的网络代理， 实现 Kubernetes 服务（Service） 概念的一部分。kube-proxy 保护节点上的网络规定。这些网络规定容许从集群外部或内部的网络会话与 Pod 进行网络通信。 |
| docker | 节点组件 | yum装置 | docker引擎，k8s反对的一种容器运行时实现 |
| kubeadm | 集群辅助部署工具 | yum装置 | k8s官网集群部署工具 |
| kubectl | 客户端工具 | yum装置 | 用来与集群通信的命令行工具 |
2.3、筹备工作确认
无论是本地物理机还是私有云上的虚拟机，都应该保障以下几点要求
- 满足装置 Docker 我的项目所需的要求，比方 64 位的 Linux 操作系统、3.10 及以上的内核版本
- x86 或者 ARM 架构均可
- 机器之间网络互通，容器网络互通的前提
- 有外网拜访权限，因为须要拉取镜像（也可手动下载好镜像）
- 节点之中不能够有反复的主机名、MAC 地址或 product_uuid
- 开启机器上的某些端口，详情参照k8s官网，k8s组件端口协定
- 禁用替换分区。为了保障 kubelet 失常工作
2.4、实际部署指标
- 在所有节点上装置 Docker 和 kubeadm
- 部署 Kubernetes Master
- 部署容器网络插件
- 部署 Kubernetes Worker
- 部署 Dashboard 可视化插件
- 部署容器存储插件
三、集群搭建
3.1、装置docker引擎
k8s通过下层CRI接口对接容器运行时实现，这里咱们采纳docker引擎这个容器运行时实现。因为咱们采纳的是centos7的机器部署k8s集群，此处采纳yum源装置docker引擎
- Centos7 yum源装置docker引擎
- 参考docker官网的装置手册
留神：配置 Docker 守护程序，尤其是应用 systemd 来治理容器的 cgroup，放弃docker与kubelet的cgroup驱动统一
[root@vm-k8s-master ~]# cat /etc/docker/daemon.json { #配置docker的镜像仓库，减速镜像的拉取 "registry-mirrors": ["https://hub-mirror.c.163.com","https://reg-mirror.qiniu.com"], #配置cgroup驱动为systemd "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, #对于运行 Linux 内核版本 4.0 或更高版本，或应用 3.10.0-51 及更高版本的 RHEL 或 CentOS 的零碎，overlay2是首选的存储驱动程序。 "storage-driver": "overlay2"}3.2、装置kubeadm
3.2.1、配置主机名
k8s集群节点之中不能够有反复的主机名、MAC 地址或 product_uuid
#配置主机名hostnamectl set-hostname master#查看主机名，查看是否k8s集群节点的主机名是否有反复cat /etc/hostname #product_uuid校验sudo cat /sys/class/dmi/id/product_uuid#应用命令ip link 或 ifconfig -a来获取网络接口的MAC地址ifconfig -a3.2.2、批改节点host
cat >> /etc/hosts << EOF192.168.31.254 k8s.master.com192.168.31.254 k8s.cluster-endpoint192.168.31.58 k8s.worker01.com192.168.31.20 k8s.worker02.comEOF阐明：以上是k8s三个节点的IP域名映射。
3.2.3、容许iptables查看桥接流量
- 确保br_netfilter模块被加载
#查看是否加载了br_netfilter模块lsmod | grep br_netfiltercat <<EOF | sudo tee /etc/modules-load.d/k8s.confbr_netfilterEOF#若没有加载br_netfilter模块，则显式加载br_netfilter模块sudo modprobe br_netfilter- 确保iptables可能正确地查看桥接流量
为了让你的 Linux 节点上的 iptables 可能正确地查看桥接流量，你须要确保在你的 sysctl 配置中将 net.bridge.bridge-nf-call-iptables 设置为 1
cat <<EOF | sudo tee /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1EOFsudo sysctl --system3.2.4、禁用swap替换分区
#禁用所有替换替换设施swapoff -a# 正文swap行vim /etc/fstab#重启机器reboot3.2.5、配置kubernetes的yum源
配置官网的kubernetes yum源（网络容许的状况下）
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo[kubernetes]name=Kubernetesbaseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearchenabled=1gpgcheck=1repo_gpgcheck=1gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpgexclude=kubelet kubeadm kubectlEOF配置采纳阿里云镜像的kubernetes yum源
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo[kubernetes]name=Kubernetesbaseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-\$basearchenabled=1gpgcheck=0repo_gpgcheck=0gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpgexclude=kubelet kubeadm kubectlEOF
3.2.6、yum源装置kubeadm、kubelet、kubectl
# 将SELinux设置为permissive模式（相当于将其禁用）# 通过运行命令 setenforce 0 和 sed ... 将 SELinux 设置为 permissive 模式 能够无效地将其禁用。 这是容许容器拜访主机文件系统所必须的，而这些操作时为了例如 Pod 网络工作失常sudo setenforce 0sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config#敞开防火墙，这个不是必须得，然而须要保障k8s的各个组件的端口交互的规定的保护，试验环境简略敞开解决systemctl stop firewalldsystemctl disable firewalld#yum源装置kubeadm、kubelet、kubectlsudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes#查问kubelet版本，以便于找到到一个版本，用于下一步指定版本装置yum list kubelet --showduplicates | sort -r #指定版本装置sudo yum install -y kubelet-<version> kubeadm-<version> kubectl-<version> --disableexcludes=kubernetes#开机自启动kubeletsudo systemctl enable --now kubelet#开机即设置kubectl命令补全echo "source <(kubectl completion bash)" >> ~/.bash_profilesource .bash_profile- 在上述装置 kubeadm 的过程中，kubeadm 和 kubelet、kubectl、kubernetes-cni 这几个二进制文件都会被主动装置好
================================================================================================================================================================================================================== Package 架构 版本 源 大小==================================================================================================================================================================================================================正在装置: kubeadm x86_64 1.23.5-0 kubernetes 9.0 M kubectl x86_64 1.23.5-0 kubernetes 9.5 M kubelet x86_64 1.23.5-0 kubernetes 21 M为依赖而装置: conntrack-tools x86_64 1.4.4-7.el7 base 187 k cri-tools x86_64 1.23.0-0 kubernetes 7.1 M kubernetes-cni x86_64 0.8.7-0 kubernetes 19 M libnetfilter_cthelper x86_64 1.0.0-11.el7 base 18 k libnetfilter_cttimeout x86_64 1.0.0-7.el7 base 18 k libnetfilter_queue x86_64 1.0.2-2.el7_2 base 23 k socat x86_64 1.7.3.2-2.el7 base 290 k事务概要==================================================================================================================================================================================================================装置 3 软件包 (+7 依赖软件包)总下载量：65 M装置大小：297 MDownloading packages:(1/10): conntrack-tools-1.4.4-7.el7.x86_64.rpm | 187 kB 00:00:00 (2/10): 4d300a7655f56307d35f127d99dc192b6aa4997f322234e754f16aaa60fd8906-cri-tools-1.23.0-0.x86_64.rpm | 7.1 MB 00:00:00 (3/10): ab0e12925be5251baf5dd3b31493663d46e4a7b458c7a5b6b717f4ae87a81bd4-kubeadm-1.23.5-0.x86_64.rpm | 9.0 MB 00:00:00 (4/10): 96b208380314a19ded917eaf125ed748f5e2b28a3cc8707a10a76a9f5b61c0df-kubectl-1.23.5-0.x86_64.rpm | 9.5 MB 00:00:00 (5/10): libnetfilter_cthelper-1.0.0-11.el7.x86_64.rpm | 18 kB 00:00:00 (6/10): socat-1.7.3.2-2.el7.x86_64.rpm | 290 kB 00:00:00 (7/10): libnetfilter_cttimeout-1.0.0-7.el7.x86_64.rpm | 18 kB 00:00:00 (8/10): d39aa6eb38a6a8326b7e88c622107327dfd02ac8aaae32eceb856643a2ad9981-kubelet-1.23.5-0.x86_64.rpm | 21 MB 00:00:01 (9/10): db7cb5cb0b3f6875f54d10f02e625573988e3e91fd4fc5eef0b1876bb18604ad-kubernetes-cni-0.8.7-0.x86_64.rpm | 19 MB 00:00:01 (10/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm | 23 kB 00:00:01 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------总计 26 MB/s | 65 MB 00:00:02 Running transaction checkRunning transaction testTransaction test succeededRunning transaction 正在装置 : libnetfilter_cthelper-1.0.0-11.el7.x86_64 1/10 正在装置 : socat-1.7.3.2-2.el7.x86_64 2/10 正在装置 : libnetfilter_cttimeout-1.0.0-7.el7.x86_64 3/10 正在装置 : cri-tools-1.23.0-0.x86_64 4/10 正在装置 : libnetfilter_queue-1.0.2-2.el7_2.x86_64 5/10 正在装置 : conntrack-tools-1.4.4-7.el7.x86_64 6/10 正在装置 : kubernetes-cni-0.8.7-0.x86_64 7/10 正在装置 : kubelet-1.23.5-0.x86_64 8/10 正在装置 : kubectl-1.23.5-0.x86_64 9/10 正在装置 : kubeadm-1.23.5-0.x86_64 10/10 验证中 : conntrack-tools-1.4.4-7.el7.x86_64 1/10 验证中 : kubernetes-cni-0.8.7-0.x86_64 2/10 验证中 : kubectl-1.23.5-0.x86_64 3/10 验证中 : kubeadm-1.23.5-0.x86_64 4/10 验证中 : libnetfilter_queue-1.0.2-2.el7_2.x86_64 5/10 验证中 : cri-tools-1.23.0-0.x86_64 6/10 验证中 : kubelet-1.23.5-0.x86_64 7/10 验证中 : libnetfilter_cttimeout-1.0.0-7.el7.x86_64 8/10 验证中 : socat-1.7.3.2-2.el7.x86_64 9/10 验证中 : libnetfilter_cthelper-1.0.0-11.el7.x86_64 10/10 已装置: kubeadm.x86_64 0:1.23.5-0 kubectl.x86_64 0:1.23.5-0 kubelet.x86_64 0:1.23.5-0 作为依赖被装置: conntrack-tools.x86_64 0:1.4.4-7.el7 cri-tools.x86_64 0:1.23.0-0 kubernetes-cni.x86_64 0:0.8.7-0 libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 结束！阐明：kubelet当初每隔几秒就会重启，因为它陷入了一个期待kubeadm指令的死循环。
留神：如果校验签名报错，敞开校验，设置（gpgcheck=0、repo_gpgcheck=0）即可
https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for kubernetes3.3、部署Kubernetes的Master节点
3.3.1、查看kubeadm须要下载的镜像
[root@vm-k8s-master ~]# kubeadm config images list --kubernetes-version=1.23.5k8s.gcr.io/kube-apiserver:v1.23.5k8s.gcr.io/kube-controller-manager:v1.23.5k8s.gcr.io/kube-scheduler:v1.23.5k8s.gcr.io/kube-proxy:v1.23.5k8s.gcr.io/pause:3.6k8s.gcr.io/etcd:3.5.1-0k8s.gcr.io/coredns/coredns:v1.8.6阐明：kubernetes所有皆容器的思维体现，k8s管制立体的外围组件均采纳pod治理容器部署。默认状况下, kubeadm 会从 k8s.gcr.io 仓库拉取镜像。如果申请的 Kubernetes 版本是 CI 标签 （例如 ci/latest），则应用 gcr.io/k8s-staging-ci-images，如果网络不容许，咱们能够手动下载以上所有的容器镜像，也能够指定从自定义的镜像仓库拉取
3.3.2、指定镜像仓库初始化管制立体（master节点）
kubeadm init \--apiserver-advertise-address=192.168.31.254 \--control-plane-endpoint=k8s.cluster-endpoint \--image-repository registry.aliyuncs.com/google_containers \--kubernetes-version v1.23.5 \--service-cidr=10.1.0.0/16 \--pod-network-cidr=10.244.0.0/16- apiserver-advertise-address：API 服务器所颁布的其正在监听的 IP 地址。如果未设置，则应用默认网络接口
- control-plane-endpoint：为管制立体指定一个稳固的IP地址或DNS名称，
- image-repository：抉择用于拉取管制平面镜像的容器仓库，默认值："k8s.gcr.io"
- kubernetes-version：为管制立体抉择一个特定的Kubernetes版本， 默认值："stable-1"
- service-cidr：为服务的虚构IP地址另外指定IP地址段，默认值："10.96.0.0/12"
- pod-network-cidr：指明pod网络能够应用的IP地址段。如果设置了这个参数，管制立体将会为每一个节点主动调配 CIDRs
3.3.2.1、初始化管制立体日志
[root@vm-k8s-master k8s-init]# kubeadm init \> --apiserver-advertise-address=192.168.31.254 \> --control-plane-endpoint=k8s.cluster-endpoint \> --image-repository registry.aliyuncs.com/google_containers \> --kubernetes-version v1.23.5 \> --service-cidr=10.1.0.0/16 \> --pod-network-cidr=10.244.0.0/16[init] Using Kubernetes version: v1.23.5[preflight] Running pre-flight checks[preflight] Pulling images required for setting up a Kubernetes cluster[preflight] This might take a minute or two, depending on the speed of your internet connection[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'[certs] Using certificateDir folder "/etc/kubernetes/pki"[certs] Generating "ca" certificate and key[certs] Generating "apiserver" certificate and key[certs] apiserver serving cert is signed for DNS names [k8s.cluster-endpoint kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local vm-k8s-master] and IPs [10.1.0.1 192.168.31.254][certs] Generating "apiserver-kubelet-client" certificate and key[certs] Generating "front-proxy-ca" certificate and key[certs] Generating "front-proxy-client" certificate and key[certs] Generating "etcd/ca" certificate and key[certs] Generating "etcd/server" certificate and key[certs] etcd/server serving cert is signed for DNS names [localhost vm-k8s-master] and IPs [192.168.31.254 127.0.0.1 ::1][certs] Generating "etcd/peer" certificate and key[certs] etcd/peer serving cert is signed for DNS names [localhost vm-k8s-master] and IPs [192.168.31.254 127.0.0.1 ::1][certs] Generating "etcd/healthcheck-client" certificate and key[certs] Generating "apiserver-etcd-client" certificate and key[certs] Generating "sa" key and public key[kubeconfig] Using kubeconfig folder "/etc/kubernetes"[kubeconfig] Writing "admin.conf" kubeconfig file[kubeconfig] Writing "kubelet.conf" kubeconfig file[kubeconfig] Writing "controller-manager.conf" kubeconfig file[kubeconfig] Writing "scheduler.conf" kubeconfig file[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"[kubelet-start] Starting the kubelet[control-plane] Using manifest folder "/etc/kubernetes/manifests"[control-plane] Creating static Pod manifest for "kube-apiserver"[control-plane] Creating static Pod manifest for "kube-controller-manager"[control-plane] Creating static Pod manifest for "kube-scheduler"[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s[apiclient] All control plane components are healthy after 6.002185 seconds[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the clusterNOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.[upload-certs] Skipping phase. Please see --upload-certs[mark-control-plane] Marking the node vm-k8s-master as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers][mark-control-plane] Marking the node vm-k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule][bootstrap-token] Using token: vz7g8m.fhgc8sby6tm6mi25[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key[addons] Applied essential addon: CoreDNS[addons] Applied essential addon: kube-proxyYour Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/You can now join any number of control-plane nodes by copying certificate authoritiesand service account keys on each node and then running the following as root: kubeadm join k8s.cluster-endpoint:6443 --token vz7g8m.fhgc8sby6tm6mi25 \ --discovery-token-ca-cert-hash sha256:e368eb6bdd202ff442cef277b97f12fe374d63071ff9dc277add24301609204d \ --control-plane Then you can join any number of worker nodes by running the following on each as root:kubeadm join k8s.cluster-endpoint:6443 --token vz7g8m.fhgc8sby6tm6mi25 \ --discovery-token-ca-cert-hash sha256:e368eb6bdd202ff442cef277b97f12fe374d63071ff9dc277add24301609204d 阐明：以上日志中有两个比拟重要的信息，如下：
- 集群增加工作节点的指令
kubeadm join k8s.cluster-endpoint:6443 --token vz7g8m.fhgc8sby6tm6mi25 \ --discovery-token-ca-cert-hash sha256:e368eb6bdd202ff442cef277b97f12fe374d63071ff9dc277add24301609204d \ --control-plane - kubectl客户端依赖的集群配置
#普通用户须要将集群配置复制到用户的家目录下，kubectl客户端会默认读取mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config#root用户为了简略不便起见，设置环境变量echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profilesource ~/.bash_profile阐明：Kubernetes 集群默认须要加密形式拜访。所以，这几条命令，就是将刚刚部署生成的 Kubernetes 集群的平安配置文件，保留到以后用户的.kube 目录下，kubectl 默认会应用这个目录下的受权信息拜访 Kubernetes集群。
3.3.3、为管制立体装置网络插件
初始化管制立体后，咱们首先来查看下节点的状态，会发现Master节点的状态是NotReady。
[root@vm-k8s-master k8s-init]# kubectl get nodesNAME STATUS ROLES AGE VERSIONvm-k8s-master NotReady control-plane,master 24m v1.23.5进一步，通过 kubectl describe 指令的输入，咱们能够看到 NodeNotReady 的起因在于，咱们尚未部署任何网络插件。
[root@vm-k8s-master k8s-init]# kubectl describe node vm-k8s-masterName: vm-k8s-masterRoles: control-plane,masterLabels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=vm-k8s-master kubernetes.io/os=linux node-role.kubernetes.io/control-plane= node-role.kubernetes.io/master= node.kubernetes.io/exclude-from-external-load-balancers=Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: trueCreationTimestamp: Sat, 26 Mar 2022 16:48:43 +0800Taints: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/not-ready:NoScheduleUnschedulable: falseLease: HolderIdentity: vm-k8s-master AcquireTime: <unset> RenewTime: Sat, 26 Mar 2022 17:18:00 +0800Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Sat, 26 Mar 2022 17:14:24 +0800 Sat, 26 Mar 2022 16:48:40 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Sat, 26 Mar 2022 17:14:24 +0800 Sat, 26 Mar 2022 16:48:40 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Sat, 26 Mar 2022 17:14:24 +0800 Sat, 26 Mar 2022 16:48:40 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready False Sat, 26 Mar 2022 17:14:24 +0800 Sat, 26 Mar 2022 16:48:40 +0800 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitializedAddresses: InternalIP: 192.168.31.254 Hostname: vm-k8s-masterCapacity: cpu: 2 ephemeral-storage: 39617640Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 5925672Ki pods: 110Allocatable: cpu: 2 ephemeral-storage: 36511616964 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 5823272Ki pods: 110System Info: Machine ID: 5bf19e1a8ca94a5c987c497fd9d169f9 System UUID: 4A364D56-F429-F892-581C-C7E2B013277E Boot ID: 8a40b366-2645-4ee1-bf40-90e60b23a2df Kernel Version: 3.10.0-1160.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://20.10.13 Kubelet Version: v1.23.5 Kube-Proxy Version: v1.23.5PodCIDR: 10.244.0.0/24PodCIDRs: 10.244.0.0/24Non-terminated Pods: (5 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-system etcd-vm-k8s-master 100m (5%) 0 (0%) 100Mi (1%) 0 (0%) 29m kube-system kube-apiserver-vm-k8s-master 250m (12%) 0 (0%) 0 (0%) 0 (0%) 29m kube-system kube-controller-manager-vm-k8s-master 200m (10%) 0 (0%) 0 (0%) 0 (0%) 29m kube-system kube-proxy-dvrkb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 29m kube-system kube-scheduler-vm-k8s-master 100m (5%) 0 (0%) 0 (0%) 0 (0%) 29mAllocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 650m (32%) 0 (0%) memory 100Mi (1%) 0 (0%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%)Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Starting 28m kube-proxy Normal Starting 29m kubelet Starting kubelet. Normal NodeAllocatableEnforced 29m kubelet Updated Node Allocatable limit across pods Normal NodeHasSufficientMemory 29m kubelet Node vm-k8s-master status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 29m kubelet Node vm-k8s-master status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 29m kubelet Node vm-k8s-master status is now: NodeHasSufficientPID另外，能够看到，CoreDNS这个依赖于网络的Pod处于Pending状态，即调度失败。这当然是合乎预期的，因为这个 Master 节点的网络尚未就绪。
[root@vm-k8s-master k8s-init]# kubectl get pods -n kube-systemNAME READY STATUS RESTARTS AGEcoredns-6d8c4cb4d-6ldld 0/1 Pending 0 31mcoredns-6d8c4cb4d-8z59l 0/1 Pending 0 31metcd-vm-k8s-master 1/1 Running 0 32mkube-apiserver-vm-k8s-master 1/1 Running 0 32mkube-controller-manager-vm-k8s-master 1/1 Running 0 32mkube-proxy-dvrkb 1/1 Running 0 31mkube-scheduler-vm-k8s-master 1/1 Running 0 32m3.3.3.1、装置flannel网络插件
在Kubernetes我的项目所有皆容器的设计理念领导下，部署网络插件也是采纳pod部署。
#如果网络容许可间接执行kubectl apply指令即可kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml#如果网络不容许，无奈下载kube-flannel.yml，能够手动复制以下的kube-flannel.yml文件内容，留神手动创立的kube-flannel.yml地位执行以下命令kubectl apply -f ./kube-flannel.yml- kube-flannel.yml文件内容
---apiVersion: policy/v1beta1kind: PodSecurityPolicymetadata: name: psp.flannel.unprivileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/defaultspec: privileged: false volumes: - configMap - secret - emptyDir - hostPath allowedHostPaths: - pathPrefix: "/etc/cni/net.d" - pathPrefix: "/etc/kube-flannel" - pathPrefix: "/run/flannel" readOnlyRootFilesystem: false # Users and groups runAsUser: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny # Privilege Escalation allowPrivilegeEscalation: false defaultAllowPrivilegeEscalation: false # Capabilities allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] defaultAddCapabilities: [] requiredDropCapabilities: [] # Host namespaces hostPID: false hostIPC: false hostNetwork: true hostPorts: - min: 0 max: 65535 # SELinux seLinux: # SELinux is unused in CaaSP rule: 'RunAsAny'---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata: name: flannelrules:- apiGroups: ['extensions'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: ['psp.flannel.unprivileged']- apiGroups: - "" resources: - pods verbs: - get- apiGroups: - "" resources: - nodes verbs: - list - watch- apiGroups: - "" resources: - nodes/status verbs: - patch---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata: name: flannelroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannelsubjects:- kind: ServiceAccount name: flannel namespace: kube-system---apiVersion: v1kind: ServiceAccountmetadata: name: flannel namespace: kube-system---kind: ConfigMapapiVersion: v1metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node app: flanneldata: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } }---apiVersion: apps/v1kind: DaemonSetmetadata: name: kube-flannel-ds namespace: kube-system labels: tier: node app: flannelspec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni-plugin #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 command: - cp args: - -f - /flannel - /opt/cni/bin/flannel volumeMounts: - name: cni-plugin mountPath: /opt/cni/bin - name: install-cni #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) image: rancher/mirrored-flannelcni-flannel:v0.17.0 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) image: rancher/mirrored-flannelcni-flannel:v0.17.0 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ - name: xtables-lock mountPath: /run/xtables.lock volumes: - name: run hostPath: path: /run/flannel - name: cni-plugin hostPath: path: /opt/cni/bin - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg - name: xtables-lock hostPath: path: /run/xtables.lock type: FileOrCreate- 装置flannel网络插件日志
[root@vm-k8s-master k8s-init]# kubectl apply -f ./kube-flannel.yml Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+podsecuritypolicy.policy/psp.flannel.unprivileged createdclusterrole.rbac.authorization.k8s.io/flannel createdclusterrolebinding.rbac.authorization.k8s.io/flannel createdserviceaccount/flannel unchangedconfigmap/kube-flannel-cfg configureddaemonset.apps/kube-flannel-ds created- 网络插件部署实现后，咱们能够通过 kubectl get从新查看Pod的状态
[root@vm-k8s-master k8s-init]# kubectl get pod -n kube-systemNAME READY STATUS RESTARTS AGEcoredns-6d8c4cb4d-6ldld 1/1 Running 0 95mcoredns-6d8c4cb4d-8z59l 1/1 Running 0 95metcd-vm-k8s-master 1/1 Running 0 95mkube-apiserver-vm-k8s-master 1/1 Running 0 95mkube-controller-manager-vm-k8s-master 1/1 Running 0 95mkube-flannel-ds-gwccf 1/1 Running 0 38skube-proxy-dvrkb 1/1 Running 0 95mkube-scheduler-vm-k8s-master 1/1 Running 0 95m能够看到，所有的零碎 Pod 都胜利启动了，而刚刚部署的flannel网络插件则在kube-system上面新建了一个名叫kube-flannel-ds-gwccf的Pod，一般来说，这些Pod就是容器网络插件在每个节点上的管制组件。
至此，Kubernetes 的 Master 节点就部署实现了。如果你只须要一个单节点的 Kubernetes，当初你就能够应用了。不过，在默认状况下，Kubernetes 的 Master 节点是不能运行用户 Pod 的，所以还须要额定做一个小操作。
3.3.4、管制立体节点污点打消
默认状况下Master节点是不容许运行用户Pod的。而Kubernetes做到这一点，依附的是 Kubernetes的Taint/Toleration机制。
它的原理非常简单：一旦某个节点被加上了一个Taint，即被“打上了污点”，那么所有Pod就都不能在这个节点上运行，因为Kubernetes的Pod都有“洁癖”。除非，有个别的Pod申明本人能“容忍”这个“污点”，即申明了Toleration，它才能够在这个节点上运行。
- 为节点打上“污点”（Taint）的命令是
#该node1节点上就会减少一个键值对格局的 Taint，即：foo=bar:NoSchedule。其中值外面的 NoSchedule，意味着这个Taint只会在调度新Pod时产生作用，而不会影响曾经在 node1 上运行的 Pod，哪怕它们没有Tolerationkubectl taint nodes node1 foo=bar:NoSchedule- Pod申明Toleration污点
apiVersion: v1kind: Pod...spec: tolerations: - key: "foo" operator: "Equal" value: "bar" effect: "NoSchedule"这个 Toleration 的含意是，这个Pod能“容忍”所有键值对为foo=bar的Taint（operator: “Equal”，“等于”操作）。
- kubectl describe查看Master节点的Taint字段
[root@vm-k8s-master k8s-init]# kubectl describe node vm-k8s-masterName: vm-k8s-masterRoles: control-plane,masterLabels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=vm-k8s-master kubernetes.io/os=linux node-role.kubernetes.io/control-plane= node-role.kubernetes.io/master= node.kubernetes.io/exclude-from-external-load-balancers=Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"d6:07:13:de:95:c6"} flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: true flannel.alpha.coreos.com/public-ip: 192.168.31.254 kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: trueCreationTimestamp: Sat, 26 Mar 2022 16:48:43 +0800Taints: node-role.kubernetes.io/master:NoSchedule能够看到，Master节点默认被加上了node-role.kubernetes.io/master:NoSchedule这样一个“污点”，其中“键”是node-role.kubernetes.io/master，而没有提供“值”。
- 删除master节点的默认污点
为了不便起见，测试学习环境须要master节点也能被调度pod运行，简略粗犷地为master节点删除掉默认污点。
#在“node-role.kubernetes.io/master”这个键前面加上了一个短横线“-”，这个格局就意味着移除所有以“node-role.kubernetes.io/master”为键的 Taint。kubectl taint nodes --all node-role.kubernetes.io/master-3.4、为Kubernetes集群增加工作节点
Kubernetes的Worker节点跟Master节点简直是雷同的，它们运行着的都是一个 kubelet 组件。惟一的区别在于，在kubeadm init的过程中，kubelet 启动后，Master 节点上还会主动运行 kube-apiserver、kube-scheduler、kube-controller-manger 这三个零碎 Pod。
到这里默认工作节点曾经装置了kubeadm、kubectl、kubelet、docker。接下来只有执行kubeadm init的过程中生成的kubeadm join执行就能够了。
- 执行部署Master节点时生成的kubeadm join指令
kubeadm join k8s.cluster-endpoint:6443 --token vz7g8m.fhgc8sby6tm6mi25 \ --discovery-token-ca-cert-hash sha256:e368eb6bdd202ff442cef277b97f12fe374d63071ff9dc277add24301609204d- worker节点退出集群日志输入
[root@vm-k8s-worker01 manifests]# kubeadm join k8s.cluster-endpoint:6443 --token vz7g8m.fhgc8sby6tm6mi25 \> --discovery-token-ca-cert-hash sha256:e368eb6bdd202ff442cef277b97f12fe374d63071ff9dc277add24301609204d[preflight] Running pre-flight checks[preflight] Reading configuration from the cluster...[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"[kubelet-start] Starting the kubelet[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...This node has joined the cluster:* Certificate signing request was sent to apiserver and a response was received.* The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.3.5、部署和拜访 Kubernetes 仪表板（Dashboard）
Dashboard 是基于网页的 Kubernetes 用户界面。 你能够应用 Dashboard 将容器利用部署到 Kubernetes 集群中，也能够对容器利用排错，还能治理集群资源。你能够应用 Dashboard 获取运行在集群中的利用的概览信息，也能够创立或者批改 Kubernetes资源（如 Deployment，Job，DaemonSet 等等）。 例如，你能够对Deployment实现弹性伸缩、发动滚动降级、重启 Pod 或者应用向导创立新的利用。
Dashboard 同时展现了 Kubernetes 集群中的资源状态信息和所有报错信息。
- 参考官网部署文档
3.5.1、部署Dashboard可视化插件
#向k8s集群提交dashboard资源组件配置，如果网络不通，可复制以下提供的配置文件内容kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml- kubernetes-dashboard.yaml文件内容
# Copyright 2017 The Kubernetes Authors.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.apiVersion: v1kind: Namespacemetadata: name: kubernetes-dashboard---apiVersion: v1kind: ServiceAccountmetadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard---kind: ServiceapiVersion: v1metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboardspec: ports: - port: 443 targetPort: 8443 selector: k8s-app: kubernetes-dashboard---apiVersion: v1kind: Secretmetadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-certs namespace: kubernetes-dashboardtype: Opaque---apiVersion: v1kind: Secretmetadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-csrf namespace: kubernetes-dashboardtype: Opaquedata: csrf: ""---apiVersion: v1kind: Secretmetadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-key-holder namespace: kubernetes-dashboardtype: Opaque---kind: ConfigMapapiVersion: v1metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-settings namespace: kubernetes-dashboard---kind: RoleapiVersion: rbac.authorization.k8s.io/v1metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboardrules: # Allow Dashboard to get, update and delete Dashboard exclusive secrets. - apiGroups: [""] resources: ["secrets"] resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"] verbs: ["get", "update", "delete"] # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] resourceNames: ["kubernetes-dashboard-settings"] verbs: ["get", "update"] # Allow Dashboard to get metrics. - apiGroups: [""] resources: ["services"] resourceNames: ["heapster", "dashboard-metrics-scraper"] verbs: ["proxy"] - apiGroups: [""] resources: ["services/proxy"] resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"] verbs: ["get"]---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboardrules: # Allow Metrics Scraper to get metrics from the Metrics server - apiGroups: ["metrics.k8s.io"] resources: ["pods", "nodes"] verbs: ["get", "list", "watch"]---apiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboardroleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubernetes-dashboardsubjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: kubernetes-dashboardroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubernetes-dashboardsubjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard---kind: DeploymentapiVersion: apps/v1metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboardspec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard spec: securityContext: seccompProfile: type: RuntimeDefault containers: - name: kubernetes-dashboard image: kubernetesui/dashboard:v2.5.0 imagePullPolicy: Always ports: - containerPort: 8443 protocol: TCP args: - --auto-generate-certificates - --namespace=kubernetes-dashboard # Uncomment the following line to manually specify Kubernetes API server Host # If not specified, Dashboard will attempt to auto discover the API server and connect # to it. Uncomment only if the default does not work. # - --apiserver-host=http://my-address:port volumeMounts: - name: kubernetes-dashboard-certs mountPath: /certs # Create on-disk volume to store exec logs - mountPath: /tmp name: tmp-volume livenessProbe: httpGet: scheme: HTTPS path: / port: 8443 initialDelaySeconds: 30 timeoutSeconds: 30 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 volumes: - name: kubernetes-dashboard-certs secret: secretName: kubernetes-dashboard-certs - name: tmp-volume emptyDir: {} serviceAccountName: kubernetes-dashboard nodeSelector: "kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule---kind: ServiceapiVersion: v1metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboardspec: ports: - port: 8000 targetPort: 8000 selector: k8s-app: dashboard-metrics-scraper---kind: DeploymentapiVersion: apps/v1metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboardspec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: dashboard-metrics-scraper template: metadata: labels: k8s-app: dashboard-metrics-scraper spec: securityContext: seccompProfile: type: RuntimeDefault containers: - name: dashboard-metrics-scraper image: kubernetesui/metrics-scraper:v1.0.7 ports: - containerPort: 8000 protocol: TCP livenessProbe: httpGet: scheme: HTTP path: / port: 8000 initialDelaySeconds: 30 timeoutSeconds: 30 volumeMounts: - mountPath: /tmp name: tmp-volume securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 serviceAccountName: kubernetes-dashboard nodeSelector: "kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule volumes: - name: tmp-volume emptyDir: {}- kubernetes-dashboard装置日志
[root@vm-k8s-master k8s-init]# kubectl apply -f ./kubernetes-dashboard.yaml namespace/kubernetes-dashboard createdserviceaccount/kubernetes-dashboard createdservice/kubernetes-dashboard createdsecret/kubernetes-dashboard-certs createdsecret/kubernetes-dashboard-csrf createdsecret/kubernetes-dashboard-key-holder createdconfigmap/kubernetes-dashboard-settings createdrole.rbac.authorization.k8s.io/kubernetes-dashboard createdclusterrole.rbac.authorization.k8s.io/kubernetes-dashboard createdrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard createdclusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard createddeployment.apps/kubernetes-dashboard createdservice/dashboard-metrics-scraper createddeployment.apps/dashboard-metrics-scraper created3.5.2、创立超级管理员的账号用于登录Dashboard
Dashboard是一个Web Server，很多人常常会在本人的私有云上无心地裸露 Dashboard 的端口，从而造成安全隐患。所以，1.7 版本之后的 Dashboard 我的项目部署实现后，默认只能通过 Proxy的形式在本地拜访。
- 创立用户，并绑定集群admin权限
apiVersion: v1kind: ServiceAccountmetadata: name: admin-user namespace: kubernetes-dashboard---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: admin-userroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects:- kind: ServiceAccount name: admin-user namespace: kubernetes-dashboard- 生成Bearer Token
kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"#打印输出信息如下eyJhbGciOiJSUzI1NiIsImtpZCI6Il9uU25iZHpROE5WZkZOcDFFeGhGT0JRQjZOal93Zk1qOHNVVlRsQVU2QmMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWRzbDR2Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI0MzJkNTA5Yy0zNTQ4LTRlYTktOTNmZi03NTM2ZDkxY2YwMTIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.RLl_jz41Snyc4qEHFk9-o_JYKfv22Lv2JquTgXX92_9k9VQOswd7RhF1OsWywVfd5TKq9_tagMDfVRqHo6fVe-jB4blUL6j6iTxtkFno7P9snqboFsEoegSFCM-S9OaF1C5Dk_rcIp2yJzbALkmTC6VjdWxPt3TvjEclXRQl0cITlE1NFmJKpJqbNlG65l4SjnRO5KLinrgYfpcLxlroqlKd4ZFWenKrz16VOicCUHuP_mWlq4yxp_WRrfpCglX92u-sDY3ouJW5cL0eqk-w5XKhREFvm-C2Opoba-mDZVcbaQxEIw_VkeudLliSUs5_v9CKQGKPYK6r6rdESYi6sw3.5.3、批改kubernetes-dashboard的service
为了更敌对的拜访dashboard服务，须要批改kubernetes-dashboard.yaml的service配置局部，将service type从ClusterIP批改为NodePort的形式。
kind: ServiceapiVersion: v1metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboardspec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 30001 selector: k8s-app: kubernetes-dashboard- 更新以上配置文件后，执行kubectl apply更新资源对象
kubectl apply -f ./kubernetes-dashboard.yaml- 查看更新后的service资源对象
[root@vm-k8s-master k8s-init]# kubectl get svc -n kubernetes-dashboardNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEdashboard-metrics-scraper ClusterIP 10.1.245.206 <none> 8000/TCP 62mkubernetes-dashboard NodePort 10.1.50.101 <none> 443:30001/TCP 62m3.5.4、拜访kubernetes-dashboard
#留神此处的https协定，抉择token的认证形式，复制早前生成的token即可https://k8s.master.com:30001/3.6、部署rook存储插件
Rook 我的项目是一个基于Ceph的Kubernetes 存储插件（它前期也在退出对更多存储实现的反对）。不过，不同于对Ceph的简略封装，Rook 在本人的实现中退出了程度扩大、迁徙、劫难备份、监控等大量的企业级性能，使得这个我的项目变成了一个残缺的、生产级别可用的容器存储插件。
可在生产环境中对文件存储、块存储和对象存储进行治理。Rook由云原生计算基金会(CNCF) 作为毕业级我的项目托管。Rook是用golang 实现的。Ceph 是用C++实现的。
- 更多信息参考rook官网
- 查看rook github
3.6.1、部署rook存储插件前置条件
以以后最新的rook版本为v1.8.7为例，进行以下的装置步骤。为了配置Ceph存储集群，至多须要以下本地存储选项之一。
- 原始设施（无分区或格式化文件系统）
- 原始分区（无格式化文件系统）
- 可通过block模式从存储类取得PV
- 集群至多须要三个节点，每个节点至多须要一个OSD（对象存储设备），保障存储类的创立
- 须要Kubernetesv1.16或更高版本
3.6.2、部署rook存储插件前置筹备
因为本次搭建的环境，机器是通过VMware创立进去的，所以能够很不便地为每个虚拟机增加一块全新的硬盘。用于后续rook集群作为一个OSD（对象存储设备）。
- VMware增加虚拟机硬盘，参考vmware官网文档
1、抉择该虚拟机，而后抉择虚拟机 > 设置。2、在硬件选项卡中，单击增加。3、在新建硬件向导中，抉择硬盘。4、抉择创立新虚构磁盘。5、抉择磁盘类型。- 查看磁盘块设施
[root@vm-k8s-worker01 ~]# lsblk -fNAME FSTYPE LABEL UUID MOUNTPOINTsda ├─sda1 xfs 8fd14e7c-6b98-4845-bd19-2f1eb6b175b0 /boot└─sda2 LVM2_member ZT7jhW-IRXs-0KBo-FGSk-C85B-hnOA-upyrdP ├─centos_vm--k8s--jayway-root xfs 477d2f79-9e28-49a0-8f5f-01c69014014b / └─centos_vm--k8s--jayway-swap swap 949d1c1f-abe7-4fd2-9232-40526e1b8637 sdb sr0 iso9660 CentOS 7 x86_64 2020-11-04-11-36-43-00 阐明：留神FSTYPE字段的值，如果为空，则示意以后的设施尚未写入文件系统，例如以上的sdb设施就是刚新增的硬盘设施。
3.6.3、部署rook存储插件前置筹备
- 克隆rook git仓库
#以后最新版本为v1.8.7git clone --single-branch --branch v1.8.7 https://github.com/rook/rook.git- 查看rook集群部署的相干K8S资源配置文件
进入rook/deploy/examples目录，可查看rook我的项目提供了很多K8S资源配置文件。临时咱们只须要关注个别的资源配置文件。
| 资源文件 | 形容 |
|---|---|
| crds.yaml | 创立rook集群前，首先须要被创立的CRD资源对象，让k8s意识rook的自定义资源 |
| common.yaml | 启动operator和ceph cluster所必须得通用资源，必须先于operator.yaml、cluster.yaml资源被创立 |
| operator.yaml | rook operator控制器 |
| cluster.yaml | rook-ceph cluster配置 |
| storageclass.yaml | 创立存储类，为k8s集群提供动态创建pv的能力 |
| mysql.yaml | rook官网提供的mysql测试服务，依赖于storageclass创立的pv长久化 |
| wordpress.yaml | rook官网提供的wordpress测试服务，依赖于storageclass创立的pv长久化 |
-rw-r--r--. 1 root root 696 3月 27 10:27 bucket-notification-endpoint.yaml-rw-r--r--. 1 root root 1258 3月 27 10:27 bucket-notification.yaml-rw-r--r--. 1 root root 842 3月 27 10:27 bucket-topic.yaml-rw-r--r--. 1 root root 456 3月 27 10:27 ceph-client.yaml-rw-r--r--. 1 root root 1088 3月 27 10:27 cluster-external-management.yaml-rw-r--r--. 1 root root 1275 3月 27 10:27 cluster-external.yaml-rw-r--r--. 1 root root 7350 3月 27 10:27 cluster-on-local-pvc.yaml-rw-r--r--. 1 root root 8539 3月 27 10:27 cluster-on-pvc.yaml-rw-r--r--. 1 root root 5376 3月 27 10:27 cluster-stretched-aws.yaml-rw-r--r--. 1 root root 3312 3月 27 10:27 cluster-stretched.yaml-rw-r--r--. 1 root root 1798 3月 27 10:27 cluster-test.yaml-rw-r--r--. 1 root root 15343 3月 27 10:27 cluster.yaml-rw-r--r--. 1 root root 2418 3月 27 10:27 common-external.yaml-rw-r--r--. 1 root root 3908 3月 27 10:27 common-second-cluster.yaml-rw-r--r--. 1 root root 38180 3月 27 10:27 common.yaml-rw-r--r--. 1 root root 739053 3月 27 10:27 crds.yaml-rw-r--r--. 1 root root 75912 3月 27 10:27 create-external-cluster-resources.py-rw-r--r--. 1 root root 3696 3月 27 10:27 create-external-cluster-resources.shdrwxr-xr-x. 4 root root 31 3月 27 10:27 csi-rw-r--r--. 1 root root 430 3月 27 10:27 dashboard-external-https.yaml-rw-r--r--. 1 root root 429 3月 27 10:27 dashboard-external-http.yaml-rw-r--r--. 1 root root 988 3月 27 10:27 dashboard-ingress-https.yaml-rw-r--r--. 1 root root 432 3月 27 10:27 dashboard-loadbalancer.yaml-rw-r--r--. 1 root root 1904 3月 27 10:27 direct-mount.yaml-rw-r--r--. 1 root root 3729 3月 27 10:27 filesystem-ec.yaml-rw-r--r--. 1 root root 1253 3月 27 10:27 filesystem-mirror.yaml-rw-r--r--. 1 root root 825 3月 27 10:27 filesystem-test.yaml-rw-r--r--. 1 root root 5378 3月 27 10:27 filesystem.yaml-rw-r--r--. 1 root root 416 3月 27 10:27 images.txt-rw-r--r--. 1 root root 5142 3月 27 10:27 import-external-cluster.shdrwxr-xr-x. 2 root root 4096 3月 27 10:27 monitoring-rw-r--r--. 1 root root 1269 3月 27 10:27 mysql.yaml-rw-r--r--. 1 root root 747 3月 27 10:27 nfs-test.yaml-rw-r--r--. 1 root root 2635 3月 27 10:27 nfs.yaml-rw-r--r--. 1 root root 604 3月 27 10:27 object-bucket-claim-delete.yaml-rw-r--r--. 1 root root 502 3月 27 10:27 object-bucket-claim-notification.yaml-rw-r--r--. 1 root root 604 3月 27 10:27 object-bucket-claim-retain.yaml-rw-r--r--. 1 root root 3793 3月 27 10:27 object-ec.yaml-rw-r--r--. 1 root root 836 3月 27 10:27 object-external.yaml-rw-r--r--. 1 root root 1991 3月 27 10:27 object-multisite-pull-realm-test.yaml-rw-r--r--. 1 root root 2064 3月 27 10:27 object-multisite-pull-realm.yaml-rw-r--r--. 1 root root 1339 3月 27 10:27 object-multisite-test.yaml-rw-r--r--. 1 root root 1365 3月 27 10:27 object-multisite.yaml-rw-r--r--. 1 root root 6090 3月 27 10:27 object-openshift.yaml-rw-r--r--. 1 root root 707 3月 27 10:27 object-test.yaml-rw-r--r--. 1 root root 795 3月 27 10:27 object-user.yaml-rw-r--r--. 1 root root 5835 3月 27 10:27 object.yaml-rw-r--r--. 1 root root 23608 3月 27 10:27 operator-openshift.yaml-rw-r--r--. 1 root root 21444 3月 27 10:27 operator.yaml-rw-r--r--. 1 root root 944 3月 27 10:27 osd-env-override.yaml-rw-r--r--. 1 root root 3150 3月 27 10:27 osd-purge.yaml-rw-r--r--. 1 root root 797 3月 27 10:27 pool-device-health-metrics.yaml-rw-r--r--. 1 root root 1127 3月 27 10:27 pool-ec.yaml-rw-r--r--. 1 root root 507 3月 27 10:27 pool-mirrored.yaml-rw-r--r--. 1 root root 539 3月 27 10:27 pool-test.yaml-rw-r--r--. 1 root root 3461 3月 27 10:27 pool.yaml-rw-r--r--. 1 root root 1515 3月 27 10:27 rbdmirror.yaml-rw-r--r--. 1 root root 130 3月 27 10:27 README.md-rw-r--r--. 1 root root 546 3月 27 10:27 rgw-external.yaml-rw-r--r--. 1 root root 754 3月 27 10:27 storageclass-bucket-delete.yaml-rw-r--r--. 1 root root 752 3月 27 10:27 storageclass-bucket-retain.yaml-rw-r--r--. 1 root root 281 3月 27 10:27 subvolumegroup.yaml-rw-r--r--. 1 root root 1796 3月 27 10:27 toolbox-job.yaml-rw-r--r--. 1 root root 1672 3月 27 10:27 toolbox.yaml-rw-r--r--. 1 root root 460 3月 27 10:27 volume-replication-class.yaml-rw-r--r--. 1 root root 352 3月 27 10:27 volume-replication.yaml-rw-r--r--. 1 root root 1369 3月 27 10:27 wordpress.yaml3.6.4、rook存储插件装置
rook各个资源对象依赖的镜像如下所示，这些镜像也是在谷歌的镜像仓库。
[root@vm-k8s-master examples]# pwd/root/k8s-init/rook-git/deploy/examples[root@vm-k8s-master examples]# cat images.txt k8s.gcr.io/sig-storage/csi-attacher:v3.4.0 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0 k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0 k8s.gcr.io/sig-storage/csi-resizer:v1.4.0 k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1 quay.io/ceph/ceph:v16.2.7 quay.io/cephcsi/cephcsi:v3.5.1 quay.io/csiaddons/k8s-sidecar:v0.2.1 quay.io/csiaddons/volumereplication-operator:v0.3.0 rook/ceph:v1.8.73.6.4.1、创立的CRD资源对象
创立rook集群前，首先须要被创立的CRD资源对象，让k8s意识rook的自定义资源；此资源文件未依赖镜像。
#以后门路：rook/deploy/exampleskubectl apply -f crds.yaml- 执行日志
[root@vm-k8s-master examples]# kubectl apply -f crds.yamlcustomresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io createdcustomresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io createdcustomresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created3.6.4.2、创立通用资源对象
启动operator和ceph cluster所必须得通用资源，必须先于operator.yaml、cluster.yaml资源被创立。此资源文件未依赖镜像。
#以后门路：rook/deploy/exampleskubectl apply -f common.yaml- 执行日志
[root@vm-k8s-master examples]# kubectl apply -f common.yamlnamespace/rook-ceph createdclusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin createdclusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner createdclusterrole.rbac.authorization.k8s.io/psp:rook createdclusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin createdclusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner createdclusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt createdclusterrole.rbac.authorization.k8s.io/rook-ceph-global createdclusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster createdclusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system createdclusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket createdclusterrole.rbac.authorization.k8s.io/rook-ceph-osd createdclusterrole.rbac.authorization.k8s.io/rook-ceph-system createdclusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin createdclusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role createdclusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin createdclusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system createdclusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp createdclusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp createdclusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp createdclusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp createdclusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp createdWarning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+podsecuritypolicy.policy/00-rook-privileged createdrole.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg createdrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin createdrole.rbac.authorization.k8s.io/rbd-external-provisioner-cfg createdrole.rbac.authorization.k8s.io/rook-ceph-cmd-reporter createdrole.rbac.authorization.k8s.io/rook-ceph-mgr createdrole.rbac.authorization.k8s.io/rook-ceph-osd createdrole.rbac.authorization.k8s.io/rook-ceph-purge-osd createdrole.rbac.authorization.k8s.io/rook-ceph-system createdrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg createdrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg createdrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg createdrolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt createdrolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter createdrolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp createdrolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp createdrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr createdrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp createdrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system createdrolebinding.rbac.authorization.k8s.io/rook-ceph-osd createdrolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp createdrolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd createdrolebinding.rbac.authorization.k8s.io/rook-ceph-system createdserviceaccount/rook-ceph-cmd-reporter createdserviceaccount/rook-ceph-mgr createdserviceaccount/rook-ceph-osd createdserviceaccount/rook-ceph-purge-osd createdserviceaccount/rook-ceph-system createdserviceaccount/rook-csi-cephfs-plugin-sa createdserviceaccount/rook-csi-cephfs-provisioner-sa createdserviceaccount/rook-csi-rbd-plugin-sa createdserviceaccount/rook-csi-rbd-provisioner-sa created3.6.4.3、创立rook operator控制器
依赖于rook/ceph:v1.8.7镜像，此镜像国内网络能够失常拉取。然而也默认依赖CSI相干的镜像，须要拜访谷歌镜像仓库。
#以后门路：rook/deploy/exampleskubectl apply -f operator.yaml- 如果无奈拉取谷歌镜像，请批改相干依赖镜像仓库地址
ROOK_CSI_ATTACHER_IMAGE: "willdockerhub/csi-attacher:v3.4.0"ROOK_CSI_REGISTRAR_IMAGE: "willdockerhub/csi-node-driver-registrar:v2.5.0"ROOK_CSI_PROVISIONER_IMAGE: "willdockerhub/csi-provisioner:v3.1.0"ROOK_CSI_RESIZER_IMAGE: "willdockerhub/csi-resizer:v1.4.0"ROOK_CSI_SNAPSHOTTER_IMAGE: "willdockerhub/csi-snapshotter:v5.0.1"ROOK_CSI_CEPH_IMAGE: "willdockerhub/cephcsi:v3.5.1"ROOK_CSIADDONS_IMAGE: "willdockerhub/k8s-sidecar:v0.2.1"CSI_VOLUME_REPLICATION_IMAGE: "willdockerhub/volumereplication-operator:v0.3.0"- 批改过镜像后的operator.yaml文件内容
################################################################################################################## The deployment for the rook operator# Contains the common settings for most Kubernetes deployments.# For example, to create the rook-ceph cluster:# kubectl create -f crds.yaml -f common.yaml -f operator.yaml# kubectl create -f cluster.yaml## Also see other operator sample files for variations of operator.yaml:# - operator-openshift.yaml: Common settings for running in OpenShift################################################################################################################ Rook Ceph Operator Config ConfigMap# Use this ConfigMap to override Rook-Ceph Operator configurations.# NOTE! Precedence will be given to this config if the same Env Var config also exists in the# Operator Deployment.# To move a configuration(s) from the Operator Deployment to this ConfigMap, add the config# here. It is recommended to then remove it from the Deployment to eliminate any future confusion.kind: ConfigMapapiVersion: v1metadata: name: rook-ceph-operator-config # should be in the namespace of the operator namespace: rook-ceph # namespace:operatordata: # The logging level for the operator: ERROR | WARNING | INFO | DEBUG ROOK_LOG_LEVEL: "INFO" # Enable the CSI driver. # To run the non-default version of the CSI driver, see the override-able image properties in operator.yaml ROOK_CSI_ENABLE_CEPHFS: "true" # Enable the default version of the CSI RBD driver. To start another version of the CSI driver, see image properties below. ROOK_CSI_ENABLE_RBD: "true" ROOK_CSI_ENABLE_GRPC_METRICS: "false" # Set to true to enable host networking for CSI CephFS and RBD nodeplugins. This may be necessary # in some network configurations where the SDN does not provide access to an external cluster or # there is significant drop in read/write performance. # CSI_ENABLE_HOST_NETWORK: "true" # Set logging level for csi containers. # Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. # CSI_LOG_LEVEL: "0" # Set replicas for csi provisioner deployment. CSI_PROVISIONER_REPLICAS: "2" # OMAP generator will generate the omap mapping between the PV name and the RBD image. # CSI_ENABLE_OMAP_GENERATOR need to be enabled when we are using rbd mirroring feature. # By default OMAP generator sidecar is deployed with CSI provisioner pod, to disable # it set it to false. # CSI_ENABLE_OMAP_GENERATOR: "false" # set to false to disable deployment of snapshotter container in CephFS provisioner pod. CSI_ENABLE_CEPHFS_SNAPSHOTTER: "true" # set to false to disable deployment of snapshotter container in RBD provisioner pod. CSI_ENABLE_RBD_SNAPSHOTTER: "true" # Enable cephfs kernel driver instead of ceph-fuse. # If you disable the kernel client, your application may be disrupted during upgrade. # See the upgrade guide: https://rook.io/docs/rook/latest/ceph-upgrade.html # NOTE! cephfs quota is not supported in kernel version < 4.17 CSI_FORCE_CEPHFS_KERNEL_CLIENT: "true" # (Optional) policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted. # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html CSI_RBD_FSGROUPPOLICY: "ReadWriteOnceWithFSType" # (Optional) policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted. # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html CSI_CEPHFS_FSGROUPPOLICY: "ReadWriteOnceWithFSType" # (Optional) Allow starting unsupported ceph-csi image ROOK_CSI_ALLOW_UNSUPPORTED_VERSION: "false" # (Optional) control the host mount of /etc/selinux for csi plugin pods. CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT: "false" # The default version of CSI supported by Rook will be started. To change the version # of the CSI driver to something other than what is officially supported, change # these images to the desired release of the CSI driver. # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.5.1" # ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" # ROOK_CSI_RESIZER_IMAGE: "k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" # ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" # ROOK_CSI_SNAPSHOTTER_IMAGE: "k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" # ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" ROOK_CSI_ATTACHER_IMAGE: "willdockerhub/csi-attacher:v3.4.0" ROOK_CSI_REGISTRAR_IMAGE: "willdockerhub/csi-node-driver-registrar:v2.5.0" ROOK_CSI_PROVISIONER_IMAGE: "willdockerhub/csi-provisioner:v3.1.0" ROOK_CSI_RESIZER_IMAGE: "willdockerhub/csi-resizer:v1.4.0" ROOK_CSI_SNAPSHOTTER_IMAGE: "willdockerhub/csi-snapshotter:v5.0.1" ROOK_CSI_CEPH_IMAGE: "willdockerhub/cephcsi:v3.5.1" ROOK_CSIADDONS_IMAGE: "willdockerhub/k8s-sidecar:v0.2.1" CSI_VOLUME_REPLICATION_IMAGE: "willdockerhub/volumereplication-operator:v0.3.0" # (Optional) set user created priorityclassName for csi plugin pods. # CSI_PLUGIN_PRIORITY_CLASSNAME: "system-node-critical" # (Optional) set user created priorityclassName for csi provisioner pods. # CSI_PROVISIONER_PRIORITY_CLASSNAME: "system-cluster-critical" # CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. # Default value is RollingUpdate. # CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY: "OnDelete" # CSI RBD plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. # Default value is RollingUpdate. # CSI_RBD_PLUGIN_UPDATE_STRATEGY: "OnDelete" # kubelet directory path, if kubelet configured to use other than /var/lib/kubelet path. # ROOK_CSI_KUBELET_DIR_PATH: "/var/lib/kubelet" # Labels to add to the CSI CephFS Deployments and DaemonSets Pods. # ROOK_CSI_CEPHFS_POD_LABELS: "key1=value1,key2=value2" # Labels to add to the CSI RBD Deployments and DaemonSets Pods. # ROOK_CSI_RBD_POD_LABELS: "key1=value1,key2=value2" # (Optional) CephCSI provisioner NodeAffinity(applied to both CephFS and RBD provisioner). # CSI_PROVISIONER_NODE_AFFINITY: "role=storage-node; storage=rook, ceph" # (Optional) CephCSI provisioner tolerations list(applied to both CephFS and RBD provisioner). # Put here list of taints you want to tolerate in YAML format. # CSI provisioner would be best to start on the same nodes as other ceph daemons. # CSI_PROVISIONER_TOLERATIONS: | # - effect: NoSchedule # key: node-role.kubernetes.io/controlplane # operator: Exists # - effect: NoExecute # key: node-role.kubernetes.io/etcd # operator: Exists # (Optional) CephCSI plugin NodeAffinity(applied to both CephFS and RBD plugin). # CSI_PLUGIN_NODE_AFFINITY: "role=storage-node; storage=rook, ceph" # (Optional) CephCSI plugin tolerations list(applied to both CephFS and RBD plugin). # Put here list of taints you want to tolerate in YAML format. # CSI plugins need to be started on all the nodes where the clients need to mount the storage. # CSI_PLUGIN_TOLERATIONS: | # - effect: NoSchedule # key: node-role.kubernetes.io/controlplane # operator: Exists # - effect: NoExecute # key: node-role.kubernetes.io/etcd # operator: Exists # (Optional) CephCSI RBD provisioner NodeAffinity(if specified, overrides CSI_PROVISIONER_NODE_AFFINITY). # CSI_RBD_PROVISIONER_NODE_AFFINITY: "role=rbd-node" # (Optional) CephCSI RBD provisioner tolerations list(if specified, overrides CSI_PROVISIONER_TOLERATIONS). # Put here list of taints you want to tolerate in YAML format. # CSI provisioner would be best to start on the same nodes as other ceph daemons. # CSI_RBD_PROVISIONER_TOLERATIONS: | # - key: node.rook.io/rbd # operator: Exists # (Optional) CephCSI RBD plugin NodeAffinity(if specified, overrides CSI_PLUGIN_NODE_AFFINITY). # CSI_RBD_PLUGIN_NODE_AFFINITY: "role=rbd-node" # (Optional) CephCSI RBD plugin tolerations list(if specified, overrides CSI_PLUGIN_TOLERATIONS). # Put here list of taints you want to tolerate in YAML format. # CSI plugins need to be started on all the nodes where the clients need to mount the storage. # CSI_RBD_PLUGIN_TOLERATIONS: | # - key: node.rook.io/rbd # operator: Exists # (Optional) CephCSI CephFS provisioner NodeAffinity(if specified, overrides CSI_PROVISIONER_NODE_AFFINITY). # CSI_CEPHFS_PROVISIONER_NODE_AFFINITY: "role=cephfs-node" # (Optional) CephCSI CephFS provisioner tolerations list(if specified, overrides CSI_PROVISIONER_TOLERATIONS). # Put here list of taints you want to tolerate in YAML format. # CSI provisioner would be best to start on the same nodes as other ceph daemons. # CSI_CEPHFS_PROVISIONER_TOLERATIONS: | # - key: node.rook.io/cephfs # operator: Exists # (Optional) CephCSI CephFS plugin NodeAffinity(if specified, overrides CSI_PLUGIN_NODE_AFFINITY). # CSI_CEPHFS_PLUGIN_NODE_AFFINITY: "role=cephfs-node" # (Optional) CephCSI CephFS plugin tolerations list(if specified, overrides CSI_PLUGIN_TOLERATIONS). # Put here list of taints you want to tolerate in YAML format. # CSI plugins need to be started on all the nodes where the clients need to mount the storage. # CSI_CEPHFS_PLUGIN_TOLERATIONS: | # - key: node.rook.io/cephfs # operator: Exists # (Optional) CEPH CSI RBD provisioner resource requirement list, Put here list of resource # requests and limits you want to apply for provisioner pod # CSI_RBD_PROVISIONER_RESOURCE: | # - name : csi-provisioner # resource: # requests: # memory: 128Mi # cpu: 100m # limits: # memory: 256Mi # cpu: 200m # - name : csi-resizer # resource: # requests: # memory: 128Mi # cpu: 100m # limits: # memory: 256Mi # cpu: 200m # - name : csi-attacher # resource: # requests: # memory: 128Mi # cpu: 100m # limits: # memory: 256Mi # cpu: 200m # - name : csi-snapshotter # resource: # requests: # memory: 128Mi # cpu: 100m # limits: # memory: 256Mi # cpu: 200m # - name : csi-rbdplugin # resource: # requests: # memory: 512Mi # cpu: 250m # limits: # memory: 1Gi # cpu: 500m # - name : liveness-prometheus # resource: # requests: # memory: 128Mi # cpu: 50m # limits: # memory: 256Mi # cpu: 100m # (Optional) CEPH CSI RBD plugin resource requirement list, Put here list of resource # requests and limits you want to apply for plugin pod # CSI_RBD_PLUGIN_RESOURCE: | # - name : driver-registrar # resource: # requests: # memory: 128Mi # cpu: 50m # limits: # memory: 256Mi # cpu: 100m # - name : csi-rbdplugin # resource: # requests: # memory: 512Mi # cpu: 250m # limits: # memory: 1Gi # cpu: 500m # - name : liveness-prometheus # resource: # requests: # memory: 128Mi # cpu: 50m # limits: # memory: 256Mi # cpu: 100m # (Optional) CEPH CSI CephFS provisioner resource requirement list, Put here list of resource # requests and limits you want to apply for provisioner pod # CSI_CEPHFS_PROVISIONER_RESOURCE: | # - name : csi-provisioner # resource: # requests: # memory: 128Mi # cpu: 100m # limits: # memory: 256Mi # cpu: 200m # - name : csi-resizer # resource: # requests: # memory: 128Mi # cpu: 100m # limits: # memory: 256Mi # cpu: 200m # - name : csi-attacher # resource: # requests: # memory: 128Mi # cpu: 100m # limits: # memory: 256Mi # cpu: 200m # - name : csi-cephfsplugin # resource: # requests: # memory: 512Mi # cpu: 250m # limits: # memory: 1Gi # cpu: 500m # - name : liveness-prometheus # resource: # requests: # memory: 128Mi # cpu: 50m # limits: # memory: 256Mi # cpu: 100m # (Optional) CEPH CSI CephFS plugin resource requirement list, Put here list of resource # requests and limits you want to apply for plugin pod # CSI_CEPHFS_PLUGIN_RESOURCE: | # - name : driver-registrar # resource: # requests: # memory: 128Mi # cpu: 50m # limits: # memory: 256Mi # cpu: 100m # - name : csi-cephfsplugin # resource: # requests: # memory: 512Mi # cpu: 250m # limits: # memory: 1Gi # cpu: 500m # - name : liveness-prometheus # resource: # requests: # memory: 128Mi # cpu: 50m # limits: # memory: 256Mi # cpu: 100m # Configure CSI CSI Ceph FS grpc and liveness metrics port # CSI_CEPHFS_GRPC_METRICS_PORT: "9091" # CSI_CEPHFS_LIVENESS_METRICS_PORT: "9081" # Configure CSI RBD grpc and liveness metrics port # CSI_RBD_GRPC_METRICS_PORT: "9090" # CSI_RBD_LIVENESS_METRICS_PORT: "9080" # CSIADDONS_PORT: "9070" # Whether the OBC provisioner should watch on the operator namespace or not, if not the namespace of the cluster will be used ROOK_OBC_WATCH_OPERATOR_NAMESPACE: "true" # Whether to start the discovery daemon to watch for raw storage devices on nodes in the cluster. # This daemon does not need to run if you are only going to create your OSDs based on StorageClassDeviceSets with PVCs. ROOK_ENABLE_DISCOVERY_DAEMON: "false" # The timeout value (in seconds) of Ceph commands. It should be >= 1. If this variable is not set or is an invalid value, it's default to 15. ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS: "15" # Enable the volume replication controller. # Before enabling, ensure the Volume Replication CRDs are created. # See https://rook.io/docs/rook/latest/ceph-csi-drivers.html#rbd-mirroring CSI_ENABLE_VOLUME_REPLICATION: "false" # CSI_VOLUME_REPLICATION_IMAGE: "quay.io/csiaddons/volumereplication-operator:v0.3.0" # Enable the csi addons sidecar. CSI_ENABLE_CSIADDONS: "false" # ROOK_CSIADDONS_IMAGE: "quay.io/csiaddons/k8s-sidecar:v0.2.1"---# OLM: BEGIN OPERATOR DEPLOYMENTapiVersion: apps/v1kind: Deploymentmetadata: name: rook-ceph-operator namespace: rook-ceph # namespace:operator labels: operator: rook storage-backend: ceph app.kubernetes.io/name: rook-ceph app.kubernetes.io/instance: rook-ceph app.kubernetes.io/component: rook-ceph-operator app.kubernetes.io/part-of: rook-ceph-operatorspec: selector: matchLabels: app: rook-ceph-operator replicas: 1 template: metadata: labels: app: rook-ceph-operator spec: serviceAccountName: rook-ceph-system containers: - name: rook-ceph-operator image: rook/ceph:v1.8.7 args: ["ceph", "operator"] securityContext: runAsNonRoot: true runAsUser: 2016 runAsGroup: 2016 volumeMounts: - mountPath: /var/lib/rook name: rook-config - mountPath: /etc/ceph name: default-config-dir - mountPath: /etc/webhook name: webhook-cert ports: - containerPort: 9443 name: https-webhook protocol: TCP env: # If the operator should only watch for cluster CRDs in the same namespace, set this to "true". # If this is not set to true, the operator will watch for cluster CRDs in all namespaces. - name: ROOK_CURRENT_NAMESPACE_ONLY value: "false" # Rook Discover toleration. Will tolerate all taints with all keys. # Choose between NoSchedule, PreferNoSchedule and NoExecute: # - name: DISCOVER_TOLERATION # value: "NoSchedule" # (Optional) Rook Discover toleration key. Set this to the key of the taint you want to tolerate # - name: DISCOVER_TOLERATION_KEY # value: "<KeyOfTheTaintToTolerate>" # (Optional) Rook Discover tolerations list. Put here list of taints you want to tolerate in YAML format. # - name: DISCOVER_TOLERATIONS # value: | # - effect: NoSchedule # key: node-role.kubernetes.io/controlplane # operator: Exists # - effect: NoExecute # key: node-role.kubernetes.io/etcd # operator: Exists # (Optional) Rook Discover priority class name to set on the pod(s) # - name: DISCOVER_PRIORITY_CLASS_NAME # value: "<PriorityClassName>" # (Optional) Discover Agent NodeAffinity. # - name: DISCOVER_AGENT_NODE_AFFINITY # value: "role=storage-node; storage=rook, ceph" # (Optional) Discover Agent Pod Labels. # - name: DISCOVER_AGENT_POD_LABELS # value: "key1=value1,key2=value2" # The duration between discovering devices in the rook-discover daemonset. - name: ROOK_DISCOVER_DEVICES_INTERVAL value: "60m" # Whether to start pods as privileged that mount a host path, which includes the Ceph mon and osd pods. # Set this to true if SELinux is enabled (e.g. OpenShift) to workaround the anyuid issues. # For more details see https://github.com/rook/rook/issues/1314#issuecomment-355799641 - name: ROOK_HOSTPATH_REQUIRES_PRIVILEGED value: "false" # In some situations SELinux relabelling breaks (times out) on large filesystems, and doesn't work with cephfs ReadWriteMany volumes (last relabel wins). # Disable it here if you have similar issues. # For more details see https://github.com/rook/rook/issues/2417 - name: ROOK_ENABLE_SELINUX_RELABELING value: "true" # In large volumes it will take some time to chown all the files. Disable it here if you have performance issues. # For more details see https://github.com/rook/rook/issues/2254 - name: ROOK_ENABLE_FSGROUP value: "true" # Disable automatic orchestration when new devices are discovered - name: ROOK_DISABLE_DEVICE_HOTPLUG value: "false" # Provide customised regex as the values using comma. For eg. regex for rbd based volume, value will be like "(?i)rbd[0-9]+". # In case of more than one regex, use comma to separate between them. # Default regex will be "(?i)dm-[0-9]+,(?i)rbd[0-9]+,(?i)nbd[0-9]+" # Add regex expression after putting a comma to blacklist a disk # If value is empty, the default regex will be used. - name: DISCOVER_DAEMON_UDEV_BLACKLIST value: "(?i)dm-[0-9]+,(?i)rbd[0-9]+,(?i)nbd[0-9]+" # Time to wait until the node controller will move Rook pods to other # nodes after detecting an unreachable node. # Pods affected by this setting are: # mgr, rbd, mds, rgw, nfs, PVC based mons and osds, and ceph toolbox # The value used in this variable replaces the default value of 300 secs # added automatically by k8s as Toleration for # <node.kubernetes.io/unreachable> # The total amount of time to reschedule Rook pods in healthy nodes # before detecting a <not ready node> condition will be the sum of: # --> node-monitor-grace-period: 40 seconds (k8s kube-controller-manager flag) # --> ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS: 5 seconds - name: ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS value: "5" # The name of the node to pass with the downward API - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName # The pod name to pass with the downward API - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name # The pod namespace to pass with the downward API - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace # Recommended resource requests and limits, if desired #resources: # limits: # cpu: 500m # memory: 256Mi # requests: # cpu: 100m # memory: 128Mi # Uncomment it to run lib bucket provisioner in multithreaded mode #- name: LIB_BUCKET_PROVISIONER_THREADS # value: "5" # Uncomment it to run rook operator on the host network #hostNetwork: true volumes: - name: rook-config emptyDir: {} - name: default-config-dir emptyDir: {} - name: webhook-cert emptyDir: {}# OLM: END OPERATOR DEPLOYMENT- 执行日志
[root@vm-k8s-master examples]# kubectl apply -f operator.yamlconfigmap/rook-ceph-operator-config createddeployment.apps/rook-ceph-operator created3.6.4.4、创立rook-ceph cluster
#以后门路：rook/deploy/exampleskubectl apply -f cluster.yaml- 查看pod状态
#查看pod状态[root@vm-k8s-master examples]# kubectl get pods -n rook-cephNAME READY STATUS RESTARTS AGEcsi-cephfsplugin-2qr5s 2/3 ImagePullBackOff 0 5m6scsi-cephfsplugin-mwzc6 2/3 ErrImagePull 0 5m6scsi-cephfsplugin-provisioner-5dc9cbcc87-gpbbg 2/6 ErrImagePull 0 5m6scsi-cephfsplugin-provisioner-5dc9cbcc87-zjflm 0/6 ContainerCreating 0 5m6scsi-cephfsplugin-qlqb5 0/3 ErrImagePull 0 5m6scsi-rbdplugin-4hwfg 2/3 ErrImagePull 0 5m6scsi-rbdplugin-8hzz4 2/3 ImagePullBackOff 0 5m6scsi-rbdplugin-f6d6z 2/3 ImagePullBackOff 0 5m6scsi-rbdplugin-provisioner-58f584754c-b57xb 2/6 ErrImagePull 0 5m6scsi-rbdplugin-provisioner-58f584754c-t87ts 0/6 ContainerCreating 0 5m6srook-ceph-mon-a-7dd7f95f87-mzrbt 1/1 Running 0 5m52srook-ceph-mon-b-b7645dd5b-ls7nm 1/1 Running 0 4m56srook-ceph-mon-c-5dc96ff9c-zgq6x 0/1 Init:0/2 0 2m35srook-ceph-operator-846695c777-jzg4z 1/1 Running 0 15m#[root@vm-k8s-master examples]# kubectl describe pod -n rook-ceph csi-cephfsplugin-2qr5sEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 9m45s default-scheduler Successfully assigned rook-ceph/csi-cephfsplugin-2qr5s to vm-k8s-worker02 Normal Pulled 9m14s kubelet Container image "quay.io/cephcsi/cephcsi:v3.5.1" already present on machine Normal Created 9m14s kubelet Created container liveness-prometheus Normal Started 9m14s kubelet Started container liveness-prometheus Normal Pulled 9m14s kubelet Container image "quay.io/cephcsi/cephcsi:v3.5.1" already present on machine Normal Created 9m14s kubelet Created container csi-cephfsplugin Normal Started 9m14s kubelet Started container csi-cephfsplugin Normal Pulling 7m33s (x3 over 9m44s) kubelet Pulling image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" Warning Failed 6m44s (x3 over 9m14s) kubelet Failed to pull image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0": rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Warning Failed 6m44s (x3 over 9m14s) kubelet Error: ErrImagePull Warning Failed 6m17s (x5 over 9m14s) kubelet Error: ImagePullBackOff Normal BackOff 4m33s (x8 over 9m14s) kubelet Back-off pulling image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0"3.6.4.5、创立存储类
创立存储类，为k8s集群提供动态创建pv的能力
#以后门路：rook/deploy/examples/csi/rbdkubectl apply -f storageclass.yaml[root@vm-k8s-master rbd]# kubectl get scNAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGErook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate true 65s- storageclass.yaml资源文件内容
apiVersion: ceph.rook.io/v1kind: CephBlockPoolmetadata: name: replicapool namespace: rook-ceph # namespace:clusterspec: failureDomain: host replicated: size: 3 # Disallow setting pool with replica 1, this could lead to data loss without recovery. # Make sure you're *ABSOLUTELY CERTAIN* that is what you want requireSafeReplicaSize: true # gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool # for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size #targetSizeRatio: .5---apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: rook-ceph-block# Change "rook-ceph" provisioner prefix to match the operator namespace if neededprovisioner: rook-ceph.rbd.csi.ceph.comparameters: # clusterID is the namespace where the rook cluster is running # If you change this namespace, also change the namespace below where the secret namespaces are defined clusterID: rook-ceph # namespace:cluster # If you want to use erasure coded pool with RBD, you need to create # two pools. one erasure coded and one replicated. # You need to specify the replicated pool here in the `pool` parameter, it is # used for the metadata of the images. # The erasure coded pool must be set as the `dataPool` parameter below. #dataPool: ec-data-pool pool: replicapool # (optional) mapOptions is a comma-separated list of map options. # For krbd options refer # https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options # For nbd options refer # https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options # mapOptions: lock_on_read,queue_depth=1024 # (optional) unmapOptions is a comma-separated list of unmap options. # For krbd options refer # https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options # For nbd options refer # https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options # unmapOptions: force # RBD image format. Defaults to "2". imageFormat: "2" # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature. imageFeatures: layering # The secrets contain Ceph admin credentials. These are generated automatically by the operator # in the same namespace as the cluster. csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph # namespace:cluster csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph # namespace:cluster csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # namespace:cluster # Specify the filesystem type of the volume. If not specified, csi-provisioner # will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock # in hyperconverged settings where the volume is mounted on the same node as the osds. csi.storage.k8s.io/fstype: ext4# uncomment the following to use rbd-nbd as mounter on supported nodes# **IMPORTANT**: CephCSI v3.4.0 onwards a volume healer functionality is added to reattach# the PVC to application pod if nodeplugin pod restart.# Its still in Alpha support. Therefore, this option is not recommended for production use.#mounter: rbd-nbdallowVolumeExpansion: truereclaimPolicy: Delete3.6.4.6、创立rook官网提供的mysql测试服务
rook官网提供的mysql测试服务，依赖于storageclass创立的pv长久化
#以后门路：rook/deploy/exampleskubectl apply -f mysql.yaml[root@vm-k8s-master examples]# kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEmysql-pv-claim Bound pvc-effe80d6-a1ae-42a7-b112-d11e50c06fb8 20Gi RWO rook-ceph-block 4m27s- mysql.yaml资源文件内容
apiVersion: v1kind: Servicemetadata: name: wordpress-mysql labels: app: wordpressspec: ports: - port: 3306 selector: app: wordpress tier: mysql clusterIP: None---apiVersion: v1kind: PersistentVolumeClaimmetadata: name: mysql-pv-claim labels: app: wordpressspec: storageClassName: rook-ceph-block accessModes: - ReadWriteOnce resources: requests: storage: 20Gi---apiVersion: apps/v1kind: Deploymentmetadata: name: wordpress-mysql labels: app: wordpress tier: mysqlspec: selector: matchLabels: app: wordpress tier: mysql strategy: type: Recreate template: metadata: labels: app: wordpress tier: mysql spec: containers: - image: mysql:5.6 name: mysql env: - name: MYSQL_ROOT_PASSWORD value: changeme ports: - containerPort: 3306 name: mysql volumeMounts: - name: mysql-persistent-storage mountPath: /var/lib/mysql volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: mysql-pv-claim3.6.4.7、创立toolbox
默认启动的Ceph集群，是开启Ceph认证的，这样你登陆Ceph组件所在的Pod里，是没法去获取集群状态，以及执行CLI命令，这时须要部署Ceph toolbox。
#以后门路：rook/deploy/exampleskubectl apply -f toolbox.yaml3.6.4.8、创立rook官网提供的wordpress测试服务
rook官网提供的wordpress测试服务，依赖于storageclass创立的pv长久化
#以后门路：rook/deploy/exampleskubectl apply -f wordpress.yaml[root@vm-k8s-master examples]# kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEmysql-pv-claim Bound pvc-effe80d6-a1ae-42a7-b112-d11e50c06fb8 20Gi RWO rook-ceph-block 5m49swp-pv-claim Bound pvc-240ea773-ace2-4a9a-99ff-89f52bbd8084 20Gi RWO rook-ceph-block 7s阐明：这里的pv会主动创立，当提交了蕴含StorageClass字段的PVC之后，Kubernetes 就会依据这个StorageClass创立出对应的PV，这是用到的是Dynamic Provisioning机制来动态创建pv。
- 拜访wordpress
[root@vm-k8s-master examples]# kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEkubernetes ClusterIP 10.1.0.1 <none> 443/TCP 26hwordpress LoadBalancer 10.1.43.219 <pending> 80:31225/TCP 14mwordpress-mysql ClusterIP None <none> 3306/TCP 20m