一、筹备工作
1.1 环境介绍
本试验采纳kubeasz作为kubernetets环境部署工具，它是一个基于二进制形式部署和利用ansible-playbook实现自动化来疾速部署高可用kubernetets集群的工具，具体介绍请查看kubeasz官网。本试验用到的所有虚拟机默认软件源更新为阿里云的源，操作系统为最小化装置，已预装好vim、net-tools、ssh等常用工具，时钟默认曾经全副同步至阿里云，默认已敞开操作系统自带防火墙。
本试验用到的操作系统版本为：
零碎：Ubuntu Server 20.04 LTS 64位
Kubernetets：v1.26
运行时：containerd v1.6.8
网络：calico
1.2 地址布局
| 角色 | IP地址 | 主机名 | VIP |
|---|---|---|---|
| ETCD | 192.168.10.101 | etcd01 | |
| ETCD | 192.168.10.102 | etcd02 | |
| ETCD | 192.168.10.103 | etcd03 | |
| MATSER/ANSIBLE | 192.168.10.104 | master01 | |
| MASTER | 192.168.10.105 | master02 | |
| MATSER | 192.168.10.106 | master03 | |
| NODE | 192.168.10.107 | node01 | |
| NODE | 192.168.10.108 | node02 | |
| NODE | 192.168.10.109 | node03 | |
| HA | 192.168.10.110 | ha01 | 192.168.10.115 |
| HA | 192.168.10.111 | ha02 | 192.168.10.115 |
二、环境部署
2.1 根底环境搭建
首先解决ubuntu零碎开机默认会把dns配置成127.0.0.53的问题，所有节点都要操作
参考https://blog.csdn.net/qifei71...
具体解决办法
批改 /etc/systemd/resolved.conf 文件[Resolve]DNS=8.8.8.8之后以root身份执行sudo systemctl restart systemd-resolvedsudo systemctl enable systemd-resolvedsudo mv /etc/resolv.conf /etc/resolv.conf.baksudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf批改所有节点时区为上海
root@master01:~# timedatectl set-timezone Asia/Shanghai在master01节点操作
装置ansible:root@k8s-master01:~# apt install ansible配置免密登录，须要配置为master01节点到所有节点都毋庸明码即可登录，只拿master02作为示范，其余节点操作相似。生成秘钥对root@master01:~# ssh-keygen Generating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsaYour public key has been saved in /root/.ssh/id_rsa.pubThe key fingerprint is:SHA256:2kfD/vlpbkKZtG90oujjd90CLZZQHb4buOFSHx4p+so root@k8s-master01The key's randomart image is:+---[RSA 3072]----+| ... || ... || . . || .. .. o || S +o=** || o o =X*+=.|| . . =+o*++.|| oo++.B.o|| oE++O+. |+----[SHA256]-----+散发公钥至各个节点root@master01:~# ssh-copy-id 192.168.10.102/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"The authenticity of host '192.168.10.102 (192.168.10.102)' can't be established.ECDSA key fingerprint is SHA256:LHdJ1aX0Rx+tQlCcGKwIk7aJsFjsUm4/Ze7vwhMqsS8.Are you sure you want to continue connecting (yes/no/[fingerprint])? yes/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keysroot@192.168.10.102's password: Number of key(s) added: 1Now try logging into the machine, with: "ssh '192.168.10.102'"and check to make sure that only the key(s) you wanted were added.配置DNS解析，并且将hosts文件同步到所有结点，只拿master02节点作为示范，其余节点操作相似。
root@master01:~# cat /etc/hosts127.0.0.1 localhost192.168.10.101 etcd01192.168.10.102 etcd02192.168.10.103 etcd03192.168.10.104 master01192.168.10.105 master02192.168.10.106 master03192.168.10.107 node01192.168.10.108 node02192.168.10.109 node03192.168.10.110 ha1192.168.10.111 ha2将hosts文件同步到所有节点（仅应用个别节点举例）root@master01:~# scp /etc/hosts 192.168.10.101:/etc/hostshosts 100% 512 349.0KB/s 00:00 root@master01:~# scp /etc/hosts 192.168.10.102:/etc/hostshosts 100% 512 346.3KB/s 00:00 2.2开始部署环境
在master01节点下载我的项目源码、二进制文件及离线镜像，下载工具脚本ezdown，举例应用kubeasz版本3.5.0
root@master01:~# export release=3.5.0root@master01:~# wget https://github.com/easzlab/kubeasz/releases/download/${release}/ezdownroot@master01:~# chmod +x ezdown下载kubeasz代码、二进制、默认容器镜像root@master01:~# ./ezdown -D下载实现后会提醒INFO Action successed: download_all生成ansible hosts等相干配置文件。
root@master01:/etc/kubeasz# ./ezctl new k8s012022-12-31 22:46:36 DEBUG generate custom cluster files in /etc/kubeasz/clusters/k8s012022-12-31 22:46:36 DEBUG set versions2022-12-31 22:46:36 DEBUG cluster k8s01: files successfully created.2022-12-31 22:46:36 INFO next steps 1: to config '/etc/kubeasz/clusters/k8s01/hosts'2022-12-31 22:46:36 INFO next steps 2: to config '/etc/kubeasz/clusters/k8s01/config.yml'编辑hosts文件
root@k8s-master01:/etc/kubeasz/clusters/k8s01# cat hosts # 'etcd' cluster should have odd member(s) (1,3,5,...)[etcd]192.168.10.101192.168.10.102192.168.10.103# master node(s)[kube_master]192.168.10.104192.168.10.105192.168.10.106# work node(s)[kube_node]192.168.10.107192.168.10.108192.168.10.109# [optional] harbor server, a private docker registry# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one[harbor]#192.168.1.8 NEW_INSTALL=false# [optional] loadbalance for accessing k8s from outside[ex_lb]192.168.10.110 LB_ROLE=master EX_APISERVER_VIP=192.168.10.115 EX_APISERVER_PORT=6443192.168.10.111 LB_ROLE=backup EX_APISERVER_VIP=192.168.10.115 EX_APISERVER_PORT=6443# [optional] ntp server for the cluster[chrony]#192.168.1.1[all:vars]# --------- Main Variables ---------------# Secure port for apiserversSECURE_PORT="6443"# Cluster container-runtime supported: docker, containerd# if k8s version >= 1.24, docker is not supportedCONTAINER_RUNTIME="containerd"# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovnCLUSTER_NETWORK="calico"# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'PROXY_MODE="ipvs"# K8S Service CIDR, not overlap with node(host) networkingSERVICE_CIDR="10.68.0.0/16"# Cluster CIDR (Pod CIDR), not overlap with node(host) networkingCLUSTER_CIDR="172.20.0.0/16"# NodePort RangeNODE_PORT_RANGE="30000-32767"# Cluster DNS DomainCLUSTER_DNS_DOMAIN="cluster.local"# -------- Additional Variables (don't change the default value right now) ---# Binaries Directorybin_dir="/opt/kube/bin"# Deploy Directory (kubeasz workspace)base_dir="/etc/kubeasz"# Directory for a specific clustercluster_dir="{{ base_dir }}/clusters/k8s01"# CA and other components cert/key Directoryca_dir="/etc/kubernetes/ssl"编辑config.yaml文件，次要将coredns与metric-server主动装置关上。
root@master01:/etc/kubeasz/clusters/k8s01# cat config.yml ############################# prepare############################# 可选离线装置零碎软件包 (offline|online)INSTALL_SOURCE: "online"# 可选进行系统安全加固 github.com/dev-sec/ansible-collection-hardeningOS_HARDEN: false############################# role:deploy############################# default: ca will expire in 100 years# default: certs issued by the ca will expire in 50 yearsCA_EXPIRY: "876000h"CERT_EXPIRY: "438000h"# force to recreate CA and other certs, not suggested to set 'true'CHANGE_CA: false# kubeconfig 配置参数CLUSTER_NAME: "cluster1"CONTEXT_NAME: "context-{{ CLUSTER_NAME }}"# k8s versionK8S_VER: "1.26.0"############################# role:etcd############################# 设置不同的wal目录，能够防止磁盘io竞争，进步性能ETCD_DATA_DIR: "/var/lib/etcd"ETCD_WAL_DIR: ""############################# role:runtime [containerd,docker]############################# ------------------------------------------- containerd# [.]启用容器仓库镜像ENABLE_MIRROR_REGISTRY: true# [containerd]根底容器镜像SANDBOX_IMAGE: "easzlab.io.local:5000/easzlab/pause:3.9"# [containerd]容器长久化存储目录CONTAINERD_STORAGE_DIR: "/var/lib/containerd"# ------------------------------------------- docker# [docker]容器存储目录DOCKER_STORAGE_DIR: "/var/lib/docker"# [docker]开启Restful APIENABLE_REMOTE_API: false# [docker]信赖的HTTP仓库INSECURE_REG: '["http://easzlab.io.local:5000"]'############################# role:kube-master############################# k8s 集群 master 节点证书配置，能够增加多个ip和域名（比方减少公网ip和域名）MASTER_CERT_HOSTS: - "10.1.1.1" - "k8s.easzlab.io" - "www.snow.com"# node 节点上 pod 网段掩码长度（决定每个节点最多能调配的pod ip地址）# 如果flannel 应用 --kube-subnet-mgr 参数，那么它将读取该设置为每个节点调配pod网段# https://github.com/coreos/flannel/issues/847NODE_CIDR_LEN: 24############################# role:kube-node############################# Kubelet 根目录KUBELET_ROOT_DIR: "/var/lib/kubelet"# node节点最大pod 数MAX_PODS: 110# 配置为kube组件（kubelet,kube-proxy,dockerd等）预留的资源量# 数值设置详见templates/kubelet-config.yaml.j2KUBE_RESERVED_ENABLED: "no"# k8s 官网不倡议粗率开启 system-reserved, 除非你基于长期监控，理解零碎的资源占用情况；# 并且随着零碎运行工夫，须要适当减少资源预留，数值设置详见templates/kubelet-config.yaml.j2# 零碎预留设置基于 4c/8g 虚机，最小化装置零碎服务，如果应用高性能物理机能够适当减少预留# 另外，集群装置时候apiserver等资源占用会短时较大，倡议至多预留1g内存SYS_RESERVED_ENABLED: "no"############################# role:network [flannel,calico,cilium,kube-ovn,kube-router]############################# ------------------------------------------- flannel# [flannel]设置flannel 后端"host-gw","vxlan"等FLANNEL_BACKEND: "vxlan"DIRECT_ROUTING: false# [flannel] flannel_ver: "v0.19.2"# ------------------------------------------- calico# [calico] IPIP隧道模式可选项有: [Always, CrossSubnet, Never],跨子网能够配置为Always与CrossSubnet(私有云倡议应用always比拟省事，其余的话须要批改各自私有云的网络配置，具体能够参考各个私有云阐明)# 其次CrossSubnet为隧道+BGP路由混合模式能够晋升网络性能，同子网配置为Never即可.CALICO_IPV4POOL_IPIP: "Always"# [calico]设置 calico-node应用的host IP，bgp街坊通过该地址建设，可手工指定也能够主动发现IP_AUTODETECTION_METHOD: "can-reach={{ groups['kube_master'][0] }}"# [calico]设置calico 网络 backend: brid, vxlan, noneCALICO_NETWORKING_BACKEND: "brid"# [calico]设置calico 是否应用route reflectors# 如果集群规模超过50个节点，倡议启用该个性CALICO_RR_ENABLED: false# CALICO_RR_NODES 配置route reflectors的节点，如果未设置默认应用集群master节点 # CALICO_RR_NODES: ["192.168.1.1", "192.168.1.2"]CALICO_RR_NODES: []# [calico]更新反对calico 版本: ["3.19", "3.23"]calico_ver: "v3.23.5"# [calico]calico 主版本calico_ver_main: "{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}"# ------------------------------------------- cilium# [cilium]镜像版本cilium_ver: "1.12.4"cilium_connectivity_check: truecilium_hubble_enabled: falsecilium_hubble_ui_enabled: false# ------------------------------------------- kube-ovn# [kube-ovn]抉择 OVN DB and OVN Control Plane 节点，默认为第一个master节点OVN_DB_NODE: "{{ groups['kube_master'][0] }}"# [kube-ovn]离线镜像tar包kube_ovn_ver: "v1.5.3"# ------------------------------------------- kube-router# [kube-router]私有云上存在限度，个别须要始终开启 ipinip；自有环境能够设置为 "subnet"OVERLAY_TYPE: "full"# [kube-router]NetworkPolicy 反对开关FIREWALL_ENABLE: true# [kube-router]kube-router 镜像版本kube_router_ver: "v0.3.1"busybox_ver: "1.28.4"############################# role:cluster-addon############################# coredns 主动装置dns_install: "yes"corednsVer: "1.9.3"ENABLE_LOCAL_DNS_CACHE: truednsNodeCacheVer: "1.22.13"# 设置 local dns cache 地址LOCAL_DNS_CACHE: "169.254.20.10"# metric server 主动装置metricsserver_install: "yes"metricsVer: "v0.5.2"# dashboard 主动装置dashboard_install: "no"dashboardVer: "v2.7.0"dashboardMetricsScraperVer: "v1.0.8"# prometheus 主动装置prom_install: "no"prom_namespace: "monitor"prom_chart_ver: "39.11.0"# nfs-provisioner 主动装置nfs_provisioner_install: "no"nfs_provisioner_namespace: "kube-system"nfs_provisioner_ver: "v4.0.2"nfs_storage_class: "managed-nfs-storage"nfs_server: "192.168.1.10"nfs_path: "/data/nfs"# network-check 主动装置network_check_enabled: false network_check_schedule: "*/5 * * * *"############################# role:harbor############################# harbor version，残缺版本号HARBOR_VER: "v2.1.5"HARBOR_DOMAIN: "harbor.easzlab.io.local"HARBOR_PATH: /var/dataHARBOR_TLS_PORT: 8443HARBOR_REGISTRY: "{{ HARBOR_DOMAIN }}:{{ HARBOR_TLS_PORT }}"# if set 'false', you need to put certs named harbor.pem and harbor-key.pem in directory 'down'HARBOR_SELF_SIGNED_CERT: true# install extra componentHARBOR_WITH_NOTARY: falseHARBOR_WITH_TRIVY: falseHARBOR_WITH_CLAIR: falseHARBOR_WITH_CHARTMUSEUM: true部署k8s集群
根底环境初始化，筹备CA和根底零碎设置
root@k8s-master01:/etc/kubeasz# ./ezctl setup k8s01 01执行结束后提醒如下信息示意装置没问题PLAY RECAP *************************************************************************************************************************************************************************************************192.168.10.101 : ok=23 changed=6 unreachable=0 failed=0 skipped=97 rescued=0 ignored=0 192.168.10.102 : ok=23 changed=6 unreachable=0 failed=0 skipped=97 rescued=0 ignored=0 192.168.10.103 : ok=23 changed=6 unreachable=0 failed=0 skipped=97 rescued=0 ignored=0 192.168.10.104 : ok=24 changed=6 unreachable=0 failed=0 skipped=96 rescued=0 ignored=0 192.168.10.105 : ok=23 changed=6 unreachable=0 failed=0 skipped=97 rescued=0 ignored=0 192.168.10.106 : ok=23 changed=6 unreachable=0 failed=0 skipped=97 rescued=0 ignored=0 192.168.10.107 : ok=23 changed=6 unreachable=0 failed=0 skipped=97 rescued=0 ignored=0 192.168.10.108 : ok=23 changed=6 unreachable=0 failed=0 skipped=97 rescued=0 ignored=0 192.168.10.109 : ok=23 changed=6 unreachable=0 failed=0 skipped=97 rescued=0 ignored=0 192.168.10.110 : ok=1 changed=0 unreachable=0 failed=0 skipped=78 rescued=0 ignored=0 192.168.10.111 : ok=1 changed=0 unreachable=0 failed=0 skipped=78 rescued=0 ignored=0 localhost : ok=31 changed=21 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0部署etcd集群
root@k8s-master01:/etc/kubeasz# ./ezctl setup k8s01 02执行结束后提醒如下信息示意装置没问题PLAY RECAP *************************************************************************************************************************************************************************************************192.168.10.101 : ok=10 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 192.168.10.102 : ok=8 changed=4 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 192.168.10.103 : ok=8 changed=4 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0验证etcd集群
在etcd01节点执行，返回如下信息证实etcd集群运行失常
root@etcd01:~# export ETCD_IPS="192.168.10.101 192.168.10.102 192.168.10.103"root@etcd01:~# cp -a /opt/kube/bin/etcdctl /usr/local/bin/root@etcd01:~# for ip in ${ETCD_IPS}; do ETCD_API=3 /usr/local/bin/etcdctl --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem endpoint health; donehttps://192.168.10.101:2379 is healthy: successfully committed proposal: took = 35.684337mshttps://192.168.10.102:2379 is healthy: successfully committed proposal: took = 21.125465mshttps://192.168.10.103:2379 is healthy: successfully committed proposal: took = 28.284479ms装置容器运行时
root@k8s-master01:/etc/kubeasz# ./ezctl setup k8s01 03执行结束后提醒如下信息示意装置没问题PLAY RECAP *************************************************************************************************************************************************************************************************192.168.10.104 : ok=2 changed=1 unreachable=0 failed=0 skipped=28 rescued=0 ignored=0 192.168.10.105 : ok=2 changed=1 unreachable=0 failed=0 skipped=25 rescued=0 ignored=0 192.168.10.106 : ok=2 changed=1 unreachable=0 failed=0 skipped=25 rescued=0 ignored=0 192.168.10.107 : ok=2 changed=1 unreachable=0 failed=0 skipped=25 rescued=0 ignored=0 192.168.10.108 : ok=2 changed=1 unreachable=0 failed=0 skipped=25 rescued=0 ignored=0 192.168.10.109 : ok=2 changed=1 unreachable=0 failed=0 skipped=25 rescued=0 ignored=0装置master节点
root@k8s-master01:/etc/kubeasz# ./ezctl setup k8s01 04执行结束后提醒如下信息示意装置没问题PLAY RECAP *************************************************************************************************************************************************************************************************192.168.10.104 : ok=55 changed=36 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 192.168.10.105 : ok=54 changed=36 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 192.168.10.106 : ok=54 changed=36 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0部署node节点
root@master01:/etc/kubeasz# ./ezctl setup k8s01 05执行结束后提醒如下信息示意装置没问题PLAY RECAP *************************************************************************************************************************************************************************************************192.168.10.107 : ok=35 changed=21 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 192.168.10.108 : ok=35 changed=21 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 192.168.10.109 : ok=35 changed=21 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0部署网络服务
root@master01:/etc/kubeasz# ./ezctl setup k8s01 06执行结束后提醒如下信息示意装置没问题PLAY RECAP *************************************************************************************************************************************************************************************************192.168.10.104 : ok=13 changed=7 unreachable=0 failed=0 skipped=39 rescued=0 ignored=0 192.168.10.105 : ok=7 changed=3 unreachable=0 failed=0 skipped=16 rescued=0 ignored=0 192.168.10.106 : ok=7 changed=3 unreachable=0 failed=0 skipped=16 rescued=0 ignored=0 192.168.10.107 : ok=7 changed=3 unreachable=0 failed=0 skipped=16 rescued=0 ignored=0 192.168.10.108 : ok=7 changed=3 unreachable=0 failed=0 skipped=16 rescued=0 ignored=0 192.168.10.109 : ok=7 changed=3 unreachable=0 failed=0 skipped=16 rescued=0 ignored=0验证网络服务,输入如下信息示意网络服务失常。
root@master01:~# calicoctl node statusCalico process is running.IPv4 BGP status+----------------+-------------------+-------+----------+-------------+| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |+----------------+-------------------+-------+----------+-------------+| 192.168.10.105 | node-to-node mesh | up | 15:45:31 | Established || 192.168.10.106 | node-to-node mesh | up | 15:45:30 | Established || 192.168.10.107 | node-to-node mesh | up | 15:45:31 | Established || 192.168.10.108 | node-to-node mesh | up | 15:45:31 | Established || 192.168.10.109 | node-to-node mesh | up | 15:45:30 | Established |+----------------+-------------------+-------+----------+-------------+IPv6 BGP statusNo IPv6 peers found.部署负载平衡服务
root@master01:/etc/kubeasz# ./ezctl setup k8s01 10执行结束后提醒如下信息示意装置没问题PLAY RECAP ***************************************************************************************************192.168.10.110 : ok=17 changed=14 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 192.168.10.111 : ok=16 changed=14 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0部署coredns与metric-server
root@master01:/etc/kubeasz# ./ezctl setup k8s01 07执行结束后提醒如下信息示意装置没问题localhost : ok=8 changed=7 unreachable=0 failed=0 skipped=34 rescued=0 ignored=0三、集群验证
3.1 集群部署实现验证
root@k8s-master01:~# kubectl get noNAME STATUS ROLES AGE VERSION192.168.10.104 Ready,SchedulingDisabled master 51m v1.25.4192.168.10.105 Ready,SchedulingDisabled master 51m v1.25.4192.168.10.106 Ready,SchedulingDisabled master 51m v1.25.4192.168.10.107 Ready node 38m v1.25.4192.168.10.108 Ready node 38m v1.25.4192.168.10.109 Ready node 38m v1.25.43.2 metric-server验证
root@master01:/etc/kubeasz# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 192.168.10.104 111m 11% 1776Mi 70% 192.168.10.105 114m 11% 669Mi 98% 192.168.10.106 130m 13% 657Mi 97% 192.168.10.107 61m 6% 482Mi 71% 192.168.10.108 82m 8% 530Mi 78% 192.168.10.109 59m 5% 513Mi 75%