共计 21372 个字符,预计需要花费 54 分钟才能阅读完成。
1、资源筹备
零碎:CentOS 7.9.2009
主机名 | ip | 组件 |
---|---|---|
k8s-master1 | 192.168.219.161 | kube-apiserver,kube-controller-manager,kube-scheduler,etcd |
k8s-node1 | 192.168.219.162 | kubelet,kube-proxy,docker,etcd |
k8s-node2 | 192.168.219.163 | kubelet,kube-proxy,docker,etcd |
2、软件信息
软件 | 版本 |
---|---|
docker | 19.03.11 |
kubernetes | 1.18.18 |
3、docker 部署
参考:https://segmentfault.com/a/11…
须要删除 ”exec-opts”: [“native.cgroupdriver=systemd”],
cat > /etc/docker/daemon.json << EOF | |
{ | |
"log-driver": "json-file", | |
"log-opts": {"max-size": "100m"}, | |
"storage-driver": "overlay2", | |
"storage-opts": ["overlay2.override_kernel_check=true"], | |
"registry-mirrors": ["https://n0k07cz2.mirror.aliyuncs.com"] | |
} | |
EOF |
4、系统配置(master、node 节点)
参考:https://segmentfault.com/a/11…
5、部署 Etcd 集群
Etcd 是一个分布式键值存储系统,Kubernetes 应用 Etcd 进行数据存储,所以先筹备一个 Etcd 数据库,为解决 Etcd 单点故障,应采纳集群形式部署,这里应用 3 台组建集群,可容忍 1 台机器故障,当然,你也能够应用 5 台组建集群,可容忍 2 台机器故障。
节点名称 | IP |
---|---|
etcd-1 | 192.168.219.161 |
etcd-2 | 192.168.219.162 |
etcd-3 | 192.168.219.163 |
注:为了节俭机器,这里与 K8s 节点机器复用。也能够独立于 k8s 集群之外部署,只有 apiserver 能连贯到就行。
5.1、筹备 cfssl 证书生成工具
cfssl 是一个开源的证书管理工具,应用 json 文件生成证书,相比 openssl 更方便使用。
找任意一台服务器操作,这里用 Master 节点。
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 | |
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 | |
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 | |
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 | |
mv cfssl_linux-amd64 /usr/local/bin/cfssl | |
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson | |
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo |
5.2、生成 Etcd 证书
5.2.1、创立工作目录
mkdir -p ~/TLS/etcd | |
cd ~/TLS/etcd |
5.2.2、自签证书颁发机构(CA)
自签 CA:
cat > ca-config.json << EOF | |
{ | |
"signing": { | |
"default": {"expiry": "87600h"}, | |
"profiles": { | |
"www": { | |
"expiry": "87600h", | |
"usages": [ | |
"signing", | |
"key encipherment", | |
"server auth", | |
"client auth" | |
] | |
} | |
} | |
} | |
} | |
EOF | |
cat > ca-csr.json << EOF | |
{ | |
"CN": "etcd CA", | |
"key": { | |
"algo": "rsa", | |
"size": 2048 | |
}, | |
"names": [ | |
{ | |
"C": "CN", | |
"L": "Beijing", | |
"ST": "Beijing" | |
} | |
] | |
} | |
EOF |
生成证书:
# 生成证书 | |
cfssl gencert -initca ca-csr.json | cfssljson -bare ca - | |
# 查看证书 | |
ls *pem | |
ca-key.pem ca.pem |
5.2.3、应用自签 CA 签发 Etcd HTTPS 证书
创立证书申请文件:
cat > server-csr.json << EOF | |
{ | |
"CN": "etcd", | |
"hosts": [ | |
"192.168.219.161", | |
"192.168.219.162", | |
"192.168.219.163" | |
], | |
"key": { | |
"algo": "rsa", | |
"size": 2048 | |
}, | |
"names": [ | |
{ | |
"C": "CN", | |
"L": "BeiJing", | |
"ST": "BeiJing" | |
} | |
] | |
} | |
EOF |
注:上述文件 hosts 字段中 IP 为所有 etcd 节点的集群外部通信 IP,一个都不能少!为了不便前期扩容能够多写几个预留的 IP。
生成证书:
# 生成证书 | |
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server | |
# 查看证书 | |
ls server*pem | |
server-key.pem server.pem |
5.3、从 Github 下载二进制文件
下载地址:https://github.com/etcd-io/et…
5.4、部署 Etcd 集群
以下在节点 1 上操作,为简化操作,待会将节点 1 生成的所有文件拷贝到节点 2 和节点 3
5.4.1、创立工作目录并解压二进制包
mkdir /opt/etcd/{bin,cfg,ssl} -p | |
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz | |
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/ |
5.4.2、创立 etcd 配置文件
cat > /opt/etcd/cfg/etcd.conf << EOF | |
#[Member] | |
ETCD_NAME="etcd-1" # 批改此处,节点 2 改为 etcd-2,节点 3 改为 etcd-3 | |
ETCD_DATA_DIR="/var/lib/etcd/default.etcd" | |
ETCD_LISTEN_PEER_URLS="https://192.168.219.161:2380" # 批改此处为以后服务器 IP | |
ETCD_LISTEN_CLIENT_URLS="https://192.168.219.161:2379" # 批改此处为以后服务器 IP | |
#[Clustering] | |
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.219.161:2380" # 批改此处为以后服务器 IP | |
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.219.161:2379" # 批改此处为以后服务器 IP | |
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.219.161:2380,etcd-2=https://192.168.219.162:2380,etcd-3=https://192.168.219.163:2380" | |
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" | |
ETCD_INITIAL_CLUSTER_STATE="new" | |
EOF |
阐明:
- ETCD_NAME:节点名称,集群中惟一
- ETCD_DATA_DIR:数据目录
- ETCD_LISTEN_PEER_URLS:集群通信监听地址
- ETCD_LISTEN_CLIENT_URLS:客户端拜访监听地址
- ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址
- ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
- ETCD_INITIAL_CLUSTER:集群节点地址
- ETCD_INITIAL_CLUSTER_TOKEN:集群 Token
- ETCD_INITIAL_CLUSTER_STATE:退出集群的以后状态,new 是新集群,existing 示意退出已有集群
5.4.3、systemd 治理 etcd
cat > /usr/lib/systemd/system/etcd.service << EOF | |
[Unit] | |
Description=Etcd Server | |
After=network.target | |
After=network-online.target | |
Wants=network-online.target | |
[Service] | |
Type=notify | |
EnvironmentFile=/opt/etcd/cfg/etcd.conf | |
ExecStart=/opt/etcd/bin/etcd \ | |
--cert-file=/opt/etcd/ssl/server.pem \ | |
--key-file=/opt/etcd/ssl/server-key.pem \ | |
--peer-cert-file=/opt/etcd/ssl/server.pem \ | |
--peer-key-file=/opt/etcd/ssl/server-key.pem \ | |
--trusted-ca-file=/opt/etcd/ssl/ca.pem \ | |
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \ | |
--logger=zap | |
Restart=on-failure | |
LimitNOFILE=65536 | |
[Install] | |
WantedBy=multi-user.target | |
EOF |
5.4.4、拷贝方才生成的证书
把方才生成的证书拷贝到配置文件中的门路
cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/
5.4.5、启动并设置开机启动
systemctl daemon-reload | |
systemctl start etcd | |
systemctl enable etcd |
5.4.6、将下面节点 1 所有生成的文件拷贝到节点 2 和节点 3
scp -r /opt/etcd/ root@192.168.219.162:/opt/ | |
scp /usr/lib/systemd/system/etcd.service root@192.168.219.162:/usr/lib/systemd/system/ | |
scp -r /opt/etcd/ root@192.168.219.163:/opt/ | |
scp /usr/lib/systemd/system/etcd.service root@192.168.219.163:/usr/lib/systemd/system/ |
而后在节点 2 和节点 3 别离批改 /opt/etcd/cfg/etcd.conf 配置文件中的节点名称和以后服务器 IP。
最初启动 etcd 并设置开机启动,同上。
5.5、查看集群状态
ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.219.161:2379,https://192.168.219.162:2379,https://192.168.219.163:2379" endpoint health
输入:
https://192.168.219.161:2379 is healthy: successfully committed proposal: took = 8.154404ms | |
https://192.168.219.163:2379 is healthy: successfully committed proposal: took = 9.044117ms | |
https://192.168.219.162:2379 is healthy: successfully committed proposal: took = 10.000825ms |
如果输入下面信息,就阐明集群部署胜利。如果有问题第一步先看日志:/var/log/message 或 journalctl -u etcd
6、部署 Master Node
6.1、创立工作目录
mkdir -p ~/TLS/k8s | |
cd ~/TLS/k8s |
6.2、生成 kube-apiserver 证书
6.2.1、自签证书颁发机构(CA)
自签 CA:
cat > ca-config.json << EOF | |
{ | |
"signing": { | |
"default": {"expiry": "87600h"}, | |
"profiles": { | |
"kubernetes": { | |
"expiry": "87600h", | |
"usages": [ | |
"signing", | |
"key encipherment", | |
"server auth", | |
"client auth" | |
] | |
} | |
} | |
} | |
} | |
EOF | |
cat > ca-csr.json << EOF | |
{ | |
"CN": "kubernetes", | |
"key": { | |
"algo": "rsa", | |
"size": 2048 | |
}, | |
"names": [ | |
{ | |
"C": "CN", | |
"L": "Beijing", | |
"ST": "Beijing", | |
"O": "k8s", | |
"OU": "System" | |
} | |
] | |
} | |
EOF |
生成证书:
# 生成证书 | |
cfssl gencert -initca ca-csr.json | cfssljson -bare ca - | |
# 查看证书 | |
ls *pem | |
ca-key.pem ca.pem |
6.2.2、应用自签 CA 签发 kube-apiserver HTTPS 证书
创立证书申请文件:
cat > server-csr.json << EOF | |
{ | |
"CN": "kubernetes", | |
"hosts": [ | |
"10.0.0.1", | |
"127.0.0.1", | |
"192.168.219.161", | |
"192.168.219.162", | |
"192.168.219.163", | |
"192.168.219.164", | |
"192.168.219.181", | |
"192.168.219.182", | |
"192.168.219.188", | |
"kubernetes", | |
"kubernetes.default", | |
"kubernetes.default.svc", | |
"kubernetes.default.svc.cluster", | |
"kubernetes.default.svc.cluster.local" | |
], | |
"key": { | |
"algo": "rsa", | |
"size": 2048 | |
}, | |
"names": [ | |
{ | |
"C": "CN", | |
"L": "BeiJing", | |
"ST": "BeiJing", | |
"O": "k8s", | |
"OU": "System" | |
} | |
] | |
} | |
EOF |
注:上述文件 hosts 字段中 IP 为所有 Master/LB/VIP IP,一个都不能少!为了不便前期扩容能够多写几个预留的 IP。
6.2.3、生成证书
# 生成证书 | |
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server | |
# 查看证书 | |
ls server*pem | |
server-key.pem server.pem |
6.4、从 Github 下载二进制文件
6.4.1、下载
下载地址:https://github.com/kubernetes…
注:关上链接你会发现外面有很多包,下载一个 server 包就够了,蕴含了 Master 和 Worker Node 二进制文件。
6.4.2、解压二进制包
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} | |
tar zxvf kubernetes-server-linux-amd64.tar.gz | |
cd kubernetes/server/bin | |
cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin | |
cp kubectl /usr/bin/ |
6.4.3、部署 kube-apiserver
cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF | |
KUBE_APISERVER_OPTS="--logtostderr=false \\ | |
--v=2 \\ | |
--log-dir=/opt/kubernetes/logs \\ | |
--etcd-servers=https://192.168.219.161:2379,https://192.168.219.162:2379,https://192.168.219.163:2379 \\ | |
--bind-address=192.168.219.161 \\ | |
--secure-port=6443 \\ | |
--advertise-address=192.168.219.161 \\ | |
--allow-privileged=true \\ | |
--service-cluster-ip-range=10.0.0.0/24 \\ | |
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\ | |
--authorization-mode=RBAC,Node \\ | |
--enable-bootstrap-token-auth=true \\ | |
--token-auth-file=/opt/kubernetes/cfg/token.csv \\ | |
--service-node-port-range=30000-32767 \\ | |
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\ | |
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\ | |
--tls-cert-file=/opt/kubernetes/ssl/server.pem \\ | |
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\ | |
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\ | |
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\ | |
--etcd-cafile=/opt/etcd/ssl/ca.pem \\ | |
--etcd-certfile=/opt/etcd/ssl/server.pem \\ | |
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\ | |
--audit-log-maxage=30 \\ | |
--audit-log-maxbackup=3 \\ | |
--audit-log-maxsize=100 \\ | |
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log" | |
EOF |
注:下面两个 \ \ 第一个是本义符,第二个是换行符,应用本义符是为了应用 EOF 保留换行符。
阐明:
- –logtostderr:启用日志
- —v:日志等级
- –log-dir:日志目录
- –etcd-servers:etcd 集群地址
- –bind-address:监听地址
- –secure-port:https 平安端口
- –advertise-address:集群通告地址
- –allow-privileged:启用受权
- –service-cluster-ip-range:Service 虚构 IP 地址段
- –enable-admission-plugins:准入管制模块
- –authorization-mode:认证受权,启用 RBAC 受权和节点自治理
- –enable-bootstrap-token-auth:启用 TLS bootstrap 机制
- –token-auth-file:bootstrap token 文件
- –service-node-port-range:Service nodeport 类型默认调配端口范畴
- –kubelet-client-xxx:apiserver 拜访 kubelet 客户端证书
- –tls-xxx-file:apiserver https 证书
- –etcd-xxxfile:连贯 Etcd 集群证书
- –audit-log-xxx:审计日志
6.4.4、拷贝方才生成的证书
把方才生成的证书拷贝到配置文件中的门路:
cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/
6.4.5、启用 TLS Bootstrapping 机制
TLS Bootstraping:Master apiserver 启用 TLS 认证后,Node 节点 kubelet 和 kube-proxy 要与 kube-apiserver 进行通信,必须应用 CA 签发的无效证书才能够,当 Node 节点很多时,这种客户端证书颁发须要大量工作,同样也会减少集群扩大复杂度。为了简化流程,Kubernetes 引入了 TLS bootstraping 机制来主动颁发客户端证书,kubelet 会以一个低权限用户主动向 apiserver 申请证书,kubelet 的证书由 apiserver 动静签订。所以强烈建议在 Node 上应用这种形式,目前次要用于 kubelet,kube-proxy 还是由咱们对立颁发一个证书。
TLS bootstraping 工作流程:
创立上述配置文件中 token 文件:
cat > /opt/kubernetes/cfg/token.csv << EOF | |
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper" | |
EOF |
格局:token,用户名,UID,用户组
token 也可自行生成替换:
head -c 16 /dev/urandom | od -An -t x | tr -d ' '
6.5、systemd 治理 apiserver
6.5.1、创立 service
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF | |
[Unit] | |
Description=Kubernetes API Server | |
Documentation=https://github.com/kubernetes/kubernetes | |
[Service] | |
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf | |
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS | |
Restart=on-failure | |
[Install] | |
WantedBy=multi-user.target | |
EOF |
6.5.2、启动并设置开机启动
systemctl daemon-reload | |
systemctl start kube-apiserver | |
systemctl enable kube-apiserver |
6.5.3、受权 kubelet-bootstrap 用户容许申请证书
kubectl create clusterrolebinding kubelet-bootstrap \ | |
--clusterrole=system:node-bootstrapper \ | |
--user=kubelet-bootstrap |
6.6、部署 kube-controller-manager
6.6.1、创立配置文件
cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF | |
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\ | |
--v=2 \\ | |
--log-dir=/opt/kubernetes/logs \\ | |
--leader-elect=true \\ | |
--master=127.0.0.1:8080 \\ | |
--bind-address=127.0.0.1 \\ | |
--allocate-node-cidrs=true \\ | |
--cluster-cidr=10.244.0.0/16 \\ | |
--service-cluster-ip-range=10.0.0.0/24 \\ | |
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\ | |
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\ | |
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\ | |
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\ | |
--experimental-cluster-signing-duration=87600h0m0s" | |
EOF |
阐明:
- –master:通过本地非平安本地端口 8080 连贯 apiserver。
- –leader-elect:当该组件启动多个时,主动选举(HA)
- –cluster-signing-cert-file/–cluster-signing-key-file:主动为 kubelet 颁发证书的 CA,与 apiserver 保持一致
6.6.2、systemd 治理 controller-manager
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF | |
[Unit] | |
Description=Kubernetes Controller Manager | |
Documentation=https://github.com/kubernetes/kubernetes | |
[Service] | |
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf | |
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS | |
Restart=on-failure | |
[Install] | |
WantedBy=multi-user.target | |
EOF |
6.6.3、启动并设置开机启动
systemctl daemon-reload | |
systemctl start kube-controller-manager | |
systemctl enable kube-controller-manager |
6.7、部署 kube-scheduler
6.7.1、创立配置文件
cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF | |
KUBE_SCHEDULER_OPTS="--logtostderr=false \ | |
--v=2 \ | |
--log-dir=/opt/kubernetes/logs \ | |
--leader-elect \ | |
--master=127.0.0.1:8080 \ | |
--bind-address=127.0.0.1" | |
EOF |
阐明:
- –master:通过本地非平安本地端口 8080 连贯 apiserver。
- –leader-elect:当该组件启动多个时,主动选举(HA)
6.7.2、systemd 治理 kube-scheduler
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF | |
[Unit] | |
Description=Kubernetes Scheduler | |
Documentation=https://github.com/kubernetes/kubernetes | |
[Service] | |
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf | |
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS | |
Restart=on-failure | |
[Install] | |
WantedBy=multi-user.target | |
EOF |
6.7.3、启动并设置开机启动
systemctl daemon-reload | |
systemctl start kube-scheduler | |
systemctl enable kube-scheduler |
6.8、查看集群状态
所有组件都曾经启动胜利,通过 kubectl 工具查看以后集群组件状态:
[root@localhost ~]# kubectl get cs | |
NAME STATUS MESSAGE ERROR | |
scheduler Healthy ok | |
controller-manager Healthy ok | |
etcd-2 Healthy {"health":"true"} | |
etcd-1 Healthy {"health":"true"} | |
etcd-0 Healthy {"health":"true"} |
如上输入阐明 Master 节点组件运行失常。
7、部署 Worker Node
上面还是在 Master Node 上操作,即同时作为 Worker Node
7.1、创立工作目录并拷贝二进制文件
在所有 worker node 创立工作目录:
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
从 master 节点拷贝:
cd kubernetes/server/bin | |
cp kubelet kube-proxy /opt/kubernetes/bin # 本地拷贝 |
7.2、部署 kubelet
7.2.1、创立配置文件
cat > /opt/kubernetes/cfg/kubelet.conf << EOF | |
KUBELET_OPTS="--logtostderr=false \\ | |
--v=2 \\ | |
--log-dir=/opt/kubernetes/logs \\ | |
--hostname-override=k8s-master1 \\ | |
--network-plugin=cni \\ | |
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\ | |
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\ | |
--config=/opt/kubernetes/cfg/kubelet-config.yml \\ | |
--cert-dir=/opt/kubernetes/ssl \\ | |
--pod-infra-container-image=lizhenliang/pause-amd64:3.0" | |
EOF |
阐明:
- –hostname-override:显示名称,集群中惟一
- –network-plugin:启用 CNI
- –kubeconfig:空门路,会主动生成,前面用于连贯 apiserver
- –bootstrap-kubeconfig:首次启动向 apiserver 申请证书
- –config:配置参数文件
- –cert-dir:kubelet 证书生成目录
- –pod-infra-container-image:治理 Pod 网络容器的镜像
7.2.2、配置参数文件
cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF | |
kind: KubeletConfiguration | |
apiVersion: kubelet.config.k8s.io/v1beta1 | |
address: 0.0.0.0 | |
port: 10250 | |
readOnlyPort: 10255 | |
cgroupDriver: cgroupfs | |
clusterDNS: | |
- 10.0.0.2 | |
clusterDomain: cluster.local | |
failSwapOn: false | |
authentication: | |
anonymous: | |
enabled: false | |
webhook: | |
cacheTTL: 2m0s | |
enabled: true | |
x509: | |
clientCAFile: /opt/kubernetes/ssl/ca.pem | |
authorization: | |
mode: Webhook | |
webhook: | |
cacheAuthorizedTTL: 5m0s | |
cacheUnauthorizedTTL: 30s | |
evictionHard: | |
imagefs.available: 15% | |
memory.available: 100Mi | |
nodefs.available: 10% | |
nodefs.inodesFree: 5% | |
maxOpenFiles: 1000000 | |
maxPods: 110 | |
EOF |
7.2.3、生成 bootstrap.kubeconfig 文件
KUBE_APISERVER="https://192.168.219.161:6443" # apiserver IP:PORT | |
TOKEN="c47ffb939f5ca36231d9e3121a252940" # 与 token.csv 里保持一致 | |
# 生成 kubelet bootstrap kubeconfig 配置文件 | |
kubectl config set-cluster kubernetes \ | |
--certificate-authority=/opt/kubernetes/ssl/ca.pem \ | |
--embed-certs=true \ | |
--server=${KUBE_APISERVER} \ | |
--kubeconfig=bootstrap.kubeconfig | |
kubectl config set-credentials "kubelet-bootstrap" \ | |
--token=${TOKEN} \ | |
--kubeconfig=bootstrap.kubeconfig | |
kubectl config set-context default \ | |
--cluster=kubernetes \ | |
--user="kubelet-bootstrap" \ | |
--kubeconfig=bootstrap.kubeconfig | |
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig |
7.2.4、拷贝到配置文件门路
cp bootstrap.kubeconfig /opt/kubernetes/cfg
7.2.5、systemd 治理 kubelet
cat > /usr/lib/systemd/system/kubelet.service << EOF | |
[Unit] | |
Description=Kubernetes Kubelet | |
After=docker.service | |
[Service] | |
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf | |
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS | |
Restart=on-failure | |
LimitNOFILE=65536 | |
[Install] | |
WantedBy=multi-user.target | |
EOF |
7.2.6、启动并设置开机启动:
systemctl daemon-reload | |
systemctl start kubelet | |
systemctl enable kubelet |
7.3、批准 kubelet 证书申请并退出集群
# 查看 kubelet 证书申请 | |
kubectl get csr | |
NAME AGE SIGNERNAME REQUESTOR CONDITION | |
node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A 6m3s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending | |
# 批准申请 | |
kubectl certificate approve node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A | |
# 查看节点 | |
kubectl get node | |
NAME STATUS ROLES AGE VERSION | |
k8s-master1 NotReady <none> 7s v1.18.18 |
注:因为网络插件还没有部署,节点会没有准备就绪 NotReady
7.4、部署 kube-proxy
7.4.1、创立配置文件
cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF | |
KUBE_PROXY_OPTS="--logtostderr=false \\ | |
--v=2 \\ | |
--log-dir=/opt/kubernetes/logs \\ | |
--config=/opt/kubernetes/cfg/kube-proxy-config.yml" | |
EOF |
7.4.2、配置参数文件
cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF | |
kind: KubeProxyConfiguration | |
apiVersion: kubeproxy.config.k8s.io/v1alpha1 | |
bindAddress: 0.0.0.0 | |
metricsBindAddress: 0.0.0.0:10249 | |
clientConnection: | |
kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig | |
hostnameOverride: k8s-master1 | |
clusterCIDR: 10.0.0.0/24 | |
EOF |
7.4.3、生成 kube-proxy.kubeconfig 文件
切换工作目录:
cd ~/TLS/k8s
创立 kube-proxy 证书申请文件:
cat > kube-proxy-csr.json << EOF | |
{ | |
"CN": "system:kube-proxy", | |
"hosts": [], | |
"key": { | |
"algo": "rsa", | |
"size": 2048 | |
}, | |
"names": [ | |
{ | |
"C": "CN", | |
"L": "BeiJing", | |
"ST": "BeiJing", | |
"O": "k8s", | |
"OU": "System" | |
} | |
] | |
} | |
EOF |
生成证书:
# 生成证书 | |
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy | |
# 查看证书 | |
ls kube-proxy*pem | |
kube-proxy-key.pem kube-proxy.pem |
生成 kubeconfig 文件:
KUBE_APISERVER="https://192.168.219.161:6443" | |
kubectl config set-cluster kubernetes \ | |
--certificate-authority=/opt/kubernetes/ssl/ca.pem \ | |
--embed-certs=true \ | |
--server=${KUBE_APISERVER} \ | |
--kubeconfig=kube-proxy.kubeconfig | |
kubectl config set-credentials kube-proxy \ | |
--client-certificate=./kube-proxy.pem \ | |
--client-key=./kube-proxy-key.pem \ | |
--embed-certs=true \ | |
--kubeconfig=kube-proxy.kubeconfig | |
kubectl config set-context default \ | |
--cluster=kubernetes \ | |
--user=kube-proxy \ | |
--kubeconfig=kube-proxy.kubeconfig | |
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig |
拷贝到配置文件指定门路:
cp kube-proxy.kubeconfig /opt/kubernetes/cfg/
7.4.4、systemd 治理 kube-proxy
cat > /usr/lib/systemd/system/kube-proxy.service << EOF | |
[Unit] | |
Description=Kubernetes Proxy | |
After=network.target | |
[Service] | |
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf | |
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS | |
Restart=on-failure | |
LimitNOFILE=65536 | |
[Install] | |
WantedBy=multi-user.target | |
EOF |
7.4.5、启动并设置开机启动
systemctl daemon-reload | |
systemctl start kube-proxy | |
systemctl enable kube-proxy |
7.5、部署 CNI 网络
7.5.1、先筹备好 CNI 二进制文件
下载地址:https://github.com/containern…
7.5.2、解压二进制包并挪动到默认工作目录
mkdir -p /opt/cni/bin | |
tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin |
7.5.3、部署 CNI 网络
# 下载 kube-flannel.yml | |
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml | |
# 默认镜像地址无法访问,批改为 docker hub 镜像仓库 | |
sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml | |
# 创立 flannel 网络 | |
kubectl apply -f kube-flannel.yml | |
# 查看 pod | |
kubectl get pods -n kube-system | |
NAME READY STATUS RESTARTS AGE | |
kube-flannel-ds-amd64-2pc95 1/1 Running 0 72s | |
# 查看 node | |
kubectl get node | |
NAME STATUS ROLES AGE VERSION | |
k8s-master1 Ready <none> 41m v1.18.18 |
7.6、受权 apiserver 拜访 kubelet
cat > apiserver-to-kubelet-rbac.yaml << EOF | |
apiVersion: rbac.authorization.k8s.io/v1 | |
kind: ClusterRole | |
metadata: | |
annotations: | |
rbac.authorization.kubernetes.io/autoupdate: "true" | |
labels: | |
kubernetes.io/bootstrapping: rbac-defaults | |
name: system:kube-apiserver-to-kubelet | |
rules: | |
- apiGroups: | |
- "" | |
resources: | |
- nodes/proxy | |
- nodes/stats | |
- nodes/log | |
- nodes/spec | |
- nodes/metrics | |
- pods/log | |
verbs: | |
- "*" | |
--- | |
apiVersion: rbac.authorization.k8s.io/v1 | |
kind: ClusterRoleBinding | |
metadata: | |
name: system:kube-apiserver | |
namespace: "" | |
roleRef: | |
apiGroup: rbac.authorization.k8s.io | |
kind: ClusterRole | |
name: system:kube-apiserver-to-kubelet | |
subjects: | |
- apiGroup: rbac.authorization.k8s.io | |
kind: User | |
name: kubernetes | |
EOF | |
kubectl apply -f apiserver-to-kubelet-rbac.yaml |
7.7、新减少 Worker Node
拷贝已部署好的 Node 相干文件到新节点
7.7.1、在 master 节点将 Worker Node 波及文件拷贝到新节点 192.168.219.162/163
scp -r /opt/kubernetes root@192.168.219.162:/opt/ | |
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.219.162:/usr/lib/systemd/system | |
scp -r /opt/cni/ root@192.168.219.162:/opt/ | |
scp /opt/kubernetes/ssl/ca.pem root@192.168.219.162:/opt/kubernetes/ssl |
7.7.2、删除 kubelet 证书和 kubeconfig 文件
rm -rf /opt/kubernetes/cfg/kubelet.kubeconfig | |
rm -rf /opt/kubernetes/ssl/kubelet* |
注:这几个文件是证书申请审批后主动生成的,每个 Node 不同,必须删除从新生成。
7.7.3、批改主机名
vi /opt/kubernetes/cfg/kubelet.conf | |
--hostname-override=k8s-node1 | |
vi /opt/kubernetes/cfg/kube-proxy-config.yml | |
hostnameOverride: k8s-node1 |
7.7.4、启动并设置开机启动
systemctl daemon-reload | |
systemctl start kubelet | |
systemctl enable kubelet | |
systemctl start kube-proxy | |
systemctl enable kube-proxy |
7.7.5、在 Master 上批准新 Node kubelet 证书申请
# 查看 | |
kubectl get csr | |
NAME AGE SIGNERNAME REQUESTOR CONDITION | |
node-csr-4zTjsaVSrhuyhIGqsefxzVoZDCNKei-aE2jyTP81Uro 89s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending | |
# 批准退出 | |
kubectl certificate approve node-csr-4zTjsaVSrhuyhIGqsefxzVoZDCNKei-aE2jyTP81Uro | |
# 查看 node 状态 | |
kubectl get node | |
NAME STATUS ROLES AGE VERSION | |
k8s-master1 Ready <none> 65m v1.18.18 | |
k8s-node1 Ready <none> 12m v1.18.18 | |
k8s-node2 Ready <none> 81s v1.18.18 |
k8s-node2(192.168.219.163)节点同上。记得批改主机名!
8、部署 Dashboard
8.1、部署 Dashboard
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
默认 Dashboard 只能集群外部拜访,批改 Service 为 NodePort 类型,裸露到内部:
vi recommended.yaml | |
kind: Service | |
apiVersion: v1 | |
metadata: | |
labels: | |
k8s-app: kubernetes-dashboard | |
name: kubernetes-dashboard | |
namespace: kubernetes-dashboard | |
spec: | |
ports: | |
- port: 443 | |
targetPort: 8443 | |
nodePort: 30001 | |
type: NodePort | |
selector: | |
k8s-app: kubernetes-dashboard | |
# 创立 flannel 网络 | |
kubectl apply -f recommended.yaml |
查看:
kubectl get pods,svc -n kubernetes-dashboard | |
NAME READY STATUS RESTARTS AGE | |
pod/dashboard-metrics-scraper-694557449d-z8gfb 1/1 Running 0 2m18s | |
pod/kubernetes-dashboard-9774cc786-q2gsx 1/1 Running 0 2m19s | |
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE | |
service/dashboard-metrics-scraper ClusterIP 10.0.0.141 <none> 8000/TCP 2m19s | |
service/kubernetes-dashboard NodePort 10.0.0.239 <none> 443:30001/TCP 2m19s |
拜访地址:https://NodeIP:30001
创立 service account 并绑定默认 cluster-admin 管理员集群角色:
kubectl create serviceaccount dashboard-admin -n kube-system | |
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin | |
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}') |
应用输入的 token 登录 Dashboard。
9、部署 CoreDNS
9.1、下载 coredns.yaml
应用 wget 命令下载下来的 coredns.yaml 文件有语法问题, 所以我这里提供个没有语法问题的 yaml 文件
CoreDNS 用于集群外部 Service 名称解析。
coredns.yaml 文件下载链接
提取码为:pm5t
9.2 创立 coredns
# 创立 | |
kubectl apply -f coredns.yaml | |
# 查看 | |
kubectl get pods -n kube-system | |
NAME READY STATUS RESTARTS AGE | |
coredns-5ffbfd976d-j6shb 1/1 Running 0 32s | |
kube-flannel-ds-amd64-2pc95 1/1 Running 0 38m | |
kube-flannel-ds-amd64-7qhdx 1/1 Running 0 15m | |
kube-flannel-ds-amd64-99cr8 1/1 Running 0 26m |
9.3、DNS 解析测试
解析命令:
kubectl run -it --rm dns-test --image=busybox:1.28.4 sh
输入:
If you don't see a command prompt, try pressing enter. | |
/ # nslookup kubernetes | |
Server: 10.0.0.2 | |
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local | |
Name: kubernetes | |
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local |
解析没问题。
至此,单 Master 集群部署实现,接下来将单 Master 架构扩大为多 Master。