关于etcd:K8s-系列三-如何配置-etcd-https-证书

在 K8s 中，kube-apiserver 应用 etcd 对 REST object 资源进行长久化存储，本文介绍如何配置生成自签 https 证书，搭建 etcd 集群给 apiserver 应用，并附相干坑点记录。

1. 装置 cfssl 工具

cd /data/workwget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl_1.6.0_linux_amd64 -O cfsslwget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssljson_1.6.0_linux_amd64 -O cfssljsonwget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl-certinfo_1.6.0_linux_amd64 -O cfssl-certinfochmod +x cfssl*mv cfssl* /usr/local/bin/chmod +x cfssl*mv cfssl_linux-amd64 /usr/local/bin/cfsslmv cfssljson_linux-amd64 /usr/local/bin/cfssljsonmv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

2. 创立 ca 证书

cat > ca-csr.json <<EOF{  "CN": "etcd-ca",  "key": {      "algo": "rsa",      "size": 2048  },  "names": [    {      "C": "CN",      "ST": "Beijing",      "L": "Beijing",      "O": "etcd-ca",      "OU": "etcd-ca"    }  ],  "ca": {          "expiry": "87600h"  }}EOFcfssl gencert -initca ca-csr.json | cfssljson -bare ca=> 会生成：ca-key.pem, ca.csr, ca.pem

3. 配置 ca 证书策略

cat > ca-config.json <<EOF{  "signing": {      "default": {          "expiry": "87600h"        },      "profiles": {          "etcd-ca": {              "usages": [                  "signing",                  "key encipherment",                  "server auth",                  "client auth"              ],              "expiry": "87600h"          }      }  }}EOF

4. 配置 etcd 申请 csr

cat > etcd-csr.json <<EOF{  "CN": "etcd",  "hosts": [    "127.0.0.1",    "etcd0-0.etcd",    "etcd1-0.etcd",    "etcd2-0.etcd"  ],  "key": {    "algo": "rsa",    "size": 2048  },  "names": [{    "C": "CN",    "ST": "Beijing",    "L": "Beijing",    "O": "etcd",    "OU": "etcd"  }]}EOF

5. 生成 etcd 证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-ca etcd-csr.json | cfssljson  -bare etcd=> 会生成：etcd-key.pem, etcd.csr, etcd.pem

6. 创立 etcd cluster

yaml 文件：https://github.com/k8s-club/etcd-operator

kubectl apply -f etcd-cluster.yaml

7. 查看 DNS 解析

dnsutils 装置：https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

kubectl exec -it -n etcd dnsutils -- nslookup etcdServer:         9.165.x.xAddress:        9.165.x.x#53Name:   etcd.etcd.svc.cluster.localAddress: 9.165.x.xName:   etcd.etcd.svc.cluster.localAddress: 9.165.x.xName:   etcd.etcd.svc.cluster.localAddress: 9.165.x.x

8. 查看 etcd 集群状态

kubectl exec -it -n etcd etcd0-0 -- sh/usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://etcd0-0.etcd:2379,https://etcd1-0.etcd:2379,https://etcd2-0.etcd:2379 endpoint health+---------------------------+--------+-------------+-------+|         ENDPOINT          | HEALTH |    TOOK     | ERROR |+---------------------------+--------+-------------+-------+| https://etcd0-0.etcd:2379 |   true | 13.551982ms |       || https://etcd1-0.etcd:2379 |   true | 13.540498ms |       || https://etcd2-0.etcd:2379 |   true | 23.119639ms |       |+---------------------------+--------+-------------+-------+/usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://etcd0-0.etcd:2379,https://etcd1-0.etcd:2379,https://etcd2-0.etcd:2379 endpoint status+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+| http://etcd0-0.etcd:2379 | 4dde210279eea33a |  3.4.13 |   20 kB |      true |      false |         2 |          9 |                  9 |        || http://etcd1-0.etcd:2379 | 20669865d12a473b |  3.4.13 |   20 kB |     false |      false |         2 |          9 |                  9 |        || http://etcd2-0.etcd:2379 | 3f17922d1ed63113 |  3.4.13 |   20 kB |     false |      false |         2 |          9 |                  9 |        |+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

9. 验证 etcd 读写

kubectl exec -it -n etcd etcd0-0 -- sh/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem put hello worldOK/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get hellohelloworld查看所有 keys:/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get "" --keys-only --prefixhello查看所有 key-val:/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get "" --prefixhelloworld

10. 配置 apiserver 申请 csr

cat > apiserver-csr.json <<EOF{  "CN": "apiserver",  "hosts": [    "*.etcd"  ],  "key": {    "algo": "rsa",    "size": 2048  },  "names": [{    "C": "CN",    "ST": "Beijing",    "L": "Beijing",    "O": "apiserver",    "OU": "apiserver"  }]}EOF

11. 生成 apiserver 证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-ca apiserver-csr.json | cfssljson  -bare apiserver=> 会生成：apiserver-key.pem, apiserver.csr, apiserver.pem

12. 创立 extension-apiserver

apiserver.yaml：通过 ConfigMap 将生成的 *.pem 证书挂载给 apiserver 应用

containers:  - image: xxxxx:latest    args:    - --etcd-servers=https://etcd0-0.etcd:2379    - --etcd-cafile=/etc/kubernetes/certs/kube-apiserver-etcd-ca.crt    - --etcd-certfile=/etc/kubernetes/certs/kube-apiserver-etcd-client.crt    - --etcd-keyfile=/etc/kubernetes/certs/kube-apiserver-etcd-client.key

kubectl apply -f apiserver.yaml

13. 坑点记录

13.1 证书 hosts 不对

log:etcd0-0:{"level":"warn","ts":"2021-08-19T11:55:07.755Z","caller":"embed/config_logging.go:279","msg":"rejected connection","remote-addr":"127.0.0.1:41226","server-name":"","error":"tls: first record does not look like a TLS handshake"}etcd1-0:{"level":"info","ts":"2021-08-19T11:54:16.830Z","caller":"embed/serve.go:191","msg":"serving client traffic securely","address":"[::]:2379"}{"level":"info","ts":"2021-08-19T11:54:16.838Z","caller":"etcdserver/server.go:716","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"30dd90df9a304e97","forward-ticks":18,"forward-duration":"4.5s","election-ticks":20,"election-timeout":"5s","active-remote-members":2}{"level":"info","ts":"2021-08-19T11:54:16.867Z","caller":"membership/cluster.go:558","msg":"set initial cluster version","cluster-id":"80c7f1f6c2848777","local-member-id":"30dd90df9a304e97","cluster-version":"3.4"}{"level":"info","ts":"2021-08-19T11:54:16.867Z","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.4"}etcd2-0:{"level":"warn","ts":"2021-08-19T11:54:17.782Z","caller":"embed/config_logging.go:270","msg":"rejected connection","remote-addr":"9.165.x.x:40180","server-name":"etcd2-0.etcd","ip-addresses":["0.0.0.0","127.0.0.1"],"dns-names":["etcd0-0.etcd","etcd1-0.etcd","etcd2-0.etcd"],"error":"tls: \"9.165.x.x\" does not match any of DNSNames [\"etcd0-0.etcd\" \"etcd1-0.etcd\" \"etcd2-0.etcd\"] (lookup etcd1-0.etcd on 9.165.x.x:53: no such host)"}

解决：重新配置正确的 hosts 域名

13.2 证书 hosts 配置坑点

"hosts": [    "127.0.0.1",    "etcd0-0.etcd",    "*.etcd" // 容许 * 泛域名，但不能为空 "" 或 *  ],

13.3 dns 设置参考

举荐设置 *.xxx.ns.svc，这样扩容后也不须要重签证书

参考：https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

Go 代码参考如下：

func genEtcdWildcardDnsName(namespace, serviceName string) []string {    return []string{        fmt.Sprintf("%s.%s.%s", serviceName, namespace, "svc"),        fmt.Sprintf("*.%s.%s.%s", serviceName, namespace, "svc"),        fmt.Sprintf("%s.%s.%s", serviceName, namespace, DnsBase),        fmt.Sprintf("*.%s.%s.%s", serviceName, namespace, DnsBase),    }}

13.4 leader/follower 曾经建设胜利了，但拜访报错

# /usr/local/bin/etcdctl put hello world{"level":"warn","ts":"2021-08-19T12:32:11.200Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-05ed1825-e70f-492a-af94-03c633d0affc/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection closed"}Error: context deadline exceeded

解决：etcdctl 须要带证书拜访

/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem put hello world

13.5 http 与 https 之间不能切换

先通过 http 建设了 cluster，而后再用自签证书 https 来建设，这样就会报错：

tls: first record does not look like a TLS handshake

通过验证：无论是从 http => https，还是从 https => http 的切换都会报这个错，因为一旦建设 cluster 胜利，则把连贯的协定(http/https) 写入到 etcd 存储里了，不能再更改连贯协定。

解决：如果真正遇到须要切换协定，可尝试上面形式

• 容许删除数据：删除后从新建设 cluster
• 不容许删数据：能够尝试采纳 snapshot & restore 进行快照与复原操作

13.6 apiserver 可间接应用第 5 步生成的 etcd 证书吗？

通过验证，是能够间接应用 etcd 证书的，但生产上不倡议这样应用。

生产上倡议对 apiserver(或其余利用) 独自生成证书，可应用泛域名(*.xx.xx)、不同过期工夫等形式灵便配置，也更有利于集群管控。