关于etcd:K8s-系列三-如何配置-etcd-https-证书

51次阅读

共计 7846 个字符，预计需要花费 20 分钟才能阅读完成。

在 K8s 中，kube-apiserver 应用 etcd 对 REST object 资源进行长久化存储，本文介绍如何配置生成自签 https 证书，搭建 etcd 集群给 apiserver 应用，并附相干坑点记录。

cd /data/work

wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl_1.6.0_linux_amd64 -O cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssljson_1.6.0_linux_amd64 -O cfssljson
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl-certinfo_1.6.0_linux_amd64 -O cfssl-certinfo

chmod +x cfssl*
mv cfssl* /usr/local/bin/


chmod +x cfssl*
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

cat > ca-csr.json <<EOF
{
  "CN": "etcd-ca",
  "key": {
      "algo": "rsa",
      "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd-ca",
      "OU": "etcd-ca"
    }
  ],
  "ca": {"expiry": "87600h"}
}
EOF


cfssl gencert -initca ca-csr.json | cfssljson -bare ca

=> 会生成：ca-key.pem, ca.csr, ca.pem

cat > ca-config.json <<EOF
{
  "signing": {
      "default": {"expiry": "87600h"},
      "profiles": {
          "etcd-ca": {
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          }
      }
  }
}
EOF

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "etcd0-0.etcd",
    "etcd1-0.etcd",
    "etcd2-0.etcd"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "Beijing",
    "L": "Beijing",
    "O": "etcd",
    "OU": "etcd"
  }]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-ca etcd-csr.json | cfssljson  -bare etcd

=> 会生成：etcd-key.pem, etcd.csr, etcd.pem

yaml 文件：https://github.com/k8s-club/etcd-operator

kubectl apply -f etcd-cluster.yaml

dnsutils 装置：https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

kubectl exec -it -n etcd dnsutils -- nslookup etcd

Server:         9.165.x.x
Address:        9.165.x.x#53

Name:   etcd.etcd.svc.cluster.local
Address: 9.165.x.x
Name:   etcd.etcd.svc.cluster.local
Address: 9.165.x.x
Name:   etcd.etcd.svc.cluster.local
Address: 9.165.x.x

kubectl exec -it -n etcd etcd0-0 -- sh

/usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://etcd0-0.etcd:2379,https://etcd1-0.etcd:2379,https://etcd2-0.etcd:2379 endpoint health

+---------------------------+--------+-------------+-------+
|         ENDPOINT          | HEALTH |    TOOK     | ERROR |
+---------------------------+--------+-------------+-------+
| https://etcd0-0.etcd:2379 |   true | 13.551982ms |       |
| https://etcd1-0.etcd:2379 |   true | 13.540498ms |       |
| https://etcd2-0.etcd:2379 |   true | 23.119639ms |       |
+---------------------------+--------+-------------+-------+

/usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://etcd0-0.etcd:2379,https://etcd1-0.etcd:2379,https://etcd2-0.etcd:2379 endpoint status

+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://etcd0-0.etcd:2379 | 4dde210279eea33a |  3.4.13 |   20 kB |      true |      false |         2 |          9 |                  9 |        |
| http://etcd1-0.etcd:2379 | 20669865d12a473b |  3.4.13 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
| http://etcd2-0.etcd:2379 | 3f17922d1ed63113 |  3.4.13 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

kubectl exec -it -n etcd etcd0-0 -- sh

/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem put hello world
OK

/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get hello
hello
world

查看所有 keys:
/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get "" --keys-only --prefix
hello

查看所有 key-val:
/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get "" --prefix
hello
world

cat > apiserver-csr.json <<EOF
{
  "CN": "apiserver",
  "hosts": ["*.etcd"],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "Beijing",
    "L": "Beijing",
    "O": "apiserver",
    "OU": "apiserver"
  }]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-ca apiserver-csr.json | cfssljson  -bare apiserver

=> 会生成：apiserver-key.pem, apiserver.csr, apiserver.pem

apiserver.yaml：通过 ConfigMap 将生成的 *.pem 证书挂载给 apiserver 应用

containers:
  - image: xxxxx:latest
    args:
    - --etcd-servers=https://etcd0-0.etcd:2379
    - --etcd-cafile=/etc/kubernetes/certs/kube-apiserver-etcd-ca.crt
    - --etcd-certfile=/etc/kubernetes/certs/kube-apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/certs/kube-apiserver-etcd-client.key

kubectl apply -f apiserver.yaml

log:
etcd0-0:
{"level":"warn","ts":"2021-08-19T11:55:07.755Z","caller":"embed/config_logging.go:279","msg":"rejected connection","remote-addr":"127.0.0.1:41226","server-name":"","error":"tls: first record does not look like a TLS handshake"}


etcd1-0:
{"level":"info","ts":"2021-08-19T11:54:16.830Z","caller":"embed/serve.go:191","msg":"serving client traffic securely","address":"[::]:2379"}
{"level":"info","ts":"2021-08-19T11:54:16.838Z","caller":"etcdserver/server.go:716","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"30dd90df9a304e97","forward-ticks":18,"forward-duration":"4.5s","election-ticks":20,"election-timeout":"5s","active-remote-members":2}
{"level":"info","ts":"2021-08-19T11:54:16.867Z","caller":"membership/cluster.go:558","msg":"set initial cluster version","cluster-id":"80c7f1f6c2848777","local-member-id":"30dd90df9a304e97","cluster-version":"3.4"}
{"level":"info","ts":"2021-08-19T11:54:16.867Z","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.4"}


etcd2-0:
{"level":"warn","ts":"2021-08-19T11:54:17.782Z","caller":"embed/config_logging.go:270","msg":"rejected connection","remote-addr":"9.165.x.x:40180","server-name":"etcd2-0.etcd","ip-addresses":["0.0.0.0","127.0.0.1"],"dns-names":["etcd0-0.etcd","etcd1-0.etcd","etcd2-0.etcd"],"error":"tls: \"9.165.x.x\"does not match any of DNSNames [\"etcd0-0.etcd\"\"etcd1-0.etcd\"\"etcd2-0.etcd\"] (lookup etcd1-0.etcd on 9.165.x.x:53: no such host)"}

解决：重新配置正确的 hosts 域名

"hosts": [
    "127.0.0.1",
    "etcd0-0.etcd",
    "*.etcd" // 容许 * 泛域名，但不能为空 "" 或 *
  ],

举荐设置 *.xxx.ns.svc，这样扩容后也不须要重签证书

参考：https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

Go 代码参考如下：

func genEtcdWildcardDnsName(namespace, serviceName string) []string {return []string{fmt.Sprintf("%s.%s.%s", serviceName, namespace, "svc"),
        fmt.Sprintf("*.%s.%s.%s", serviceName, namespace, "svc"),
        fmt.Sprintf("%s.%s.%s", serviceName, namespace, DnsBase),
        fmt.Sprintf("*.%s.%s.%s", serviceName, namespace, DnsBase),
    }
}

# /usr/local/bin/etcdctl put hello world
{"level":"warn","ts":"2021-08-19T12:32:11.200Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-05ed1825-e70f-492a-af94-03c633d0affc/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection closed"}
Error: context deadline exceeded

解决：etcdctl 须要带证书拜访

/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem put hello world

先通过 http 建设了 cluster，而后再用自签证书 https 来建设，这样就会报错：

tls: first record does not look like a TLS handshake

通过验证：无论是从 http => https，还是从 https => http 的切换都会报这个错，因为一旦建设 cluster 胜利，则把连贯的协定 (http/https) 写入到 etcd 存储里了，不能再更改连贯协定。

解决：如果真正遇到须要切换协定，可尝试上面形式

容许删除数据：删除后从新建设 cluster
不容许删数据：能够尝试采纳 snapshot & restore 进行快照与复原操作

通过验证，是能够间接应用 etcd 证书的，但生产上不倡议这样应用。

生产上倡议对 apiserver(或其余利用) 独自生成证书，可应用泛域名 (*.xx.xx)、不同过期工夫等形式灵便配置，也更有利于集群管控。

正文完

etcd

发表至： etcd

2021-08-26

0

关于etcd:etcd单点迁移到高可用集群

关于etcd:不使用-K8s-API如何直接修改-etcd-数据

关于etcd:K8s-系列三-如何配置-etcd-https-证书

关于etcd:CNCF-宣布-etcd-项目毕业

关于数据挖掘:思迈特软件Smartbi自助式BI工具为什么会受到企业的如此青睐

关于etcd:K8s-系列三-如何配置-etcd-https-证书

1. 装置 cfssl 工具

2. 创立 ca 证书

3. 配置 ca 证书策略

4. 配置 etcd 申请 csr

5. 生成 etcd 证书

6. 创立 etcd cluster

7. 查看 DNS 解析

8. 查看 etcd 集群状态

9. 验证 etcd 读写

10. 配置 apiserver 申请 csr

11. 生成 apiserver 证书

12. 创立 extension-apiserver

13. 坑点记录

13.1 证书 hosts 不对

13.2 证书 hosts 配置坑点

13.3 dns 设置参考

13.4 leader/follower 曾经建设胜利了，但拜访报错

13.5 http 与 https 之间不能切换

13.6 apiserver 可间接应用第 5 步生成的 etcd 证书吗？

Just My Socks（注册教程内含优惠码）

关于etcd:K8s-系列三-如何配置-etcd-https-证书

1. 装置 cfssl 工具

2. 创立 ca 证书

3. 配置 ca 证书策略

4. 配置 etcd 申请 csr

5. 生成 etcd 证书

6. 创立 etcd cluster

7. 查看 DNS 解析

8. 查看 etcd 集群状态

9. 验证 etcd 读写

10. 配置 apiserver 申请 csr

11. 生成 apiserver 证书

12. 创立 extension-apiserver

13. 坑点记录

13.1 证书 hosts 不对

13.2 证书 hosts 配置坑点

13.3 dns 设置参考

13.4 leader/follower 曾经建设胜利了，但拜访报错

13.5 http 与 https 之间不能切换

13.6 apiserver 可间接应用第 5 步生成的 etcd 证书吗？

Just My Socks（注册教程 内含优惠码）

Just My Socks（注册教程内含优惠码）