乐趣区

关于kubernetes:Harbor私有仓库搭建并配置https对接docker与kubernetes

jiezi

一、环境介绍

默认状况下，Harbor 不附带证书。能够在没有平安爱护的状况下部署 Harbor，以便您能够通过 HTTP 连贯到它。在生产环境中，举荐始终应用 HTTPS。要配置 HTTPS，必须创立 SSL 证书。能够应用由受信赖的第三方 CA 签名的证书，也能够应用自签名证书。本文以自签名证书为例。

应用到的各个软件版本 

 操作系统版本：ubuntu 20.04
harbor 版本：v2.5.3-797c3536
docker 版本：20.10.8
kubernetets 版本：1.22.2
harbor 地址：192.168.10.112   域名：harbor.snow.com
master01 地址：192.168.10.100

二、仓库部署

配置主机名与 hosts 文件 

root@harbor:~# cat /etc/hosts
127.0.0.1 localhost
192.168.10.112 harbor.snow.com
批改主机名
root@barbor:~# hostnamectl set-hostname harbor
root@harbor:~# bash
root@harbor:~# hostname
harbor

下载 harbor 安装包 

root@harbor:~#wget https://github.com/goharbor/harbor/releases/download/v2.5.3/harbor-offline-installer-v2.5.3.tgz
解压 harbor 安装包
root@harbor:~# tar xf harbor-offline-installer-v2.5.3.tgz -C /usr/local/src/
装置 docker-compose
root@harbor:~# curl -SL https://github.com/docker/compose/releases/download/v2.7.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose

生成证书颁发机构证书及私钥 

root@harbor:/usr/local/src/harbor/certs# openssl genrsa -out ca.key 4096
Generating RSA private key, 4096 bit long modulus (2 primes)
............................++++
............................................++++
e is 65537 (0x010001)
root@harbor:/usr/local/src/harbor/certs# openssl req -x509 -new -nodes -sha512 -days 3650 \
>  -subj "/C=CN/ST=Shanghai/L=Shanghai/O=SmartX/OU=Lab/CN=harbor.snow.com" \
>  -key ca.key \
>  -out ca.crt
root@harbor:/usr/local/src/harbor/certs# ls
ca.crt  ca.key

生成服务器私钥及证书签名申请（CSR）

root@harbor:/usr/local/src/harbor/certs# openssl genrsa -out harbor.snow.com.key 4096
Generating RSA private key, 4096 bit long modulus (2 primes)
...........++++
.....................................................................++++
e is 65537 (0x010001)

生成证书签名申请
root@harbor:/usr/local/src/harbor/certs# openssl req -sha512 -new \
> -subj "/C=CN/ST=Shanghai/L=Shanghai/O=SmartX/OU=Lab/CN=harbor.snow.com" \
> -key harbor.snow.com.key \
> -out harbor.snow.com.csr
root@harbor:/usr/local/src/harbor/certs# ls
ca.crt  ca.key  harbor.snow.com.csr  harbor.snow.com.key

生成 x509 v3 扩大文件。
无论应用 FQDN 还是 IP 地址连贯到 Harbor 服务器，都必须创立此文件，以便 keyi 为的 Harbor 主机生成合乎主题备用名称 (SAN) 和 x509 v3 的证书扩大要求。

root@harbor:/usr/local/src/harbor/certs# cat v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=harbor.snow.top
DNS.2=snow.top
DNS.3=harbor

应用该 v3.ext 文件为 Harbor 服务器生成证书。

root@harbor:/usr/local/src/harbor/certs# openssl x509 -req -sha512 -days 3650 \
> -extfile v3.ext \
> -CA ca.crt -CAkey ca.key -CAcreateserial \
> -in harbor.snow.com.csr \
> -out harbor.snow.com.crt
Signature ok
subject=C = CN, ST = Shanghai, L = Shanghai, O = SmartX, OU = Lab, CN = harbor.snow.com
Getting CA Private Key

将 harbor.snow.com.crt 转换为 harbor.snow.com.cert , 供 Docker 应用。Docker 守护过程将.crt 文件解释为 CA 证书，.cert 将文件解释为客户端证书。

root@harbor:/usr/local/src/harbor/certs# openssl x509 -inform PEM -in harbor.snow.com.crt -out harbor.snow.com.cert

配置 harbor yml 文件，留神须要批改 hostname 与指定证书地位。

root@harbor:/usr/local/src/harbor# cat harbor.yml |more
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: harbor.snow.com

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /usr/local/src/harbor/certs/harbor.snow.com.cert
  private_key: /usr/local/src/harbor/certs/harbor.snow.com.key

装置 harbor

root@harbor:/usr/local/src/harbor# ./install.sh 

[Step 0]: checking if docker is installed ...

Note: docker version: 20.10.12

[Step 1]: checking docker-compose is installed ...

Note: docker-compose version: 1.25.0

[Step 2]: loading Harbor images ...
Loaded image: goharbor/harbor-portal:v2.5.3
Loaded image: goharbor/harbor-core:v2.5.3
Loaded image: goharbor/redis-photon:v2.5.3
Loaded image: goharbor/prepare:v2.5.3
Loaded image: goharbor/harbor-db:v2.5.3
Loaded image: goharbor/chartmuseum-photon:v2.5.3
Loaded image: goharbor/harbor-jobservice:v2.5.3
Loaded image: goharbor/harbor-registryctl:v2.5.3
Loaded image: goharbor/nginx-photon:v2.5.3
Loaded image: goharbor/notary-signer-photon:v2.5.3
Loaded image: goharbor/harbor-log:v2.5.3
Loaded image: goharbor/harbor-exporter:v2.5.3
Loaded image: goharbor/registry-photon:v2.5.3
Loaded image: goharbor/notary-server-photon:v2.5.3
Loaded image: goharbor/trivy-adapter-photon:v2.5.3


[Step 3]: preparing environment ...

[Step 4]: preparing harbor configs ...
prepare base dir is set to /usr/local/src/harbor
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/db/env
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir



[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-portal ... done
Creating registryctl   ... done
Creating registry      ... done
Creating redis         ... done
Creating harbor-db     ... done
Creating harbor-core   ... done
Creating nginx             ... done
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----

装置完 harbor 后验证 

root@harbor:/usr/local/src/harbor# docker ps 
CONTAINER ID   IMAGE                                COMMAND                  CREATED          STATUS                    PORTS                                                                            NAMES
0304ebf0cddc   goharbor/nginx-photon:v2.5.3         "nginx -g'daemon of…"   38 seconds ago   Up 35 seconds (healthy)   0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp   nginx
10d1ff19120b   goharbor/harbor-jobservice:v2.5.3    "/harbor/entrypoint.…"   38 seconds ago   Up 36 seconds (healthy)                                                                                    harbor-jobservice
2d5e3d8877f8   goharbor/harbor-core:v2.5.3          "/harbor/entrypoint.…"   39 seconds ago   Up 38 seconds (healthy)                                                                                    harbor-core
ee6c12da89d9   goharbor/harbor-db:v2.5.3            "/docker-entrypoint.…"   43 seconds ago   Up 39 seconds (healthy)                                                                                    harbor-db
5b7455e56733   goharbor/redis-photon:v2.5.3         "redis-server /etc/r…"   43 seconds ago   Up 39 seconds (healthy)                                                                                    redis
a0c279ca0206   goharbor/harbor-registryctl:v2.5.3   "/home/harbor/start.…"   43 seconds ago   Up 41 seconds (healthy)                                                                                    registryctl
478f35433f69   goharbor/registry-photon:v2.5.3      "/home/harbor/entryp…"   43 seconds ago   Up 39 seconds (healthy)                                                                                    registry
eb33121f4126   goharbor/harbor-portal:v2.5.3        "nginx -g'daemon of…"   43 seconds ago   Up 39 seconds (healthy)                                                                                    harbor-portal
b0c7ab3a7133   goharbor/harbor-log:v2.5.3           "/bin/sh -c /usr/loc…"   44 seconds ago   Up 43 seconds (healthy)   127.0.0.1:1514->10514/tcp                                                        harbor-log

批改宿主机 hosts 文件，做好 harbor 主机的域名解析，在浏览器应用域名拜访 harbor 服务器发现提醒（疏忽域名，域名是第一次测试的时候用的）

在以后页面用键盘输入 thisisunsafe，不是在地址栏输出，就间接敲键盘就行了，页面即会主动刷新进入网页

应用默认用户名 admin，明码 Harbor12345 即可登录。

三、配置 docker 客户端

将证书发送到 master01 服务器, 留神在操作之前要先在 master01 创立好寄存证书的文件夹 /etc/docker/certs.d/harbor.snow.com

root@master01:~# mkdir -p /etc/docker/certs.d/harbor.snow.com
root@harbor:/usr/local/src/harbor/certs# scp harbor.snow.com.cert 192.168.10.100:/etc/docker/certs.d/harbor.snow.com/
root@192.168.10.100's password: 
harbor.snow.com.cert                                                                                        100% 2102     2.3MB/s   00:00      

root@harbor:/usr/local/src/harbor/certs# scp harbor.snow.com.key 192.168.10.100:/etc/docker/certs.d/harbor.snow.com/
root@192.168.10.100's password: 
harbor.snow.com.key                                                                                         100% 2045     2.0MB/s   00:00
  
root@harbor:/usr/local/src/harbor/certs# scp ca.crt  192.168.10.100:/etc/docker/certs.d/harbor.snow.com
root@192.168.10.100's password: 
ca.crt                                                                                                      100% 2045     2.0MB/s   00:00

在 master01 节点看到证书曾经胜利发送过去 

root@master01:/etc/docker/certs.d/harbor.snow.com# ls
ca.crt  harbor.snow.com.cert  harbor.snow.com.key

批改 docker 的 daemon.json 文件，使其信赖 harbor.snow.com 仓库 

root@master01:/etc/docker# cat daemon.json 
{"exec-opts": ["native.cgroupdriver=systemd"],
  "insecure-registries":["harbor.snow.com"],
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker"
}

重启 docker 服务并批改其 hosts 文件使 master01 节点可能通过域名找到对应的 harbor 仓库 

root@master01:~# systemctl restart docker
root@master01:~# cat /etc/hosts
127.0.0.1 localhost
192.168.10.112 harbor.snow.com

登录仓库 

root@master01:~# docker login harbor.snow.com
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

上传镜像进行测试，留神要提前在 web 页面创立好新的我的项目。

root@master01:~# docker images
REPOSITORY                                TAG             IMAGE ID       CREATED         SIZE
calico/cni                                v3.19.2         05bf027c9836   17 months ago   146MB

为镜像改名字
root@master01:~# docker tag calico/cni:v3.19.2 harbor.snow.com/calico/cni:v3.19.2

推送到镜像仓库
root@master01:~# docker push harbor.snow.com/calico/cni:v3.19.2
The push refers to repository [harbor.snow.com/calico/cni]
758850cabe72: Pushed 
e88fd392bb0c: Pushed 
fb44224b4823: Pushed 
v3.19.2: digest: sha256:00a619424a20d1b31c0f65d1194484aebe9f90dfc1d4d6396ce706941004860b size: 946

推送到镜像仓库后即可在 web 页面看到刚刚推送下来镜像

镜像拉取测试 

root@master01:~# docker images
REPOSITORY                                  TAG             IMAGE ID       CREATED         SIZE
kubernetesui/dashboard                      v2.3.1          e1482a24335a   18 months ago   220MB

root@master01:~# docker pull harbor.snow.com/calico/cni:v3.19.2
v3.19.2: Pulling from calico/cni
Digest: sha256:00a619424a20d1b31c0f65d1194484aebe9f90dfc1d4d6396ce706941004860b
Status: Downloaded newer image for harbor.snow.com/calico/cni:v3.19.2
harbor.snow.com/calico/cni:v3.19.2

root@master01:~# docker images
REPOSITORY                                  TAG             IMAGE ID       CREATED         SIZE
kubernetesui/dashboard                      v2.3.1          e1482a24335a   18 months ago   220MB
harbor.snow.com/calico/cni                  v3.19.2         05bf027c9836   17 months ago   146MB

4、K8S 集群配置应用公有镜像仓库 Harbor

在配置 k8s 应用 harbor 之前须要先实现本文的第三步，先让 docker 能够失常应用 harbor。
在 docker 胜利登录 harbor 后会在用户家目录生成一个秘钥文件 

root@master01:~# cat /root/.docker/config.json
{
    "auths": {
        "harbor.snow.com": {"auth": "YWRtaW46SGFyYm9yMTIzNDU="}
    }
}

将秘钥进行 base64 加密 

root@master01:~# cat /root/.docker/config.json | base64 -w 0
ewoJImF1dGhzIjogewoJCSJoYXJib3Iuc25vdy5jb20iOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=

创立用户 harbor 认证的 secret 的 yaml 文件 

root@master01:~/secret# cat registry-pull-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: registry-pull-secret
  namespace: kube-system
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSJoYXJib3Iuc25vdy5jb20iOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=
type: kubernetes.io/dockerconfigjson

创立 secret，留神 secret 为名称空间级别的资源，调用这个 secret 的 yaml 文件要与此 secret 在同一个名称空间下。

root@master01:~/secret# kubectl delete -f registry-pull-secret.yaml 
secret "registry-pull-secret" deleted
root@master01:~/secret# kubectl apply -f registry-pull-secret.yaml 
secret/registry-pull-secret created
root@master01:~/secret# kubectl get secret
NAME                   TYPE                                  DATA   AGE
default-token-rxd89    kubernetes.io/service-account-token   3      5h2m
registry-pull-secret   kubernetes.io/dockerconfigjson        1      3h

最初只须要在须要去 harbor 拉取镜像的 pod 的 yaml 文件中增加如下配置即可（只展现出了实现 yaml 文件的局部内容）留神其实与 containers 语句是平级的。

template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      imagePullSecrets:
        - name: registry-pull-secret
      containers:
        image: harbor.snow.com/base/metrics-server:v0.5.0
        imagePullPolicy: Always

配置实现后 k8s 集群即可胜利拉取 harbor 中的镜像。

ps：以上内容在自己实现环境中已试验胜利，如发现有问题或表述不清的中央欢送斧正。

退出移动版