本文作者 徐晓伟

极狐Gitlab Runner 信赖实例域名证书，用于注册注册极狐 GitLab Runner。

问题

参见 极狐gitlab-runner-host.md

阐明

  • 解决方案是应用颁发给域名 gitlab.test.helm.xuxiaowei.cn 的证书，能够应用本人的域名去各大云厂商收费申请，或者应用本人依据域名 gitlab.test.helm.xuxiaowei.cn 生成的证书

    1. 阿里云SSL(https)证书收费申请
    2. 腾讯云SSL(https)证书收费申请
    3. 华为云SSL(https)证书收费申请
    4. 百度云SSL(https)证书收费申请

域名证书解决方案

计划1：重新配置极狐 GitLab，主动生成对应证书并主动配置，而后在极狐 GitLab runner 中信赖证书

  1. 问题1：tls: failed to verify certificate: x509: certificate signed by unknown authority

    1. 文档

      1. runners 局部
      2. 自签名证书或自定义证书颁发机构
      3. ConfigMap
      4. 配置 Pod 应用 ConfigMap

    2. 根据上述文档可知，有三种形式能够解决证书信赖问题：

      1. 在 极狐GitLab Runner 注册时配置 --tls-ca-file 参数并指向证书文件
      2. 在配置文件中减少 tls-ca-file 并指向配置文件， 此形式本文不做介绍，可参考 gitlab-runner-kubernetes-host.md ，原理雷同
      3. 将证书放在 /etc/gitlab-runner/certs/ 文件夹下，应用域名命名证书，证书后缀名是 .crt， 因为篇幅限度，这里只介绍这一种形式
    3. 因为 k8s 的 pod 可能随时会被删除（故障转移、被动删除等），所以间接在 pod 外部配置不是上策
    4. k8s 中的 ConfigMap 能够挂载到 pod 外部，所以能够思考在 ConfigMap 中配置证书，而后 pod 外部应用 ConfigMap 中的证书

    5. 下载证书

      1. 形式1

        1. 浏览器拜访域名 https://gitlab.test.helm.xuxiaowei.cn
        2. 依照图中操作，下载证书，上传之服务器，用于前期操作   

      2. 形式2

        # 下载证书openssl s_client -showcerts -connect gitlab.test.helm.xuxiaowei.cn:443 -servername gitlab.test.helm.xuxiaowei.cn < /dev/null 2>/dev/null | openssl x509 -outform PEM > gitlab.test.helm.xuxiaowei.cn.crt

    6. 将证书导入到 k8s 中

      # -n=gitlab-test：指定命名空间# create configmap etc-gitlab-runner-certs：创立 ConfigMap 名称是 etc-gitlab-runner-certs# --from-file=gitlab.test.helm.xuxiaowei.cn.crt=gitlab.test.helm.xuxiaowei.cn.crt：配置来自文件，文件名 gitlab.test.helm.xuxiaowei.cn.crt，放入 ConfigMap 中的键也是 gitlab.test.helm.xuxiaowei.cn.crtkubectl -n=gitlab-test create configmap etc-gitlab-runner-certs --from-file=gitlab.test.helm.xuxiaowei.cn.crt=gitlab.test.helm.xuxiaowei.cn.crt# 查看# kubectl -n=gitlab-test get configmap etc-gitlab-runner-certs -o yaml

    7. 导出 helm 极狐GitLab 配置

      helm -n gitlab-test get values my-gitlab > my-gitlab.yaml

    8. 批改 helm 极狐GitLab 配置

      gitlab-runner:  # 挂载卷名称  volumeMounts:    # 挂载到 pod 门路（文件/文件夹）    # 此处是在容器内运行的 gitlab runner，因为权限限度等起因，    # 所以 配置文件不是在 /etc/gitlab-runner/ 目录下，而是 /home/gitlab-runner/.gitlab-runner/    - mountPath: /home/gitlab-runner/.gitlab-runner/certs      name: etc-gitlab-runner-certs-volumes  # 卷  volumes:    # 卷类型    - configMap:        items:        # configMap 中的键        - key: gitlab.test.helm.xuxiaowei.cn.crt          # 挂载的门路          path: gitlab.test.helm.xuxiaowei.cn.crt        name: etc-gitlab-runner-certs      # 配置 ConfigMap 名称      name: etc-gitlab-runner-certs-volumes

    9. 更新 helm 极狐GitLab 配置

      helm upgrade -n gitlab-test --install my-gitlab gitlab/gitlab -f my-gitlab.yaml --timeout 600s --version 7.7.0

    10. 查看批改后果

      kubectl -n gitlab-test get deployments.apps my-gitlab-gitlab-runner -o yaml
      apiVersion: apps/v1kind: Deploymentmetadata:  annotations:    deployment.kubernetes.io/revision: "5"    meta.helm.sh/release-name: my-gitlab    meta.helm.sh/release-namespace: gitlab-test  creationTimestamp: "2023-12-22T05:03:46Z"  generation: 7  labels:    app: my-gitlab-gitlab-runner    app.kubernetes.io/managed-by: Helm    chart: gitlab-runner-0.59.2    heritage: Helm    release: my-gitlab  name: my-gitlab-gitlab-runner  namespace: gitlab-test  resourceVersion: "30086"  uid: 8c46c44a-5b67-44ae-90d0-008daa3fa388spec:  progressDeadlineSeconds: 600  replicas: 1  revisionHistoryLimit: 10  selector:    matchLabels:      app: my-gitlab-gitlab-runner  strategy:    rollingUpdate:      maxSurge: 25%      maxUnavailable: 25%    type: RollingUpdate  template:    metadata:      annotations:        checksum/configmap: f35865354f043583d0903b0a8350830a486eb0e289d18271cf3f533e7d89c5f7        checksum/secrets: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855        gitlab.com/prometheus_port: "9252"        gitlab.com/prometheus_scrape: "true"      creationTimestamp: null      labels:        app: my-gitlab-gitlab-runner        chart: gitlab-runner-0.59.2        heritage: Helm        release: my-gitlab    spec:      containers:        - command:            - /usr/bin/dumb-init            - --            - /bin/bash            - /configmaps/entrypoint          env:            - name: CI_SERVER_URL              value: https://gitlab.test.helm.xuxiaowei.cn            - name: RUNNER_EXECUTOR              value: kubernetes            - name: REGISTER_LOCKED              value: "false"            - name: RUNNER_TAG_LIST            - name: KUBERNETES_NAMESPACE              value: gitlab-test          image: registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v16.6.1          imagePullPolicy: IfNotPresent          livenessProbe:            exec:              command:                - /bin/bash                - /configmaps/check-live            failureThreshold: 3            initialDelaySeconds: 60            periodSeconds: 10            successThreshold: 1            timeoutSeconds: 3          name: my-gitlab-gitlab-runner          ports:            - containerPort: 9252              name: metrics              protocol: TCP          readinessProbe:            exec:              command:                - /usr/bin/pgrep                - gitlab.*runner            failureThreshold: 3            initialDelaySeconds: 10            periodSeconds: 10            successThreshold: 1            timeoutSeconds: 3          resources: {}          securityContext:            allowPrivilegeEscalation: false            capabilities:              drop:                - ALL            privileged: false            readOnlyRootFilesystem: false            runAsNonRoot: true          terminationMessagePath: /dev/termination-log          terminationMessagePolicy: File          volumeMounts:            - mountPath: /secrets              name: projected-secrets            - mountPath: /home/gitlab-runner/.gitlab-runner              name: etc-gitlab-runner            - mountPath: /configmaps              name: configmaps            - mountPath: /home/gitlab-runner/.gitlab-runner/certs              name: etc-gitlab-runner-certs-volumes      dnsPolicy: ClusterFirst      hostAliases:        - hostnames:            - gitlab.test.helm.xuxiaowei.cn          ip: 172.25.25.32      restartPolicy: Always      schedulerName: default-scheduler      securityContext:        fsGroup: 65533        runAsUser: 100      serviceAccount: my-gitlab-gitlab-runner      serviceAccountName: my-gitlab-gitlab-runner      terminationGracePeriodSeconds: 3600      volumes:        - emptyDir:            medium: Memory          name: runner-secrets        - emptyDir:            medium: Memory          name: etc-gitlab-runner        - name: projected-secrets          projected:            defaultMode: 420            sources:              - secret:                  name: my-gitlab-minio-secret              - secret:                  items:                    - key: runner-registration-token                      path: runner-registration-token                    - key: runner-token                      path: runner-token                  name: my-gitlab-gitlab-runner-secret        - configMap:            defaultMode: 420            name: my-gitlab-gitlab-runner          name: configmaps        - configMap:            defaultMode: 420            items:              - key: gitlab.test.helm.xuxiaowei.cn.crt                path: gitlab.test.helm.xuxiaowei.cn.crt            name: etc-gitlab-runner-certs          name: etc-gitlab-runner-certs-volumesstatus:  availableReplicas: 1  conditions:    - lastTransitionTime: "2023-12-22T07:43:25Z"      lastUpdateTime: "2023-12-22T07:43:25Z"      message: Deployment has minimum availability.      reason: MinimumReplicasAvailable      status: "True"      type: Available    - lastTransitionTime: "2023-12-22T05:03:46Z"      lastUpdateTime: "2023-12-22T07:43:25Z"      message: ReplicaSet "my-gitlab-gitlab-runner-597d6d8f7c" has successfully progressed.      reason: NewReplicaSetAvailable      status: "True"      type: Progressing  observedGeneration: 7  readyReplicas: 1  replicas: 1  updatedReplicas: 1

    11. 查看 gitlab runner 日志

      [root@anolis-7-9 ~]# kubectl -n gitlab-test get pod | grep gitlab-runnermy-gitlab-gitlab-runner-597d6d8f7c-8v466             1/1     Running     0              5m52s[root@anolis-7-9 ~]#

      依据日志判断，曾经注册胜利了

      [root@anolis-7-9 ~]# kubectl -n gitlab-test logs -f my-gitlab-gitlab-runner-597d6d8f7c-8v466Registration attempt 1 of 30Runtime platform                                    arch=amd64 os=linux pid=16 revision=f5da3c5a version=16.6.1WARNING: Running in user-mode.                     WARNING: The user-mode requires you to manually start builds processing:WARNING: $ gitlab-runner run                       WARNING: Use sudo for system-mode:                 WARNING: $ sudo gitlab-runner...Created missing unique system ID                    system_id=r_3WsywNzJqRudMerging configuration from template file "/configmaps/config.template.toml"WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://docs.gitlab.com/ee/ci/runners/new_creation_workflowRegistering runner... succeeded                     runner=BtGwLEwcRunner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!Configuration (with the authentication token) was saved in "/home/gitlab-runner/.gitlab-runner/config.toml"Runtime platform                                    arch=amd64 os=linux pid=7 revision=f5da3c5a version=16.6.1Starting multi-runner from /home/gitlab-runner/.gitlab-runner/config.toml...  builds=0 max_builds=0WARNING: Running in user-mode.                     WARNING: Use sudo for system-mode:                 WARNING: $ sudo gitlab-runner...There might be a problem with your config based on jsonschema annotations in common/config.go (experimental feature):jsonschema: '/runners/0/kubernetes/pull_policy' does not validate with https://gitlab.com/gitlab-org/gitlab-runner/common/config#/$ref/properties/runners/items/$ref/properties/kubernetes/$ref/properties/pull_policy/$ref/type: expected array, but got nullConfiguration loaded                                builds=0 max_builds=10listen_address not defined, metrics & debug endpoints disabled  builds=0 max_builds=10[session_server].listen_address not defined, session endpoints disabled  builds=0 max_builds=10Initializing executor providers                     builds=0 max_builds=10^C[root@anolis-7-9 ~]#

    12. 管理员拜访 https://gitlab.test.helm.xuxiaowei.cn/admin/runners 能够看到 pod my-gitlab-gitlab-runner-597d6d8f7c-8v466 曾经注册成了

      1. 如果存在极狐 GitLab Runner Pod 被删除（故障转译、手动删除等等），这里可能会存在不可用的 Runner

  2. 问题2： tls: failed to verify certificate: x509: certificate is valid for ingress.local, not gitlab.test.helm.xuxiaowei.cn

    1. 拜访的域名与应用的证书补匹配，可通过更换证书、域名来解决此问题（个别状况是更换证书）

    2. 如果要更换证书：

      1. 证书属于正规机构颁发的（如：上述各大云厂商的证书），请看下方计划2
      2. 如果是本人生成的证书（不受信赖），请看下方计划2，并联合上方问题1来解决此问题

计划2：配置正规机构颁发的证书（如：上述各大云厂商的证书），个别无需在 gitlab 配置信赖证书，即可失常应用

  1. 失常状况

    1. 要求证书与域名对应
    2. 将证书上传至服务器，解压，应用证书文件创建 k8s TLS Secret
    3. 批改 极狐GitLab 配置，应用 上述步骤创立的 k8s TLS Secret

  2. 异常情况：正规机构颁发的证书，在极狐 GitLab Runner 中仍然无奈失常应用，然而在浏览器中能够失常应用

    1. 起因是颁发机构的根证书不在极狐 GitLab Runner 的信赖列表中，须要手动增加信赖（例如：常见的 Windows 较低版本运行一些软件时，也是无奈执行，显示证书无奈辨认，须要在 Windows 导入软件签名所应用的证书链）