关于kubernetes:kubernetes源码调试debug-apiserver

在 近程机器 上启动集群，而后在 本地机器 上，通过 IDE 近程调试 kubernetes 的代码。

以 apiserver 为例。

一. 环境筹备(近程机器)

1. 环境

• OS: centos7.6
• kubernetes 源码：v1.20
• golang: go1.17.12
• dlv: 1.9.1

2. 装置 golang

参考：https://docs.studygolang.com/doc/install

3. 装置 dlv

参考：https://github.com/go-delve/delve/tree/master/Documentation/installation

# go install github.com/go-delve/delve/cmd/dlv@latest

4. 下载 kubernetes 源码

git clone -b release-1.20 git@github.com:kubernetes/kubernetes.git

二. 批改编译参数（近程机器）

进入到 kubernetes 的源码，批改 hack/lib/golang.sh 文件，从而让编译器不去优化掉 debug 的信息，次要包含：

• 禁止 -w -s，保留文件名、行号；
• 加上 -gcflags=”all=-N -l”，禁止优化和内联；

批改点：

kube::golang::build_binaries() {
  # Create a sub-shell so that we don't pollute the outer environment
  (# Check for `go` binary and set ${GOPATH}.
    kube::golang::setup_env
    V=2 kube::log::info "Go version: $(GOFLAGS='' go version)"

    local host_platform
    host_platform=$(kube::golang::host_platform)

    local goflags goldflags goasmflags gogcflags gotags
    # If GOLDFLAGS is unset, then set it to the a default of "-s -w".
    # Disable SC2153 for this, as it will throw a warning that the local
    # variable goldflags will exist, and it suggest changing it to this.
    # shellcheck disable=SC2153
    # goldflags="${GOLDFLAGS=-s -w -buildid=} $(kube::version::ldflags)"
    goldflags="${GOLDFLAGS=-buildid=} $(kube::version::ldflags)"        // 去掉 -s -w
    goasmflags="-trimpath=${KUBE_ROOT}"
    gogcflags="${GOGCFLAGS:-} -trimpath=${KUBE_ROOT}"
    gogcflags="${gogcflags} -N -l"                                    // 减少 -N -l

    # extract tags if any specified in GOFLAGS
    # shellcheck disable=SC2001
    gotags="selinux,notest,$(echo"${GOFLAGS:-}"| sed -e's|.*-tags=\([^-]*\).*|\1|')"

    local -a targets=()
    local arg
    ...
}

三. 装置 etcd（近程机器）

进入 kubernetes 的源码，执行以下脚本，装置 etcd:

# ./hack/install-etcd.sh
Downloading https://github.com/coreos/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz succeed
etcd v3.4.13 installed. To use:
export PATH="/root/go/src/github.com/kubernetes/kubernetes/third_party/etcd:${PATH}"

四. 启动本地集群（近程机器）

筹备工作实现后，执行上面的脚本，启动本地集群：

# ./hack/local-up-cluster.sh

查看 kube 过程是否启动：

# ps -a|grep kube
18999 pts/0    00:26:51 kube-controller
19001 pts/0    00:02:55 kube-scheduler
19162 pts/0    00:26:35 kubelet
19929 pts/0    00:00:42 kube-proxy
89307 pts/2    00:00:07 kube-apiserver

kubectl 拜访本地集群：

# export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
# cluster/kubectl.sh get ns 
NAME              STATUS   AGE
default           Active   45m
kube-node-lease   Active   45m
kube-public       Active   45m
kube-system       Active   45m

五. dlv 启动 kube-apiserver（近程机器）

查看 kube-apiserver 的启动参数：

# ps -ef|grep kube-apiserver
root      89307  30528 12 09:37 pts/2    00:00:17 /root/go/src/github.com/kubernetes/kubernetes/_output/local/bin/linux/amd64/kube-apiserver --authorization-mode=Node,RBAC --cloud-provider= --cloud-config= --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key --cors-allowed-origins=/127.0.0.1(:[0-9]+)?$,/localhost(:[0-9]+)?$

杀掉 kube-apiserver 过程，应用 dlv 启动：

• dlv 监听时，应用 12345 端口；
• 能够让 Goland 近程连贯该端口，进行近程 debug；

# kill -9 89307

## 1. dlv --headless exec 启动 apiserver 过程
## 2. 过程参数中，减少 --listen=0.0.0.0:12345 参数
# dlv --headless exec  /root/go/src/github.com/kubernetes/kubernetes/_output/local/bin/linux/amd64/kube-apiserver --listen=0.0.0.0:12345 --api-version=2 --log  --  --authorization-mode=Node,RBAC  --cloud-provider= --cloud-config=   --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving-kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key --cors-allowed-origins=/127.0.0.1\(:[0-9]+\)?$,/localhost\(:[0-9]+\)?$

六. Goland 近程调试（本地机器）

Goland 上配置近程 Debug 的 ip 和端口:

Goland 应用下面的配置，启动 Debug: