接下来的一段时间将要学习Kubernetes源码，为了更好的查看源码和调试程序，因而搭建了一个Kubernetes开发调试环境，该环境能够联合断点调试了解代码的运行过程。
筹备虚拟机并装置必要软件
$ yum install -y docker wget curl vim golang etcd openssl git g++ gcc$ echo "export GOROOT=/usr/local/goexport GOPATH=/home/workspaceexport PATH=$PATH:$GOROOT/bin:$GOPATH/bin" >> /etc/profile$ source /etc/profile装置证书生成工具
$ go get -u -v github.com/cloudflare/cfssl/cmd/...装置调试工具
$ go get -u -v github.com/go-delve/delve/cmd/dlv留神： 如果上述两种工具装置失败或者因为网络的起因无奈装置能够间接在github上将源码下载到$GOPATH/pkg/github.com下进入对应的目录执行 make install进行装置，装置胜利后在$GOPATH 目录的bin目录下生成可执行程序
筹备源码及编译出调试版本
clone Kubernetes代码
$ git clone https://github.com/kubernetes/kubernetes.git $GOPATH/src/k8s.io/kubernetes$ cd $GOPATH/src/k8s.io/kubernetes$ git checkout -b v1.20.1 # 切换到v1.20.1分支clone etcd代码
$ git clone https://https://github.com/etcd-io/etcd.git $GOPATH/src/k8s.io/etcd$ cd $GOPATH/src/k8s.io/etcd$ git checkout -b v3.4.13 # 切换到v3.4.13分支执行单机集群启动脚本
$ cd $GOPATH/src/k8s.io/kubernetes$ ./hack/local-up-cluster.sh # 执行工夫较长，急躁期待make: Entering directory `/home/workspace/src/k8s.io/kubernetes'make[1]: Entering directory `/home/workspace/src/k8s.io/kubernetes'make[1]: Leaving directory `/home/workspace/src/k8s.io/kubernetes'make[1]: Entering directory `/home/workspace/src/k8s.io/kubernetes'+++ [0818 08:46:21] Building go targets for linux/amd64: ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen+++ [0818 08:46:29] Building go targets for linux/amd64: ./vendor/k8s.io/code-generator/cmd/deepcopy-gen+++ [0818 08:46:49] Building go targets for linux/amd64: ./vendor/k8s.io/code-generator/cmd/defaulter-gen+++ [0818 08:47:10] Building go targets for linux/amd64: ./vendor/k8s.io/code-generator/cmd/conversion-gen+++ [0818 08:48:04] Building go targets for linux/amd64: ./vendor/k8s.io/kube-openapi/cmd/openapi-gen+++ [0818 08:48:35] Building go targets for linux/amd64: ./vendor/github.com/go-bindata/go-bindata/go-bindatamake[1]: Leaving directory `/home/workspace/src/k8s.io/kubernetes'+++ [0818 08:48:38] Building go targets for linux/amd64: cmd/kubectl cmd/kube-apiserver cmd/kube-controller-manager cmd/cloud-controller-manager cmd/kubelet cmd/kube-proxy cmd/kube-schedulermake: Leaving directory `/home/workspace/src/k8s.io/kubernetes'Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?Kubelet cgroup driver defaulted to use:/home/workspace/src/k8s.io/kubernetes/third_party/etcd:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/go/bin:/home/workspace/bin:/root/binetcd version 3.4.13 or greater required.You can use 'hack/install-etcd.sh' to install a copy in third_party/.解决上述问题
$ ./hack/install-etcd.sh # 执行工夫较长，急躁期待Downloading https://github.com/coreos/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz succeedetcd v3.4.13 installed. To use:export PATH="/home/workspace/src/k8s.io/kubernetes/third_party/etcd:${PATH}"# 如果在内网环境无奈下载上述包能够通过手动编译并建设软连贯解决$ go env -w GOPROXY=https://goproxy.cn,direct$ cd $GOPATH/src/k8s.io/etcd$ go mod vendor$ make$ cp bin/etcd $GOPATH/src/k8s.io/kubernetes/third_party/etcd-v3.4.13-linux-amd64$ ln -fns etcd-v3.4.13-linux-amd64 etcd# 再次执行上面脚本$ ./hack/local-up-cluster.sh -O # 前提确保docker服务失常， 其中-O参数示意间接运行上次编译后的kubernetes相干组件，不再编译，如果能够胜利启动则示意单机集群启动胜利skipped the build. WARNING: You're not using the default seccomp profileWARNING: bridge-nf-call-iptables is disabledWARNING: bridge-nf-call-ip6tables is disabledKubelet cgroup driver defaulted to use: systemd/home/workspace/src/k8s.io/kubernetes/third_party/etcd:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/go/bin:/home/workspace/bin:/root/binAPI SERVER secure port is free, proceeding...Detected host and ready to start services. Doing some housekeeping first...Using GO_OUT /home/workspace/src/k8s.io/kubernetes/_output/binStarting services now!Starting etcdetcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.oUDPwStJvj --listen-client-urls http://127.0.0.1:2379 --log-level=debug > "/tmp/etcd.log" 2>/dev/nullWaiting for etcd to come up.+++ [0818 09:28:25] On try 2, etcd: : {"health":"true"}{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"2","raft_term":"2"}}Generating a 2048 bit RSA private key..........................................+++........+++writing new private key to '/var/run/kubernetes/server-ca.key'-----Generating a 2048 bit RSA private key.....................+++..............................+++writing new private key to '/var/run/kubernetes/client-ca.key'-----Generating a 2048 bit RSA private key......................+++...........+++writing new private key to '/var/run/kubernetes/request-header-ca.key'-----2021/08/18 09:28:26 [INFO] generate received request2021/08/18 09:28:26 [INFO] received CSR2021/08/18 09:28:26 [INFO] generating key: rsa-20482021/08/18 09:28:26 [INFO] encoded CSR2021/08/18 09:28:26 [INFO] signed certificate with serial number 1944668198545486959178087658848077051979668935142021/08/18 09:28:26 [INFO] generate received request2021/08/18 09:28:26 [INFO] received CSR2021/08/18 09:28:26 [INFO] generating key: rsa-20482021/08/18 09:28:26 [INFO] encoded CSR2021/08/18 09:28:26 [INFO] signed certificate with serial number 2603362207932002728319812248475921959601264396352021/08/18 09:28:26 [INFO] generate received request2021/08/18 09:28:26 [INFO] received CSR2021/08/18 09:28:26 [INFO] generating key: rsa-20482021/08/18 09:28:27 [INFO] encoded CSR2021/08/18 09:28:27 [INFO] signed certificate with serial number 650897408210385411292065578082634437400044055792021/08/18 09:28:27 [INFO] generate received request2021/08/18 09:28:27 [INFO] received CSR2021/08/18 09:28:27 [INFO] generating key: rsa-20482021/08/18 09:28:27 [INFO] encoded CSR2021/08/18 09:28:27 [INFO] signed certificate with serial number 1328546049532766555281109603247667662361210220032021/08/18 09:28:27 [INFO] generate received request2021/08/18 09:28:27 [INFO] received CSR2021/08/18 09:28:27 [INFO] generating key: rsa-20482021/08/18 09:28:28 [INFO] encoded CSR2021/08/18 09:28:28 [INFO] signed certificate with serial number 5098373112824191733942807363773879646092935887592021/08/18 09:28:28 [INFO] generate received request2021/08/18 09:28:28 [INFO] received CSR2021/08/18 09:28:28 [INFO] generating key: rsa-20482021/08/18 09:28:29 [INFO] encoded CSR2021/08/18 09:28:29 [INFO] signed certificate with serial number 3153609466862973249791119885899873309893610143872021/08/18 09:28:29 [INFO] generate received request2021/08/18 09:28:29 [INFO] received CSR2021/08/18 09:28:29 [INFO] generating key: rsa-20482021/08/18 09:28:29 [INFO] encoded CSR2021/08/18 09:28:29 [INFO] signed certificate with serial number 1009537356452097450336378717730674703064826136382021/08/18 09:28:29 [INFO] generate received request2021/08/18 09:28:29 [INFO] received CSR2021/08/18 09:28:29 [INFO] generating key: rsa-20482021/08/18 09:28:30 [INFO] encoded CSR2021/08/18 09:28:30 [INFO] signed certificate with serial number 719845605910186326782888521350059363008185160184Waiting for apiserver to come up+++ [0818 09:28:40] On try 7, apiserver: : okclusterrolebinding.rbac.authorization.k8s.io/kube-apiserver-kubelet-admin createdclusterrolebinding.rbac.authorization.k8s.io/kubelet-csr createdCluster "local-up-cluster" set.use 'kubectl --kubeconfig=/var/run/kubernetes/admin-kube-aggregator.kubeconfig' to use the aggregated API serverservice/kube-dns createdserviceaccount/kube-dns createdconfigmap/kube-dns createddeployment.apps/kube-dns createdKube-dns addon successfully deployed.WARNING : The kubelet is configured to not fail even if swap is enabled; production deployments should disable swap.2021/08/18 09:28:43 [INFO] generate received request2021/08/18 09:28:43 [INFO] received CSR2021/08/18 09:28:43 [INFO] generating key: rsa-20482021/08/18 09:28:44 [INFO] encoded CSR2021/08/18 09:28:44 [INFO] signed certificate with serial number 185139553486611438539303446897387693685807865483kubelet ( 120208 ) is running.wait kubelet readyNo resources foundNo resources foundNo resources foundNo resources foundNo resources foundNo resources foundNo resources found127.0.0.1 NotReady <none> 2s v1.20.12021/08/18 09:29:02 [INFO] generate received request2021/08/18 09:29:02 [INFO] received CSR2021/08/18 09:29:02 [INFO] generating key: rsa-20482021/08/18 09:29:02 [INFO] encoded CSR2021/08/18 09:29:02 [INFO] signed certificate with serial number 88607895468152274682757139244368051799851531300Create default storage class forstorageclass.storage.k8s.io/standard createdLocal Kubernetes cluster is running. Press Ctrl-C to shut it down.Logs: /tmp/kube-apiserver.log /tmp/kube-controller-manager.log /tmp/kube-proxy.log /tmp/kube-scheduler.log /tmp/kubelet.logTo start using your cluster, you can open up another terminal/tab and run: export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig cluster/kubectl.shAlternatively, you can write to the default kubeconfig: export KUBERNETES_PROVIDER=local cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt cluster/kubectl.sh config set-context local --cluster=local --user=myself cluster/kubectl.sh config use-context local cluster/kubectl.sh查看Kubernetes各组件启动参数及PID
$ ps -ef|grep kube|grep -v grep能够看到挪动7个过程，别离如下：
root 119660 119248 10 09:28 pts/4 00:00:50 /home/workspace/src/k8s.io/kubernetes/_output/bin/kube-apiserver --authorization-mode=Node,RBAC --cloud-provider= --cloud-config= --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving-kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key --cors-allowed-origins=/127.0.0.1(:[0-9]+)?$,/localhost(:[0-9]+)?$root 120034 119248 3 09:28 pts/4 00:00:16 /home/workspace/src/k8s.io/kubernetes/_output/bin/kube-controller-manager --v=3 --vmodule= --service-account-private-key-file=/tmp/kube-serviceaccount.key --service-cluster-ip-range=10.0.0.0/24 --root-ca-file=/var/run/kubernetes/server-ca.crt --cluster-signing-cert-file=/var/run/kubernetes/client-ca.crt --cluster-signing-key-file=/var/run/kubernetes/client-ca.key --enable-hostpath-provisioner=false --pvclaimbinder-sync-period=15s --feature-gates=AllAlpha=false --cloud-provider= --cloud-config= --kubeconfig /var/run/kubernetes/controller.kubeconfig --use-service-account-credentials --controllers=* --leader-elect=false --cert-dir=/var/run/kubernetes --master=https://localhost:6443root 120036 119248 1 09:28 pts/4 00:00:05 /home/workspace/src/k8s.io/kubernetes/_output/bin/kube-scheduler --v=3 --config=/tmp/kube-scheduler.yaml --feature-gates=AllAlpha=false --master=https://localhost:6443root 120208 119248 0 09:28 pts/4 00:00:00 sudo -E /home/workspace/src/k8s.io/kubernetes/_output/bin/kubelet --v=3 --vmodule= --chaos-chance=0.0 --container-runtime=docker --hostname-override=127.0.0.1 --cloud-provider= --cloud-config= --bootstrap-kubeconfig=/var/run/kubernetes/kubelet.kubeconfig --kubeconfig=/var/run/kubernetes/kubelet-rotated.kubeconfig --config=/tmp/kubelet.yamlroot 120211 120208 3 09:28 pts/4 00:00:15 /home/workspace/src/k8s.io/kubernetes/_output/bin/kubelet --v=3 --vmodule= --chaos-chance=0.0 --container-runtime=docker --hostname-override=127.0.0.1 --cloud-provider= --cloud-config= --bootstrap-kubeconfig=/var/run/kubernetes/kubelet.kubeconfig --kubeconfig=/var/run/kubernetes/kubelet-rotated.kubeconfig --config=/tmp/kubelet.yamlroot 120957 119248 0 09:29 pts/4 00:00:00 sudo /home/workspace/src/k8s.io/kubernetes/_output/bin/kube-proxy --v=3 --config=/tmp/kube-proxy.yaml --master=https://localhost:6443root 120980 120957 0 09:29 pts/4 00:00:00 /home/workspace/src/k8s.io/kubernetes/_output/bin/kube-proxy --v=3 --config=/tmp/kube-proxy.yaml --master=https://localhost:6443配置近程调试
我这里应用的近程共享文件工具是samba，Samba的配置这里就不赘述了，能够在网上找到一大堆
应用Windows或者Linux上的Goland工具将虚拟机中的Kubernetes源码关上
Main menu->Run->Edit Configurations…
如果看到Edit Configurations…为灰色，无奈点击，这时能够点击Debug按钮在弹出框中抉择Edit Configurations…
点击+抉择Go Remote
其中Name依据本人的须要起一个，Host为虚拟机的IP，Port为后续dlv服务的端口号，dlv服务的启动命令也在两头地位显示进去了，一会咱们就用这个命令启动kube-apiserver的含有断点信息的二进制文件，接下来在kubernetes/cmd/kube-apiserver/apiserver.go的main函数的中央加一个断点，其余中央也能够
保留下面批改的代码和断点信息，执行以下代码即可独自编译该模块的DEBUG版本，这里以kube-apiserver为例：
$ make GOLDFLAGS="" WHAT="cmd/kube-apiserver"再次执行单集群启动脚本进行调试kube-apiserver ：
$ ./hack/local-up-cluster.sh -O # 间接启动单节点集群，不能去掉-O，不然会从新编译，笼罩kube-apiserver的DEBUG信息手动进行kube-apiserver的过程
$ ps -ef|grep kube-apiserver|grep -v grep$ kill -15 119660$ ps -ef|grep kube-apiserver|grep -v grep应用dlv启动kube-apiserver过程
dlv --listen=:39999 --headless=true --api-version=2 exec /home/workspace/src/k8s.io/kubernetes/_output/bin/kube-apiserver -- --authorization-mode=Node,RBAC --cloud-provider= --cloud-config= --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving-kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key --cors-allowed-origins='/127.0.0.1(:[0-9]+)?$,/localhost(:[0-9]+)?$'上述启动命令在配置近程调试界面曾经见过一次了，exec前面的为kube-apiserver的二进制程序 -- 前面的为kube-apiserver的启动参数，这些启动参数是通过命令ps -ef|grep kube-apiserver|grep -v grep查到的，同时启动参数应用dlv启动时须要批改一下，如上
调试截图
参考连贯：
- https://github.com/kubernetes...
- https://github.com/derekparke...
- https://github.com/kubernetes...