关于腾讯云:一文读懂-SuperEdge-分布式健康检查云端

杜杨浩，腾讯云高级工程师，热衷于开源、容器和Kubernetes。目前次要从事镜像仓库、Kubernetes集群高可用&备份还原，以及边缘计算相干研发工作。

前言

SuperEdge分布式健康检查性能由边端的edge-health-daemon以及云端的edge-health-admission组成：

edge-health-daemon：对同区域边缘节点执行分布式健康检查，并向apiserver发送衰弱状态投票后果(给node打annotation)
edge-health-admission：一直依据node edge-health annotation调整kube-controller-manager设置的node taint(去掉NoExecute taint)以及endpoints(将失联节点上的pods从endpoint subsets notReadyAddresses移到addresses中)，从而实现云端和边端独特决定节点状态

整体架构如下所示：

之所以创立edge-health-admission云端组件，是因为当云边断连时，kube-controller-manager会执行如下操作：

失联的节点被置为ConditionUnknown状态，并被增加NoSchedule和NoExecute的taints
失联的节点上的pod从Service的Endpoint列表中移除

当edge-health-daemon在边端依据健康检查判断节点状态失常时，会更新node：去掉NoExecute taint。然而在node胜利更新之后又会被kube-controller-manager给刷回去(再次增加NoExecute taint)，因而必须增加Kubernetes mutating admission webhook也即edge-health-admission，将kube-controller-manager对node api resource的更改做调整，最终实现分布式健康检查成果

在深刻源码之前先介绍一下Kubernetes Admission Controllers

An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized. The controllers consist of the list below, are compiled into the kube-apiserver binary, and may only be configured by the cluster administrator. In that list, there are two special controllers: MutatingAdmissionWebhook and ValidatingAdmissionWebhook. These execute the mutating and validating (respectively) admission control webhooks which are configured in the API.

Kubernetes Admission Controllers是kube-apiserver解决api申请的某个环节，用于在api申请认证&鉴权之后，对象长久化之前进行调用，对申请进行校验或者批改(or both)

Kubernetes Admission Controllers包含多种admission，大多数都内嵌在kube-apiserver代码中了。其中MutatingAdmissionWebhook以及ValidatingAdmissionWebhook controller比拟非凡，它们别离会调用内部结构的mutating admission control webhooks以及validating admission control webhooks

Admission webhooks are HTTP callbacks that receive admission requests and do something with them. You can define two types of admission webhooks, validating admission webhook and mutating admission webhook. Mutating admission webhooks are invoked first, and can modify objects sent to the API server to enforce custom defaults. After all object modifications are complete, and after the incoming object is validated by the API server, validating admission webhooks are invoked and can reject requests to enforce custom policies.

Admission Webhooks是一个HTTP回调服务，承受AdmissionReview申请并进行解决，依照解决形式的不同，能够将Admission Webhooks分类如下：

validating admission webhook：通过ValidatingWebhookConfiguration配置，会对api申请进行准入校验，然而不能批改申请对象
mutating admission webhook：通过MutatingWebhookConfiguration配置，会对api申请进行准入校验以及批改申请对象

两种类型的webhooks都须要定义如下Matching requests字段：

admissionReviewVersions：定义了apiserver所反对的AdmissionReview api resoure的版本列表(API servers send the first AdmissionReview version in the admissionReviewVersions list they support)
name：webhook名称(如果一个WebhookConfiguration中定义了多个webhooks，须要保障名称的唯一性)
clientConfig：定义了webhook server的拜访地址(url or service)以及CA bundle(optionally include a custom CA bundle to use to verify the TLS connection)
namespaceSelector：限定了匹配申请资源的命名空间labelSelector
objectSelector：限定了匹配申请资源自身的labelSelector
rules：限定了匹配申请的operations，apiGroups，apiVersions，resources以及resource scope，如下：
- operations：规定了申请操作列表(Can be "CREATE", "UPDATE", "DELETE", "CONNECT", or "*" to match all.)
- apiGroups：规定了申请资源的API groups列表("" is the core API group. "*" matches all API groups.)
- apiVersions：规定了申请资源的API versions列表("*" matches all API versions.)
- resources：规定了申请资源类型(node, deployment and etc)
- scope：规定了申请资源的范畴(Cluster，Namespaced or *)
timeoutSeconds：规定了webhook回应的超时工夫，如果超时了，依据failurePolicy进行解决
failurePolicy：规定了apiserver对admission webhook申请失败的解决策略：
- Ignore：means that an error calling the webhook is ignored and the API request is allowed to continue.
- Fail：means that an error calling the webhook causes the admission to fail and the API request to be rejected.
matchPolicy：规定了rules如何匹配到来的api申请，如下：
- Exact：齐全匹配rules列表限度
- Equivalent：如果批改申请资源(apiserver能够实现对象在不同版本的转化)能够转化为可能配置rules列表限度，则认为该申请匹配，能够发送给admission webhook
reinvocationPolicy：In v1.15+, to allow mutating admission plugins to observe changes made by other plugins, built-in mutating admission plugins are re-run if a mutating webhook modifies an object, and mutating webhooks can specify a reinvocationPolicy to control whether they are reinvoked as well.
- Never: the webhook must not be called more than once in a single admission evaluation
- IfNeeded: the webhook may be called again as part of the admission evaluation if the object being admitted is modified by other admission plugins after the initial webhook call.
Side effects：某些webhooks除了批改AdmissionReview的内容外，还会连带批改其它的资源("side effects")。而sideEffects批示了Webhooks是否具备"side effects"，取值如下：
- None: calling the webhook will have no side effects.
- NoneOnDryRun: calling the webhook will possibly have side effects, but if a request with dryRun: true is sent to the webhook, the webhook will suppress the side effects (the webhook is dryRun-aware).

这里给出edge-health-admission对应的MutatingWebhookConfiguration作为参考示例：

apiVersion: admissionregistration.k8s.io/v1kind: MutatingWebhookConfigurationmetadata:  name: edge-health-admissionwebhooks:  - admissionReviewVersions:      - v1    clientConfig:      caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNwRENDQVl3Q0NRQ2RaL0w2akZSSkdqQU5CZ2txaGtpRzl3MEJBUXNGQURBVU1SSXdFQVlEVlFRRERBbFgKYVhObE1tTWdRMEV3SGhjTk1qQXdOekU0TURRek9ERTNXaGNOTkRjeE1qQTBNRFF6T0RFM1dqQVVNUkl3RUFZRApWUVFEREFsWGFYTmxNbU1nUTBFd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUNSCnhHT2hrODlvVkRHZklyVDBrYVkwajdJQVJGZ2NlVVFmVldSZVhVcjh5eEVOQkF6ZnJNVVZyOWlCNmEwR2VFL3cKZzdVdW8vQWtwUEgrbzNQNjFxdWYrTkg1UDBEWHBUd1pmWU56VWtyaUVja3FOSkYzL2liV0o1WGpFZUZSZWpidgpST1V1VEZabmNWOVRaeTJISVF2UzhTRzRBTWJHVmptQXlDMStLODBKdDI3QUl4YmdndmVVTW8xWFNHYnRxOXlJCmM3Zk1QTXJMSHhaOUl5aTZla3BwMnJrNVdpeU5YbXZhSVA4SmZMaEdnTU56YlJaS1RtL0ZKdDdyV0dhQ1orNXgKV0kxRGJYQ2MyWWhmbThqU1BqZ3NNQTlaNURONDU5ellJSkVhSTFHeFI3MlhaUVFMTm8zdE5jd3IzVlQxVlpiTgo1cmhHQlVaTFlrMERtd25vWTBCekFnTUJBQUV3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUhuUDJibnJBcWlWCjYzWkpMVzM0UWFDMnRreVFScTNVSUtWR3RVZHFobWRVQ0I1SXRoSUlleUdVRVdqVExpc3BDQzVZRHh4YVdrQjUKTUxTYTlUY0s3SkNOdkdJQUdQSDlILzRaeXRIRW10aFhiR1hJQ3FEVUVmSUVwVy9ObUgvcnBPQUxhYlRvSUVzeQpVNWZPUy9PVVZUM3ZoSldlRjdPblpIOWpnYk1SZG9zVElhaHdQdTEzZEtZMi8zcEtxRW1Cd1JkbXBvTExGbW9MCmVTUFQ4SjREZExGRkh2QWJKalFVbjhKQTZjOHUrMzZJZDIrWE1sTGRZYTdnTnhvZTExQTl6eFJQczRXdlpiMnQKUXZpbHZTbkFWb0ZUSVozSlpjRXVWQXllNFNRY1dKc3FLMlM0UER1VkNFdlg0SmRCRlA2NFhvU08zM3pXaWhtLworMXg3OXZHMUpFcz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=      service:        namespace: kube-system        name: edge-health-admission        path: /node-taint    failurePolicy: Ignore    matchPolicy: Exact    name: node-taint.k8s.io    namespaceSelector: {}    objectSelector: {}    reinvocationPolicy: Never    rules:      - apiGroups:          - '*'        apiVersions:          - '*'        operations:          - UPDATE        resources:          - nodes        scope: '*'    sideEffects: None    timeoutSeconds: 5  - admissionReviewVersions:      - v1    clientConfig:      caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNwRENDQVl3Q0NRQ2RaL0w2akZSSkdqQU5CZ2txaGtpRzl3MEJBUXNGQURBVU1SSXdFQVlEVlFRRERBbFgKYVhObE1tTWdRMEV3SGhjTk1qQXdOekU0TURRek9ERTNXaGNOTkRjeE1qQTBNRFF6T0RFM1dqQVVNUkl3RUFZRApWUVFEREFsWGFYTmxNbU1nUTBFd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUNSCnhHT2hrODlvVkRHZklyVDBrYVkwajdJQVJGZ2NlVVFmVldSZVhVcjh5eEVOQkF6ZnJNVVZyOWlCNmEwR2VFL3cKZzdVdW8vQWtwUEgrbzNQNjFxdWYrTkg1UDBEWHBUd1pmWU56VWtyaUVja3FOSkYzL2liV0o1WGpFZUZSZWpidgpST1V1VEZabmNWOVRaeTJISVF2UzhTRzRBTWJHVmptQXlDMStLODBKdDI3QUl4YmdndmVVTW8xWFNHYnRxOXlJCmM3Zk1QTXJMSHhaOUl5aTZla3BwMnJrNVdpeU5YbXZhSVA4SmZMaEdnTU56YlJaS1RtL0ZKdDdyV0dhQ1orNXgKV0kxRGJYQ2MyWWhmbThqU1BqZ3NNQTlaNURONDU5ellJSkVhSTFHeFI3MlhaUVFMTm8zdE5jd3IzVlQxVlpiTgo1cmhHQlVaTFlrMERtd25vWTBCekFnTUJBQUV3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUhuUDJibnJBcWlWCjYzWkpMVzM0UWFDMnRreVFScTNVSUtWR3RVZHFobWRVQ0I1SXRoSUlleUdVRVdqVExpc3BDQzVZRHh4YVdrQjUKTUxTYTlUY0s3SkNOdkdJQUdQSDlILzRaeXRIRW10aFhiR1hJQ3FEVUVmSUVwVy9ObUgvcnBPQUxhYlRvSUVzeQpVNWZPUy9PVVZUM3ZoSldlRjdPblpIOWpnYk1SZG9zVElhaHdQdTEzZEtZMi8zcEtxRW1Cd1JkbXBvTExGbW9MCmVTUFQ4SjREZExGRkh2QWJKalFVbjhKQTZjOHUrMzZJZDIrWE1sTGRZYTdnTnhvZTExQTl6eFJQczRXdlpiMnQKUXZpbHZTbkFWb0ZUSVozSlpjRXVWQXllNFNRY1dKc3FLMlM0UER1VkNFdlg0SmRCRlA2NFhvU08zM3pXaWhtLworMXg3OXZHMUpFcz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=      service:        namespace: kube-system        name: edge-health-admission        path: /endpoint    failurePolicy: Ignore    matchPolicy: Exact    name: endpoint.k8s.io    namespaceSelector: {}    objectSelector: {}    reinvocationPolicy: Never    rules:      - apiGroups:          - '*'        apiVersions:          - '*'        operations:          - UPDATE        resources:          - endpoints        scope: '*'    sideEffects: None    timeoutSeconds: 5

kube-apiserver会发送AdmissionReview(apiGroup: admission.k8s.io，apiVersion：v1 or v1beta1)给Webhooks，并封装成JSON格局，示例如下：

# This example shows the data contained in an AdmissionReview object for a request to update the scale subresource of an apps/v1 Deployment{  "apiVersion": "admission.k8s.io/v1",  "kind": "AdmissionReview",  "request": {    # Random uid uniquely identifying this admission call    "uid": "705ab4f5-6393-11e8-b7cc-42010a800002",    # Fully-qualified group/version/kind of the incoming object    "kind": {"group":"autoscaling","version":"v1","kind":"Scale"},    # Fully-qualified group/version/kind of the resource being modified    "resource": {"group":"apps","version":"v1","resource":"deployments"},    # subresource, if the request is to a subresource    "subResource": "scale",    # Fully-qualified group/version/kind of the incoming object in the original request to the API server.    # This only differs from `kind` if the webhook specified `matchPolicy: Equivalent` and the    # original request to the API server was converted to a version the webhook registered for.    "requestKind": {"group":"autoscaling","version":"v1","kind":"Scale"},    # Fully-qualified group/version/kind of the resource being modified in the original request to the API server.    # This only differs from `resource` if the webhook specified `matchPolicy: Equivalent` and the    # original request to the API server was converted to a version the webhook registered for.    "requestResource": {"group":"apps","version":"v1","resource":"deployments"},    # subresource, if the request is to a subresource    # This only differs from `subResource` if the webhook specified `matchPolicy: Equivalent` and the    # original request to the API server was converted to a version the webhook registered for.    "requestSubResource": "scale",    # Name of the resource being modified    "name": "my-deployment",    # Namespace of the resource being modified, if the resource is namespaced (or is a Namespace object)    "namespace": "my-namespace",    # operation can be CREATE, UPDATE, DELETE, or CONNECT    "operation": "UPDATE",    "userInfo": {      # Username of the authenticated user making the request to the API server      "username": "admin",      # UID of the authenticated user making the request to the API server      "uid": "014fbff9a07c",      # Group memberships of the authenticated user making the request to the API server      "groups": ["system:authenticated","my-admin-group"],      # Arbitrary extra info associated with the user making the request to the API server.      # This is populated by the API server authentication layer and should be included      # if any SubjectAccessReview checks are performed by the webhook.      "extra": {        "some-key":["some-value1", "some-value2"]      }    },    # object is the new object being admitted.    # It is null for DELETE operations.    "object": {"apiVersion":"autoscaling/v1","kind":"Scale",...},    # oldObject is the existing object.    # It is null for CREATE and CONNECT operations.    "oldObject": {"apiVersion":"autoscaling/v1","kind":"Scale",...},    # options contains the options for the operation being admitted, like meta.k8s.io/v1 CreateOptions, UpdateOptions, or DeleteOptions.    # It is null for CONNECT operations.    "options": {"apiVersion":"meta.k8s.io/v1","kind":"UpdateOptions",...},    # dryRun indicates the API request is running in dry run mode and will not be persisted.    # Webhooks with side effects should avoid actuating those side effects when dryRun is true.    # See http://k8s.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request for more details.    "dryRun": false  }}

而Webhooks须要向kube-apiserver回应具备雷同版本的AdmissionReview，并封装成JSON格局，蕴含如下关键字段：

uid：拷贝发送给webhooks的AdmissionReview request.uid字段
allowed：true示意准许；false示意不准许
status：当不准许申请时，能够通过status给出相干起因(http code and message)
patch：base64编码，蕴含mutating admission webhook对申请对象的一系列JSON patch操作
patchType：目前只反对JSONPatch类型

示例如下：

# a webhook response to add that label would be：{  "apiVersion": "admission.k8s.io/v1",  "kind": "AdmissionReview",  "response": {    "uid": "<value from request.uid>",    "allowed": true,    "patchType": "JSONPatch",    "patch": "W3sib3AiOiAiYWRkIiwgInBhdGgiOiAiL3NwZWMvcmVwbGljYXMiLCAidmFsdWUiOiAzfV0="  }}

edge-health-admission实际上就是一个mutating admission webhook，选择性地对endpoints以及node UPDATE申请进行批改，上面将详细分析其原理

edge-health-admission源码剖析

edge-health-admission齐全参考官网示例编写，如下是监听入口：

func (eha *EdgeHealthAdmission) Run(stopCh <-chan struct{}) {    if !cache.WaitForNamedCacheSync("edge-health-admission", stopCh, eha.cfg.NodeInformer.Informer().HasSynced) {        return    }    http.HandleFunc("/node-taint", eha.serveNodeTaint)    http.HandleFunc("/endpoint", eha.serveEndpoint)    server := &http.Server{        Addr: eha.cfg.Addr,    }    go func() {        if err := server.ListenAndServeTLS(eha.cfg.CertFile, eha.cfg.KeyFile); err != http.ErrServerClosed {            klog.Fatalf("ListenAndServeTLS err %+v", err)        }    }()    for {        select {        case <-stopCh:            ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)            defer cancel()            if err := server.Shutdown(ctx); err != nil {                klog.Errorf("Server: program exit, server exit error %+v", err)            }            return        default:        }    }}

这里会注册两种路由处理函数：

node-taint：对应处理函数serveNodeTaint，负责对node UPDATE申请进行更改
endpoint：对应处理函数serveEndpoint，负责对endpoints UPDATE申请进行更改

而这两个函数都会调用serve函数，如下：

// serve handles the http portion of a request prior to handing to an admit functionfunc serve(w http.ResponseWriter, r *http.Request, admit admitFunc) {    var body []byte    if r.Body != nil {        if data, err := ioutil.ReadAll(r.Body); err == nil {            body = data        }    }    // verify the content type is accurate    contentType := r.Header.Get("Content-Type")    if contentType != "application/json" {        klog.Errorf("contentType=%s, expect application/json", contentType)        return    }    klog.V(4).Info(fmt.Sprintf("handling request: %s", body))    // The AdmissionReview that was sent to the webhook    requestedAdmissionReview := admissionv1.AdmissionReview{}    // The AdmissionReview that will be returned    responseAdmissionReview := admissionv1.AdmissionReview{}    deserializer := codecs.UniversalDeserializer()    if _, _, err := deserializer.Decode(body, nil, &requestedAdmissionReview); err != nil {        klog.Error(err)        responseAdmissionReview.Response = toAdmissionResponse(err)    } else {        // pass to admitFunc        responseAdmissionReview.Response = admit(requestedAdmissionReview)    }    // Return the same UID    responseAdmissionReview.Response.UID = requestedAdmissionReview.Request.UID    klog.V(4).Info(fmt.Sprintf("sending response: %+v", responseAdmissionReview.Response))    respBytes, err := json.Marshal(responseAdmissionReview)    if err != nil {        klog.Error(err)    }    if _, err := w.Write(respBytes); err != nil {        klog.Error(err)    }}

serve逻辑如下所示：

解析request.Body为AdmissionReview对象，并赋值给requestedAdmissionReview
对AdmissionReview对象执行admit函数，并赋值给回responseAdmissionReview
设置responseAdmissionReview.Response.UID为申请的AdmissionReview.Request.UID

其中serveNodeTaint以及serveEndpoint对应的admit函数别离为：mutateNodeTaint以及mutateEndpoint，上面顺次剖析：

1、mutateNodeTaint

mutateNodeTaint会对node UPDATE申请依照分布式健康检查后果进行批改：

func (eha *EdgeHealthAdmission) mutateNodeTaint(ar admissionv1.AdmissionReview) *admissionv1.AdmissionResponse {    klog.V(4).Info("mutating node taint")    nodeResource := metav1.GroupVersionResource{Group: "", Version: "v1", Resource: "nodes"}    if ar.Request.Resource != nodeResource {        klog.Errorf("expect resource to be %s", nodeResource)        return nil    }    var node corev1.Node    deserializer := codecs.UniversalDeserializer()    if _, _, err := deserializer.Decode(ar.Request.Object.Raw, nil, &node); err != nil {        klog.Error(err)        return toAdmissionResponse(err)    }    reviewResponse := admissionv1.AdmissionResponse{}    reviewResponse.Allowed = true    if index, condition := util.GetNodeCondition(&node.Status, v1.NodeReady); index != -1 && condition.Status == v1.ConditionUnknown {        if node.Annotations != nil {            var patches []*patch            if healthy, existed := node.Annotations[common.NodeHealthAnnotation]; existed && healthy == common.NodeHealthAnnotationPros {                if index, existed := util.TaintExistsPosition(node.Spec.Taints, common.UnreachableNoExecuteTaint); existed {                    patches = append(patches, &patch{                        OP:   "remove",                        Path: fmt.Sprintf("/spec/taints/%d", index),                    })                    klog.V(4).Infof("UnreachableNoExecuteTaint: remove %d taints %s", index, node.Spec.Taints[index])                }            }            if len(patches) > 0 {                patchBytes, _ := json.Marshal(patches)                reviewResponse.Patch = patchBytes                pt := admissionv1.PatchTypeJSONPatch                reviewResponse.PatchType = &pt            }        }    }    return &reviewResponse}

主体逻辑如下：

查看AdmissionReview.Request.Resource是否为node资源的group/version/kind
将AdmissionReview.Request.Object.Raw转化为node对象
设置AdmissionReview.Response.Allowed为true，示意无论如何都准许该申请
执行帮助边端健康检查外围逻辑：在节点处于ConditionUnknown状态且分布式健康检查后果为失常的状况下，若节点存在NoExecute(node.kubernetes.io/unreachable) taint，则将其移除

总的来说，mutateNodeTaint的作用就是：一直修改被kube-controller-manager更新的节点状态，去掉NoExecute(node.kubernetes.io/unreachable) taint，让节点不会被驱赶

2、mutateEndpoint

mutateEndpoint会对endpoints UPDATE申请依照分布式健康检查后果进行批改：

func (eha *EdgeHealthAdmission) mutateEndpoint(ar admissionv1.AdmissionReview) *admissionv1.AdmissionResponse {    klog.V(4).Info("mutating endpoint")    endpointResource := metav1.GroupVersionResource{Group: "", Version: "v1", Resource: "endpoints"}    if ar.Request.Resource != endpointResource {        klog.Errorf("expect resource to be %s", endpointResource)        return nil    }    var endpoint corev1.Endpoints    deserializer := codecs.UniversalDeserializer()    if _, _, err := deserializer.Decode(ar.Request.Object.Raw, nil, &endpoint); err != nil {        klog.Error(err)        return toAdmissionResponse(err)    }    reviewResponse := admissionv1.AdmissionResponse{}    reviewResponse.Allowed = true    for epSubsetIndex, epSubset := range endpoint.Subsets {        for notReadyAddrIndex, EndpointAddress := range epSubset.NotReadyAddresses {            if node, err := eha.nodeLister.Get(*EndpointAddress.NodeName); err == nil {                if index, condition := util.GetNodeCondition(&node.Status, v1.NodeReady); index != -1 && condition.Status == v1.ConditionUnknown {                    if node.Annotations != nil {                        var patches []*patch                        if healthy, existed := node.Annotations[common.NodeHealthAnnotation]; existed && healthy == common.NodeHealthAnnotationPros {                            // TODO: handle readiness probes failure                            // Remove address on node from endpoint notReadyAddresses                            patches = append(patches, &patch{                                OP:   "remove",                                Path: fmt.Sprintf("/subsets/%d/notReadyAddresses/%d", epSubsetIndex, notReadyAddrIndex),                            })                            // Add address on node to endpoint readyAddresses                            TargetRef := map[string]interface{}{}                            TargetRef["kind"] = EndpointAddress.TargetRef.Kind                            TargetRef["namespace"] = EndpointAddress.TargetRef.Namespace                            TargetRef["name"] = EndpointAddress.TargetRef.Name                            TargetRef["uid"] = EndpointAddress.TargetRef.UID                            TargetRef["apiVersion"] = EndpointAddress.TargetRef.APIVersion                            TargetRef["resourceVersion"] = EndpointAddress.TargetRef.ResourceVersion                            TargetRef["fieldPath"] = EndpointAddress.TargetRef.FieldPath                            patches = append(patches, &patch{                                OP:   "add",                                Path: fmt.Sprintf("/subsets/%d/addresses/0", epSubsetIndex),                                Value: map[string]interface{}{                                    "ip":        EndpointAddress.IP,                                    "hostname":  EndpointAddress.Hostname,                                    "nodeName":  EndpointAddress.NodeName,                                    "targetRef": TargetRef,                                },                            })                            if len(patches) != 0 {                                patchBytes, _ := json.Marshal(patches)                                reviewResponse.Patch = patchBytes                                pt := admissionv1.PatchTypeJSONPatch                                reviewResponse.PatchType = &pt                            }                        }                    }                }            } else {                klog.Errorf("Get pod's node err %+v", err)            }        }    }    return &reviewResponse}

主体逻辑如下：

查看AdmissionReview.Request.Resource是否为endpoints资源的group/version/kind
将AdmissionReview.Request.Object.Raw转化为endpoints对象
设置AdmissionReview.Response.Allowed为true，示意无论如何都准许该申请
遍历endpoints.Subset.NotReadyAddresses，如果EndpointAddress所在节点处于ConditionUnknown状态且分布式健康检查后果为失常，则将该EndpointAddress从endpoints.Subset.NotReadyAddresses移到endpoints.Subset.Addresses

总的来说，mutateEndpoint的作用就是：一直修改被kube-controller-manager更新的endpoints状态，将分布式健康检查失常节点上的负载从endpoints.Subset.NotReadyAddresses移到endpoints.Subset.Addresses中，让服务仍旧可用

总结

SuperEdge分布式健康检查性能由边端的edge-health-daemon以及云端的edge-health-admission组成：
- edge-health-daemon：对同区域边缘节点执行分布式健康检查，并向apiserver发送衰弱状态投票后果(给node打annotation)
- edge-health-admission：一直依据node edge-health annotation调整kube-controller-manager设置的node taint(去掉NoExecute taint)以及endpoints(将失联节点上的pods从endpoint subsets notReadyAddresses移到addresses中)，从而实现云端和边端独特决定节点状态
之所以创立edge-health-admission云端组件，是因为当云边断连时，kube-controller-manager会将失联的节点置为ConditionUnknown状态，并增加NoSchedule和NoExecute的taints；同时失联的节点上的pod从Service的Endpoint列表中移除。当edge-health-daemon在边端依据健康检查判断节点状态失常时，会更新node：去掉NoExecute taint。然而在node胜利更新之后又会被kube-controller-manager给刷回去(再次增加NoExecute taint)，因而必须增加Kubernetes mutating admission webhook也即edge-health-admission，将kube-controller-manager对node api resource的更改做调整，最终实现分布式健康检查成果
Kubernetes Admission Controllers是kube-apiserver解决api申请的某个环节，用于在api申请认证&鉴权之后，对象长久化之前进行调用，对申请进行校验或者批改(or both)；包含多种admission，大多数都内嵌在kube-apiserver代码中了。其中MutatingAdmissionWebhook以及ValidatingAdmissionWebhook controller比拟非凡，它们别离会调用内部结构的mutating admission control webhooks以及validating admission control webhooks
Admission Webhooks是一个HTTP回调服务，承受AdmissionReview申请并进行解决，依照解决形式的不同，能够将Admission Webhooks分类如下：
- validating admission webhook：通过ValidatingWebhookConfiguration配置，会对api申请进行准入校验，然而不能批改申请对象
- mutating admission webhook：通过MutatingWebhookConfiguration配置，会对api申请进行准入校验以及批改申请对象
kube-apiserver会发送AdmissionReview(apiGroup: admission.k8s.io，apiVersion：v1 or v1beta1)给Webhooks，并封装成JSON格局；而Webhooks须要向kube-apiserver回应具备雷同版本的AdmissionReview，并封装成JSON格局，蕴含如下关键字段：
- uid：拷贝发送给webhooks的AdmissionReview request.uid字段
- allowed：true示意准许；false示意不准许
- status：当不准许申请时，能够通过status给出相干起因(http code and message)
- patch：base64编码，蕴含mutating admission webhook对申请对象的一系列JSON patch操作
- patchType：目前只反对JSONPatch类型
edge-health-admission实际上就是一个mutating admission webhook，选择性地对endpoints以及node UPDATE申请进行批改，蕴含如下解决逻辑：
- mutateNodeTaint：一直修改被kube-controller-manager更新的节点状态，去掉NoExecute(node.kubernetes.io/unreachable) taint，让节点不会被驱赶
- mutateEndpoint：一直修改被kube-controller-manager更新的endpoints状态，将分布式健康检查失常节点上的负载从endpoints.Subset.NotReadyAddresses移到endpoints.Subset.Addresses中，让服务仍旧可用

【腾讯云原生】云说新品、云研新术、云游新活、云赏资讯，扫码关注同名公众号，及时获取更多干货！！