关于SegmentFault:一文读懂-Kubernetes-APIServer-原理

整个 Kubernetes 技术体系由申明式 API 以及 Controller 形成，而 kube-apiserver 是 Kubernetes 的申明式 api server，并为其它组件交互提供了桥梁。因而加深对 kube-apiserver 的了解就显得至关重要了。

kube-apiserver 作为整个 Kubernetes 集群操作 etcd 的惟一入口，负责 Kubernetes 各资源的认证 & 鉴权，校验以及 CRUD 等操作，提供 RESTful APIs，供其它组件调用：

kube-apiserver 蕴含三种 APIServer：

aggregatorServer：负责解决 apiregistration.k8s.io 组下的 APIService 资源申请，同时将来自用户的申请拦挡转发给 aggregated server(AA)
kubeAPIServer：负责对申请的一些通用解决，包含：认证、鉴权以及各个内建资源 (pod, deployment，service and etc) 的 REST 服务等
apiExtensionsServer：负责 CustomResourceDefinition（CRD）apiResources 以及 apiVersions 的注册，同时解决 CRD 以及相应 CustomResource（CR）的 REST 申请(如果对应 CR 不能被解决的话则会返回 404)，也是 apiserver Delegation 的最初一环

另外还包含 bootstrap-controller，次要负责 Kubernetes default apiserver service 的创立以及治理。

接下来将对上述组件进行概览性总结。

apiserver bootstrap-controller创立 & 运行逻辑在 k8s.io/kubernetes/pkg/master 目录
bootstrap-controller次要用于创立以及保护外部 kubernetes default apiserver service
kubernetes default apiserver service spec.selector为空，这是 default apiserver service 与其它失常 service 的最大区别，表明了这个非凡的 service 对应的 endpoints 不禁 endpoints controller 管制，而是间接受 kube-apiserver bootstrap-controller 治理(maintained by this code, not by the pod selector)
bootstrap-controller的几个次要性能如下：
- 创立 default、kube-system 和 kube-public 以及 kube-node-lease 命名空间
- 创立 & 保护 kubernetes default apiserver service 以及对应的 endpoint
- 提供基于 Service ClusterIP 的查看及修复性能 (--service-cluster-ip-range 指定范畴)
- 提供基于 Service NodePort 的查看及修复性能 (--service-node-port-range 指定范畴)

// k8s.io/kubernetes/pkg/master/controller.go:142
// Start begins the core controller loops that must exist for bootstrapping
// a cluster.
func (c *Controller) Start() {
    if c.runner != nil {return}
    // Reconcile during first run removing itself until server is ready.
    endpointPorts := createEndpointPortSpec(c.PublicServicePort, "https", c.ExtraEndpointPorts)
    if err := c.EndpointReconciler.RemoveEndpoints(kubernetesServiceName, c.PublicIP, endpointPorts); err != nil {klog.Errorf("Unable to remove old endpoints from kubernetes service: %v", err)
    }
    repairClusterIPs := servicecontroller.NewRepair(c.ServiceClusterIPInterval, c.ServiceClient, c.EventClient, &c.ServiceClusterIPRange, c.ServiceClusterIPRegistry, &c.SecondaryServiceClusterIPRange, c.SecondaryServiceClusterIPRegistry)
    repairNodePorts := portallocatorcontroller.NewRepair(c.ServiceNodePortInterval, c.ServiceClient, c.EventClient, c.ServiceNodePortRange, c.ServiceNodePortRegistry)
    // run all of the controllers once prior to returning from Start.
    if err := repairClusterIPs.RunOnce(); err != nil {
        // If we fail to repair cluster IPs apiserver is useless. We should restart and retry.
        klog.Fatalf("Unable to perform initial IP allocation check: %v", err)
    }
    if err := repairNodePorts.RunOnce(); err != nil {
        // If we fail to repair node ports apiserver is useless. We should restart and retry.
        klog.Fatalf("Unable to perform initial service nodePort check: %v", err)
    }
    // 定期执行 bootstrap controller 次要的四个性能(reconciliation)  
    c.runner = async.NewRunner(c.RunKubernetesNamespaces, c.RunKubernetesService, repairClusterIPs.RunUntil, repairNodePorts.RunUntil)
    c.runner.Start()}

更多代码原理详情，参考 kubernetes-reading-notes。

KubeAPIServer 次要提供对内建 API Resources 的操作申请，为 Kubernetes 中各 API Resources 注册路由信息，同时裸露 RESTful API，使集群中以及集群外的服务都能够通过 RESTful API 操作 Kubernetes 中的资源

另外，kubeAPIServer 是整个 Kubernetes apiserver 的外围，上面将要讲述的 aggregatorServer 以及 apiExtensionsServer 都是建设在 kubeAPIServer 根底上进行扩大的(补充了 Kubernetes 对用户自定义资源的能力反对)

kubeAPIServer 最外围的性能是为 Kubernetes 内置资源增加路由，如下：

调用 m.InstallLegacyAPI 将外围 API Resources 增加到路由中，在 apiserver 中即是以 /api 结尾的 resource；
调用 m.InstallAPIs 将扩大的 API Resources 增加到路由中，在 apiserver 中即是以 /apis 结尾的 resource；

// k8s.io/kubernetes/pkg/master/master.go:332
// New returns a new instance of Master from the given config.
// Certain config fields will be set to a default value if unset.
// Certain config fields must be specified, including:
//   KubeletClientConfig
func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*Master, error) {
    ...
    // 装置 LegacyAPI(core API)
    // install legacy rest storage
    if c.ExtraConfig.APIResourceConfigSource.VersionEnabled(apiv1.SchemeGroupVersion) {
        legacyRESTStorageProvider := corerest.LegacyRESTStorageProvider{
            StorageFactory:              c.ExtraConfig.StorageFactory,
            ProxyTransport:              c.ExtraConfig.ProxyTransport,
            KubeletClientConfig:         c.ExtraConfig.KubeletClientConfig,
            EventTTL:                    c.ExtraConfig.EventTTL,
            ServiceIPRange:              c.ExtraConfig.ServiceIPRange,
            SecondaryServiceIPRange:     c.ExtraConfig.SecondaryServiceIPRange,
            ServiceNodePortRange:        c.ExtraConfig.ServiceNodePortRange,
            LoopbackClientConfig:        c.GenericConfig.LoopbackClientConfig,
            ServiceAccountIssuer:        c.ExtraConfig.ServiceAccountIssuer,
            ServiceAccountMaxExpiration: c.ExtraConfig.ServiceAccountMaxExpiration,
            APIAudiences:                c.GenericConfig.Authentication.APIAudiences,
        }
        if err := m.InstallLegacyAPI(&c, c.GenericConfig.RESTOptionsGetter, legacyRESTStorageProvider); err != nil {return nil, err}
    }
    ...
    // 装置 APIs(named groups apis)
    if err := m.InstallAPIs(c.ExtraConfig.APIResourceConfigSource, c.GenericConfig.RESTOptionsGetter, restStorageProviders...); err != nil {return nil, err}
    ...
    return m, nil
}

整个 kubeAPIServer 提供了三类 API Resource 接口：

core group：次要在 /api/v1 下；
named groups：其 path 为 /apis/$GROUP/$VERSION；
零碎状态的一些 API：如/metrics、/version 等；

而 API 的 URL 大抵以 /apis/{group}/{version}/namespaces/{namespace}/resource/{name} 组成，构造如下图所示：

kubeAPIServer 会为每种 API 资源创立对应的 RESTStorage，RESTStorage 的目标是将每种资源的拜访门路及其后端存储的操作对应起来：通过结构的 REST Storage 实现的接口判断该资源能够执行哪些操作（如：create、update 等），将其对应的操作存入到 action 中，每一个操作对应一个规范的 REST method，如 create 对应 REST method 为 POST，而 update 对应 REST method 为 PUT。最终依据 actions 数组顺次遍历，对每一个操作增加一个 handler(handler 对应 REST Storage 实现的相干接口)，并注册到 route，最终对外提供 RESTful API，如下：

// m.GenericAPIServer.InstallLegacyAPIGroup --> s.installAPIResources --> apiGroupVersion.InstallREST --> installer.Install --> a.registerResourceHandlers
// k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/installer.go:181
func (a *APIInstaller) registerResourceHandlers(path string, storage rest.Storage, ws *restful.WebService) (*metav1.APIResource, error) {
    ...
    // 1、判断该 resource 实现了哪些 REST 操作接口，以此来判断其反对的 verbs 以便为其增加路由
    // what verbs are supported by the storage, used to know what verbs we support per path
    creater, isCreater := storage.(rest.Creater)
    namedCreater, isNamedCreater := storage.(rest.NamedCreater)
    lister, isLister := storage.(rest.Lister)
    getter, isGetter := storage.(rest.Getter)
    ...
    // 2、为 resource 增加对应的 actions(+ 依据是否反对 namespace)
    // Get the list of actions for the given scope.
    switch {
    case !namespaceScoped:
        // Handle non-namespace scoped resources like nodes.
        resourcePath := resource
        resourceParams := params
        itemPath := resourcePath + "/{name}"
        nameParams := append(params, nameParam)
        proxyParams := append(nameParams, pathParam)
        ...
        // Handler for standard REST verbs (GET, PUT, POST and DELETE).
        // Add actions at the resource path: /api/apiVersion/resource
        actions = appendIf(actions, action{"LIST", resourcePath, resourceParams, namer, false}, isLister)
        actions = appendIf(actions, action{"POST", resourcePath, resourceParams, namer, false}, isCreater)
        ...
    }
    ...
    // 3、从 rest.Storage 到 restful.Route 映射
    // 为每个操作增加对应的 handler
    for _, action := range actions {
        ...
        switch action.Verb {
        ...
        case "POST": // Create a resource.
            var handler restful.RouteFunction
            // 4、初始化 handler
            if isNamedCreater {handler = restfulCreateNamedResource(namedCreater, reqScope, admit)
            } else {handler = restfulCreateResource(creater, reqScope, admit)
            }
            handler = metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, handler)
            ...
            // 5、route 与 handler 进行绑定    
            route := ws.POST(action.Path).To(handler).
                Doc(doc).
                Param(ws.QueryParameter("pretty", "If'true', then the output is pretty printed.")).
                Operation("create"+namespaced+kind+strings.Title(subresource)+operationSuffix).
                Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
                Returns(http.StatusOK, "OK", producedObject).
                // TODO: in some cases, the API may return a v1.Status instead of the versioned object
                // but currently go-restful can't handle multiple different objects being returned.
                Returns(http.StatusCreated, "Created", producedObject).
                Returns(http.StatusAccepted, "Accepted", producedObject).
                Reads(defaultVersionedObject).
                Writes(producedObject)
            if err := AddObjectParams(ws, route, versionedCreateOptions); err != nil {return nil, err}
            addParams(route, action.Params)
            // 6、增加到路由中    
            routes = append(routes, route)
        case "DELETE": // Delete a resource.
        ...
        default:
            return nil, fmt.Errorf("unrecognized action verb: %s", action.Verb)
        }
        for _, route := range routes {
            route.Metadata(ROUTE_META_GVK, metav1.GroupVersionKind{
                Group:   reqScope.Kind.Group,
                Version: reqScope.Kind.Version,
                Kind:    reqScope.Kind.Kind,
            })
            route.Metadata(ROUTE_META_ACTION, strings.ToLower(action.Verb))
            ws.Route(route)
        }
        // Note: update GetAuthorizerAttributes() when adding a custom handler.}
    ...
}

kubeAPIServer 代码构造整顿如下：

1. apiserver 整体启动逻辑 k8s.io/kubernetes/cmd/kube-apiserver
2. apiserver bootstrap-controller 创立 & 运行逻辑 k8s.io/kubernetes/pkg/master
3. API Resource 对应后端 RESTStorage(based on genericregistry.Store)创立 k8s.io/kubernetes/pkg/registry
4. aggregated-apiserver 创立 & 解决逻辑 k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator
5. extensions-apiserver 创立 & 解决逻辑 k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver
6. apiserver 创立 & 运行 k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/server
7. 注册 API Resource 资源解决 handler(InstallREST&Install®isterResourceHandlers) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints
8. 创立存储后端(etcdv3) k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/storage
9. genericregistry.Store.CompleteWithOptions 初始化 k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/registry

调用链整顿如下：

更多代码原理详情，参考 kubernetes-reading-notes。

aggregatorServer 次要用于解决扩大 Kubernetes API Resources 的第二种形式 Aggregated APIServer(AA)，将 CR 申请代理给 AA：

这里联合 Kubernetes 官网给出的 aggregated apiserver 例子 sample-apiserver，总结原理如下：

aggregatorServer 通过 APIServices 对象关联到某个 Service 来进行申请的转发，其关联的 Service 类型进一步决定了申请转发的模式。aggregatorServer 包含一个 GenericAPIServer 和保护本身状态的 Controller。其中GenericAPIServer 次要解决 apiregistration.k8s.io 组下的 APIService 资源申请，而 Controller 包含：
- apiserviceRegistrationController：负责依据 APIService 定义的 aggregated server service 构建代理，将 CR 的申请转发给后端的 aggregated server
- availableConditionController：保护 APIServices 的可用状态，包含其援用 Service 是否可用等；
- autoRegistrationController：用于放弃 API 中存在的一组特定的 APIServices；
- crdRegistrationController：负责将 CRD GroupVersions 主动注册到 APIServices 中；
- openAPIAggregationController：将 APIServices 资源的变动同步至提供的 OpenAPI 文档；

apiserviceRegistrationController 负责依据 APIService 定义的 aggregated server service 构建代理，将 CR 的申请转发给后端的 aggregated server。apiService 有两种类型：Local(Service 为空)以及 Service(Service 非空)。apiserviceRegistrationController 负责对这两种类型 apiService 设置代理：Local 类型会间接路由给 kube-apiserver 进行解决；而 Service 类型则会设置代理并将申请转化为对 aggregated Service 的申请(proxyPath := “/apis/” + apiService.Spec.Group + “/” + apiService.Spec.Version)，而申请的负载平衡策略则是优先本地拜访 kube-apiserver(如果 service 为 kubernetes default apiserver service:443)=> 通过 service ClusterIP:Port 拜访(默认) 或者通过随机抉择 service endpoint backend 进行拜访：

func (s *APIAggregator) AddAPIService(apiService *v1.APIService) error {
  ...
    proxyPath := "/apis/" + apiService.Spec.Group + "/" + apiService.Spec.Version
    // v1. is a special case for the legacy API.  It proxies to a wider set of endpoints.
    if apiService.Name == legacyAPIServiceName {proxyPath = "/api"}
    // register the proxy handler
    proxyHandler := &proxyHandler{
        localDelegate:   s.delegateHandler,
        proxyClientCert: s.proxyClientCert,
        proxyClientKey:  s.proxyClientKey,
        proxyTransport:  s.proxyTransport,
        serviceResolver: s.serviceResolver,
        egressSelector:  s.egressSelector,
    }
  ...
    s.proxyHandlers[apiService.Name] = proxyHandler
    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle(proxyPath, proxyHandler)
    s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandlePrefix(proxyPath+"/", proxyHandler)
  ...
    // it's time to register the group aggregation endpoint
    groupPath := "/apis/" + apiService.Spec.Group
    groupDiscoveryHandler := &apiGroupHandler{
        codecs:    aggregatorscheme.Codecs,
        groupName: apiService.Spec.Group,
        lister:    s.lister,
        delegate:  s.delegateHandler,
    }
    // aggregation is protected
    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle(groupPath, groupDiscoveryHandler)
    s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandle(groupPath+"/", groupDiscoveryHandler)
    s.handledGroups.Insert(apiService.Spec.Group)
    return nil
}
// k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:109
func (r *proxyHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
    // 加载 roxyHandlingInfo 解决申请  
    value := r.handlingInfo.Load()
    if value == nil {r.localDelegate.ServeHTTP(w, req)
        return
    }
    handlingInfo := value.(proxyHandlingInfo)
  ...
    // 判断 APIService 服务是否失常
    if !handlingInfo.serviceAvailable {proxyError(w, req, "service unavailable", http.StatusServiceUnavailable)
        return
    }
    // 将原始申请转化为对 APIService 的申请
    // write a new location based on the existing request pointed at the target service
    location := &url.URL{}
    location.Scheme = "https"
    rloc, err := r.serviceResolver.ResolveEndpoint(handlingInfo.serviceNamespace, handlingInfo.serviceName, handlingInfo.servicePort)
    if err != nil {klog.Errorf("error resolving %s/%s: %v", handlingInfo.serviceNamespace, handlingInfo.serviceName, err)
        proxyError(w, req, "service unavailable", http.StatusServiceUnavailable)
        return
    }
    location.Host = rloc.Host
    location.Path = req.URL.Path
    location.RawQuery = req.URL.Query().Encode()
    newReq, cancelFn := newRequestForProxy(location, req)
    defer cancelFn()
   ...
    proxyRoundTripper = transport.NewAuthProxyRoundTripper(user.GetName(), user.GetGroups(), user.GetExtra(), proxyRoundTripper)
    handler := proxy.NewUpgradeAwareHandler(location, proxyRoundTripper, true, upgrade, &responder{w: w})
    handler.ServeHTTP(w, newReq)
}

$ kubectl get APIService           
NAME                                   SERVICE                      AVAILABLE   AGE
...
v1.apps                                Local                        True        50d
...
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        50d
...

# default APIServices
$ kubectl get -o yaml APIService/v1.apps
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    kube-aggregator.kubernetes.io/automanaged: onstart
  name: v1.apps
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1.apps
spec:
  group: apps
  groupPriorityMinimum: 17800
  version: v1
  versionPriority: 15
status:
  conditions:
  - lastTransitionTime: "2020-10-20T10:39:48Z"
    message: Local APIServices are always available
    reason: Local
    status: "True"
    type: Available

# aggregated server    
$ kubectl get -o yaml APIService/v1beta1.metrics.k8s.io
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: v1beta1.metrics.k8s.io
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2020-12-05T00:50:48Z"
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available

# CRD
$ kubectl get -o yaml APIService/v1.duyanghao.example.com
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    kube-aggregator.kubernetes.io/automanaged: "true"
  name: v1.duyanghao.example.com
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1.duyanghao.example.com
spec:
  group: duyanghao.example.com
  groupPriorityMinimum: 1000
  version: v1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2020-12-11T08:45:37Z"
    message: Local APIServices are always available
    reason: Local
    status: "True"
    type: Available

aggregatorServer 创立过程中会依据所有 kube-apiserver 定义的 API 资源创立默认的 APIService 列表，名称即是$VERSION.$GROUP，这些 APIService 都会有标签kube-aggregator.kubernetes.io/automanaged: onstart，例如：v1.apps apiService。autoRegistrationController 创立并保护这些列表中的 APIService，也即咱们看到的 Local apiService；对于自定义的 APIService(aggregated server)，则不会对其进行解决
aggregated server 实现 CR(自定义 API 资源) 的 CRUD API 接口，并能够灵便抉择后端存储，能够与 core kube-apiserver 一起专用 etcd，也可本人独立部署 etcd 数据库或者其它数据库。aggregated server 实现的 CR API 门路为：/apis/$GROUP/$VERSION，具体到 sample apiserver 为：/apis/wardle.example.com/v1alpha1，上面的资源类型有：flunders 以及 fischers
aggregated server 通过部署 APIService 类型资源，service fields 指向对应的 aggregated server service 实现与 core kube-apiserver 的集成与交互

sample-apiserver 目录构造如下，可参考编写本人的 aggregated server：

staging/src/k8s.io/sample-apiserver
├── artifacts
│   ├── example
│   │   ├── apiservice.yaml
      ...
├── hack
├── main.go
└── pkg
├── admission
├── apis
├── apiserver
├── cmd
├── generated
│   ├── clientset
│   │   └── versioned
              ...
│   │       └── typed
│   │           └── wardle
│   │               ├── v1alpha1
│   │               └── v1beta1
│   ├── informers
│   │   └── externalversions
│   │       └── wardle
│   │           ├── v1alpha1
│   │           └── v1beta1
│   ├── listers
│   │   └── wardle
│   │       ├── v1alpha1
│   │       └── v1beta1
└── registry

其中，artifacts 用于部署 yaml 示例
hack 目录寄存主动脚本(eg: update-codegen)
main.go 是 aggregated server 启动入口；pkg/cmd 负责启动 aggregated server 具体逻辑；pkg/apiserver 用于 aggregated server 初始化以及路由注册
pkg/apis 负责相干 CR 的构造体定义，主动生成(update-codegen)
pkg/admission 负责准入的相干代码
pkg/generated 负责生成拜访 CR 的 clientset，informers，以及 listers
pkg/registry 目录负责 CR 相干的 RESTStorage 实现

更多代码原理详情，参考 kubernetes-reading-notes。

apiExtensionsServer 次要负责 CustomResourceDefinition（CRD）apiResources 以及 apiVersions 的注册，同时解决 CRD 以及相应 CustomResource（CR）的 REST 申请(如果对应 CR 不能被解决的话则会返回 404)，也是 apiserver Delegation 的最初一环

原理总结如下：

Custom Resource，简称 CR，是 Kubernetes 自定义资源类型，与之绝对应的就是 Kubernetes 内置的各种资源类型，例如 Pod、Service 等。利用 CR 咱们能够定义任何想要的资源类型
CRD 通过 yaml 文件的模式向 Kubernetes 注册 CR 实现自定义 api-resources，属于第二种扩大 Kubernetes API 资源的形式，也是广泛应用的一种
APIExtensionServer 负责 CustomResourceDefinition（CRD）apiResources 以及 apiVersions 的注册，同时解决 CRD 以及相应 CustomResource（CR）的 REST 申请(如果对应 CR 不能被解决的话则会返回 404)，也是 apiserver Delegation 的最初一环
crdRegistrationController负责将 CRD GroupVersions 主动注册到 APIServices 中。具体逻辑为：枚举所有 CRDs，而后依据 CRD 定义的 crd.Spec.Group 以及 crd.Spec.Versions 字段构建 APIService，并增加到 autoRegisterController.apiServicesToSync 中，由 autoRegisterController 进行创立以及保护操作。这也是为什么创立完 CRD 后会产生对应的 APIService 对象

APIExtensionServer 蕴含的 controller 以及性能如下所示：

openapiController：将 crd 资源的变动同步至提供的 OpenAPI 文档，可通过拜访 /openapi/v2 进行查看；
crdController：负责将 crd 信息注册到 apiVersions 和 apiResources 中，两者的信息可通过 kubectl api-versions 和 kubectl api-resources 查看；

kubectl api-versions命令返回所有 Kubernetes 集群资源的版本信息（理论收回了两个申请，别离是 https://127.0.0.1:6443/api 以及https://127.0.0.1:6443/apis，并在最初将两个申请的返回后果进行了合并）

$ kubectl -v=8 api-versions 
I1211 11:44:50.276446   22493 loader.go:375] Config loaded from file:  /root/.kube/config
I1211 11:44:50.277005   22493 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s
...
I1211 11:44:50.290265   22493 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]}
I1211 11:44:50.293673   22493 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s
...
I1211 11:44:50.298360   22493 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars]
apiextensions.k8s.io/v1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
authentication.k8s.io/v1beta1
...
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1

kubectl api-resources命令就是先获取所有 API 版本信息，而后对每一个 API 版本调用接口获取该版本下的所有 API 资源类型

$ kubectl -v=8 api-resources
 5077 loader.go:375] Config loaded from file:  /root/.kube/config
 I1211 15:19:47.593450   15077 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s
 I1211 15:19:47.602273   15077 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]}
 I1211 15:19:47.606279   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s
 I1211 15:19:47.610333   15077 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars]
 I1211 15:19:47.614700   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/batch/v1?timeout=32s
 I1211 15:19:47.614804   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/authentication.k8s.io/v1?timeout=32s
 I1211 15:19:47.615687   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/auth.tkestack.io/v1?timeout=32s
 https://127.0.0.1:6443/apis/authentication.k8s.io/v1beta1?timeout=32s
 I1211 15:19:47.616794   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/coordination.k8s.io/v1?timeout=32s
 I1211 15:19:47.616863   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/apps/v1?timeout=32s
 ...
 NAME                              SHORTNAMES   APIGROUP                       NAMESPACED   KIND
 bindings                                                                      true         Binding
 endpoints                         ep                                          true         Endpoints
 events                            ev                                          true         Event
 limitranges                       limits                                      true         LimitRange
 namespaces                        ns                                          false        Namespace
 nodes                             no                                          false        Node
 ...

namingController：查看 crd obj 中是否有命名抵触，可在 crd .status.conditions 中查看；
establishingController：查看 crd 是否处于失常状态，可在 crd .status.conditions 中查看；
nonStructuralSchemaController：查看 crd obj 构造是否失常，可在 crd .status.conditions 中查看；
apiApprovalController：查看 crd 是否遵循 Kubernetes API 申明策略，可在 crd .status.conditions 中查看；
finalizingController：相似于 finalizes 的性能，与 CRs 的删除无关；

总结 CR CRUD APIServer 解决逻辑如下：

createAPIExtensionsServer=>NewCustomResourceDefinitionHandler=>crdHandler=> 注册 CR CRUD API 接口：

// New returns a new instance of CustomResourceDefinitions from the given config.
func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*CustomResourceDefinitions, error) {
  ...
    crdHandler, err := NewCustomResourceDefinitionHandler(
      versionDiscoveryHandler,
        groupDiscoveryHandler,
      s.Informers.Apiextensions().V1().CustomResourceDefinitions(),
        delegateHandler,
      c.ExtraConfig.CRDRESTOptionsGetter,
        c.GenericConfig.AdmissionControl,
      establishingController,
        c.ExtraConfig.ServiceResolver,
      c.ExtraConfig.AuthResolverWrapper,
        c.ExtraConfig.MasterCount,
        s.GenericAPIServer.Authorizer,
        c.GenericConfig.RequestTimeout,
        time.Duration(c.GenericConfig.MinRequestTimeout)*time.Second,
        apiGroupInfo.StaticOpenAPISpec,
        c.GenericConfig.MaxRequestBodyBytes,
    )
    if err != nil {return nil, err}
    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle("/apis", crdHandler)
    s.GenericAPIServer.Handler.NonGoRestfulMux.HandlePrefix("/apis/", crdHandler)
    ...
    return s, nil
}

crdHandler 解决逻辑如下：

解析 req(GET /apis/duyanghao.example.com/v1/namespaces/default/students)，依据申请门路中的 group(duyanghao.example.com)，version(v1)，以及 resource 字段 (students) 获取对应 CRD 内容(crd, err := r.crdLister.Get(crdName))
通过 crd.UID 以及 crd.Name 获取 crdInfo，若不存在则创立对应的 crdInfo(crdInfo, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name))。crdInfo 中蕴含了 CRD 定义以及该 CRD 对应 Custom Resource 的 customresource.REST storage
customresource.REST storage 由 CR 对应的 Group(duyanghao.example.com)，Version(v1)，Kind(Student)，Resource(students)等创立实现，因为 CR 在 Kubernetes 代码中并没有具体构造体定义，所以这里会先初始化一个范型构造体 Unstructured(用于保留所有类型的 Custom Resource)，并对该构造体进行 SetGroupVersionKind 操作(设置具体 Custom Resource Type)

从 customresource.REST storage 获取 Unstructured 构造体后会对其进行相应转换而后返回

// k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:223
func (r *crdHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {ctx := req.Context()
  requestInfo, ok := apirequest.RequestInfoFrom(ctx)
  ...
  crdName := requestInfo.Resource + "." + requestInfo.APIGroup
  crd, err := r.crdLister.Get(crdName)
  ...
  crdInfo, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name)
  verb := strings.ToUpper(requestInfo.Verb)
  resource := requestInfo.Resource
  subresource := requestInfo.Subresource
  scope := metrics.CleanScope(requestInfo)
  ...
  switch {
  case subresource == "status" && subresources != nil && subresources.Status != nil:
      handlerFunc = r.serveStatus(w, req, requestInfo, crdInfo, terminating, supportedTypes)
  case subresource == "scale" && subresources != nil && subresources.Scale != nil:
      handlerFunc = r.serveScale(w, req, requestInfo, crdInfo, terminating, supportedTypes)
  case len(subresource) == 0:
      handlerFunc = r.serveResource(w, req, requestInfo, crdInfo, terminating, supportedTypes)
  default:
      responsewriters.ErrorNegotiated(apierrors.NewNotFound(schema.GroupResource{Group: requestInfo.APIGroup, Resource: requestInfo.Resource}, requestInfo.Name),
          Codecs, schema.GroupVersion{Group: requestInfo.APIGroup, Version: requestInfo.APIVersion}, w, req,
      )
  }
  if handlerFunc != nil {handlerFunc = metrics.InstrumentHandlerFunc(verb, requestInfo.APIGroup, requestInfo.APIVersion, resource, subresource, scope, metrics.APIServerComponent, handlerFunc)
      handler := genericfilters.WithWaitGroup(handlerFunc, longRunningFilter, crdInfo.waitGroup)
      handler.ServeHTTP(w, req)
      return
  }
}

更多代码原理详情，参考 kubernetes-reading-notes。

本文从源码层面对 Kubernetes apiserver 进行了一个概览性总结，包含：aggregatorServer，kubeAPIServer，apiExtensionsServer 以及 bootstrap-controller 等。通过浏览本文能够对 apiserver 外部原理有一个大抵的了解，另外也有助于后续深入研究

kubernetes-reading-notes

【腾讯云原生】云说新品、云研新术、云游新活、云赏资讯，扫码关注同名公众号，及时获取更多干货！！

关于SegmentFault:一文读懂-Kubernetes-APIServer-原理

前言

整体组件性能

bootstrap-controller

kubeAPIServer

aggregatorServer

apiExtensionsServer

Conclusion

Refs