关于kubernetes:Istioctl设计分析

前言

为了不便用户装置 istio 和操作相应的 manifest，社区将本来的纯 helm 装置模式演变成现在的 cli 和 operator 模式 (https://docs.google.com/docum…)。同时为了保障后向兼容和良好的代码复用，cli 复用 operator 提供的命令行 api，operator 命令行基于 helm 的 chart 渲染进行实现。根本的模式构造如下图所示：

因而，本文从 istioctl 提供的 install 和 manifest 指令登程，依照 istioctl->operator->helm 的方向进行剖析，重点形容 operator 中 cmd 的实现流程。

Install

istioctl

命令行参数与实例

此节可参考 https://istio.io/latest/docs/… 官网文档。

代码入口

istioctl 实质上是利用 cobra 编写的命令行程序，在 /istio/istioctl/cmd/root.go 中将 InstallCmd 退出 RootCmd 中，而 InstallCmd 由 operator 实现。

installCmd := mesh.InstallCmd(loggingOptions)
hideInheritedFlags(installCmd, FlagNamespace, FlagIstioNamespace, FlagCharts)
rootCmd.AddCommand(installCmd)

operator

代码流程与重要函数

整体构造如下图所示。

接下来，以 istioctl install –set installPackagePath=/istio/manifests –set profile=empty -f test0.yaml -f test1.yaml 为例，剖析各个函数。其中 test0.yaml 和 test1.yaml 的内容如下图所示。

# test0.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: minimal
  hub: docker.io/istio
  tag: 1.1.4
  components:
    pilot:
      enabled: true

# test1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  hub: docker.io/istio
  tag: 1.1.5
  components:
    pilot:
      enabled: true

ParseYAMLFiles

ParseYAMLFiles 从左到右读取并笼罩 - f 标记所传入的 IstioOperator yaml，并提取出其中的 profile 字段。因而提取出的 profile 为 minimal，test0.yaml 和 test1.yaml 的初步处理结果 temp0.yaml 如下所示。

# temp0.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: minimal
  hub: docker.io/istio
  tag: 1.1.5
  components:
    pilot:
      enabled: true

ReadYaml(And)Profile

ReadYamlProfile 依据 –set profile=empty 对 ParseYAMLFiles 返回的 profile 后果进行笼罩，因而最终的 profile 为 empty。

GetProfileYAML

GetProfileYAML 从 –set installPackagePath=/istio/manifests 指定的目录读取 empty yaml，因为 istio 所提供的所有内置 profile 都基于 default profile，因而利用 empty yaml 去笼罩 default profile，最终失去 temp1.yaml。

#empty yaml

# The empty profile has everything disabled
# This is useful as a base for custom user configuration
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    base:
      enabled: false
    pilot:
      enabled: false
    ingressGateways:
    - name: istio-ingressgateway
      enabled: false

#default yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  hub: gcr.io/istio-testing
  tag: latest

  # You may override parts of meshconfig by uncommenting the following lines.
  meshConfig:
    defaultConfig:
      proxyMetadata: {}
    enablePrometheusMerge: true
    # Opt-out of global http2 upgrades.
    # Destination rule is used to opt-in.
    # h2_upgrade_policy: DO_NOT_UPGRADE

  # Traffic management feature
  components:
    base:
      enabled: true
    pilot:
      enabled: true

    # Istio Gateway feature
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
    egressGateways:
    - name: istio-egressgateway
      enabled: false

    # Istio CNI feature
    cni:
      enabled: false
    
    # Remote and config cluster configuration for an external istiod
    istiodRemote:
      enabled: false

  # Global values passed through to helm global.yaml.
  # Please keep this in sync with manifests/charts/global.yaml
  values:
    defaultRevision: ""
    global:
      istioNamespace: istio-system
      istiod:
        enableAnalysis: false
      logging:
        level: "default:info"
      logAsJson: false
      pilotCertProvider: istiod
      jwtPolicy: third-party-jwt
      proxy:
        image: proxyv2
        clusterDomain: "cluster.local"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 2000m
            memory: 1024Mi
        logLevel: warning
        componentLogLevel: "misc:error"
        privileged: false
        enableCoreDump: false
        statusPort: 15020
        readinessInitialDelaySeconds: 1
        readinessPeriodSeconds: 2
        readinessFailureThreshold: 30
        includeIPRanges: "*"
        excludeIPRanges: ""excludeOutboundPorts:""
        excludeInboundPorts: ""
        autoInject: enabled
        tracer: "zipkin"
      proxy_init:
        image: proxyv2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 10m
            memory: 10Mi
      # Specify image pull policy if default behavior isn't desired.
      # Default behavior: latest images will be Always else IfNotPresent.
      imagePullPolicy: ""
      operatorManageWebhooks: false
      tracer:
        lightstep: {}
        zipkin: {}
        datadog: {}
        stackdriver: {}
      imagePullSecrets: []
      oneNamespace: false
      defaultNodeSelector: {}
      configValidation: true
      multiCluster:
        enabled: false
        clusterName: ""
      omitSidecarInjectorConfigMap: false
      network: ""
      defaultResources:
        requests:
          cpu: 10m
      defaultPodDisruptionBudget:
        enabled: true
      priorityClassName: ""
      useMCP: false
      sds:
        token:
          aud: istio-ca
      sts:
        servicePort: 0
      meshNetworks: {}
      mountMtlsCerts: false
    base:
      enableCRDTemplates: false
      validationURL: ""
    pilot:
      autoscaleEnabled: true
      autoscaleMin: 1
      autoscaleMax: 5
      replicaCount: 1
      image: pilot
      traceSampling: 1.0
      env: {}
      cpu:
        targetAverageUtilization: 80
      nodeSelector: {}
      keepaliveMaxServerConnectionAge: 30m
      enableProtocolSniffingForOutbound: true
      enableProtocolSniffingForInbound: true
      deploymentLabels:
      podLabels: {}
      configMap: true

    telemetry:
      enabled: true
      v2:
        enabled: true
        metadataExchange:
          wasmEnabled: false
        prometheus:
          wasmEnabled: false
          enabled: true
        stackdriver:
          enabled: false
          logging: false
          monitoring: false
          topology: false
          configOverride: {}

    istiodRemote:
      injectionURL: ""
      
    gateways:
      istio-egressgateway:
        env: {}
        autoscaleEnabled: true
        type: ClusterIP
        name: istio-egressgateway
        secretVolumes:
          - name: egressgateway-certs
            secretName: istio-egressgateway-certs
            mountPath: /etc/istio/egressgateway-certs
          - name: egressgateway-ca-certs
            secretName: istio-egressgateway-ca-certs
            mountPath: /etc/istio/egressgateway-ca-certs

      istio-ingressgateway:
        autoscaleEnabled: true
        type: LoadBalancer
        name: istio-ingressgateway
        env: {}
        secretVolumes:
          - name: ingressgateway-certs
            secretName: istio-ingressgateway-certs
            mountPath: /etc/istio/ingressgateway-certs
          - name: ingressgateway-ca-certs
            secretName: istio-ingressgateway-ca-certs
            mountPath: /etc/istio/ingressgateway-ca-certs

#temp1.yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  components:
    base:
      enabled: false
    cni:
      enabled: false
    egressGateways:
    - enabled: false
      name: istio-egressgateway
    ingressGateways:
    - enabled: false
      name: istio-ingressgateway
    istiodRemote:
      enabled: false
    pilot:
      enabled: false
  hub: gcr.io/istio-testing
  meshConfig:
    defaultConfig:
      proxyMetadata: {}
    enablePrometheusMerge: true
  tag: latest
  values:
    base:
      enableCRDTemplates: false
      validationURL: ""defaultRevision:""
    gateways:
      istio-egressgateway:
        autoscaleEnabled: true
        env: {}
        name: istio-egressgateway
        secretVolumes:
        - mountPath: /etc/istio/egressgateway-certs
          name: egressgateway-certs
          secretName: istio-egressgateway-certs
        - mountPath: /etc/istio/egressgateway-ca-certs
          name: egressgateway-ca-certs
          secretName: istio-egressgateway-ca-certs
        type: ClusterIP
      istio-ingressgateway:
        autoscaleEnabled: true
        env: {}
        name: istio-ingressgateway
        secretVolumes:
        - mountPath: /etc/istio/ingressgateway-certs
          name: ingressgateway-certs
          secretName: istio-ingressgateway-certs
        - mountPath: /etc/istio/ingressgateway-ca-certs
          name: ingressgateway-ca-certs
          secretName: istio-ingressgateway-ca-certs
        type: LoadBalancer
    global:
      configValidation: true
      defaultNodeSelector: {}
      defaultPodDisruptionBudget:
        enabled: true
      defaultResources:
        requests:
          cpu: 10m
      imagePullPolicy: ""
      imagePullSecrets: []
      istioNamespace: istio-system
      istiod:
        enableAnalysis: false
      jwtPolicy: third-party-jwt
      logAsJson: false
      logging:
        level: default:info
      meshNetworks: {}
      mountMtlsCerts: false
      multiCluster:
        clusterName: ""
        enabled: false
      network: ""
      omitSidecarInjectorConfigMap: false
      oneNamespace: false
      operatorManageWebhooks: false
      pilotCertProvider: istiod
      priorityClassName: ""
      proxy:
        autoInject: enabled
        clusterDomain: cluster.local
        componentLogLevel: misc:error
        enableCoreDump: false
        excludeIPRanges: ""excludeInboundPorts:""
        excludeOutboundPorts: ""
        image: proxyv2
        includeIPRanges: '*'
        logLevel: warning
        privileged: false
        readinessFailureThreshold: 30
        readinessInitialDelaySeconds: 1
        readinessPeriodSeconds: 2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 128Mi
        statusPort: 15020
        tracer: zipkin
      proxy_init:
        image: proxyv2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 10m
            memory: 10Mi
      sds:
        token:
          aud: istio-ca
      sts:
        servicePort: 0
      tracer:
        datadog: {}
        lightstep: {}
        stackdriver: {}
        zipkin: {}
      useMCP: false
    istiodRemote:
      injectionURL: ""
    pilot:
      autoscaleEnabled: true
      autoscaleMax: 5
      autoscaleMin: 1
      configMap: true
      cpu:
        targetAverageUtilization: 80
      deploymentLabels: null
      enableProtocolSniffingForInbound: true
      enableProtocolSniffingForOutbound: true
      env: {}
      image: pilot
      keepaliveMaxServerConnectionAge: 30m
      nodeSelector: {}
      podLabels: {}
      replicaCount: 1
      traceSampling: 1
    telemetry:
      enabled: true
      v2:
        enabled: true
        metadataExchange:
          wasmEnabled: false
        prometheus:
          enabled: true
          wasmEnabled: false
        stackdriver:
          configOverride: {}
          enabled: false
          logging: false
          monitoring: false
          topology: false

OverlayIOP

OverlayIOP 将 ReadYaml(And)Profile 所失去的 temp0.yaml 笼罩于 temp1.yaml 之上，失去 final.yaml

#empty yaml

# The empty profile has everything disabled
# This is useful as a base for custom user configuration
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    base:
      enabled: false
    pilot:
      enabled: false
    ingressGateways:
    - name: istio-ingressgateway
      enabled: false

# final.yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  creationTimestamp: null
  namespace: istio-system
spec:
  components:
    base:
      enabled: false
    cni:
      enabled: false
    egressGateways:
    - enabled: false
      name: istio-egressgateway
    ingressGateways:
    - enabled: false
      name: istio-ingressgateway
    istiodRemote:
      enabled: false
    pilot:
      enabled: true
  hub: docker.io/istio
  installPackagePath: /istio/manifests
  meshConfig:
    defaultConfig:
      proxyMetadata: {}
    enablePrometheusMerge: true
  profile: empty
  tag: 1.1.5
  values:
    base:
      enableCRDTemplates: false
      validationURL: ""defaultRevision:""
    gateways:
      istio-egressgateway:
        autoscaleEnabled: true
        env: {}
        name: istio-egressgateway
        secretVolumes:
        - mountPath: /etc/istio/egressgateway-certs
          name: egressgateway-certs
          secretName: istio-egressgateway-certs
        - mountPath: /etc/istio/egressgateway-ca-certs
          name: egressgateway-ca-certs
          secretName: istio-egressgateway-ca-certs
        type: ClusterIP
      istio-ingressgateway:
        autoscaleEnabled: true
        env: {}
        name: istio-ingressgateway
        secretVolumes:
        - mountPath: /etc/istio/ingressgateway-certs
          name: ingressgateway-certs
          secretName: istio-ingressgateway-certs
        - mountPath: /etc/istio/ingressgateway-ca-certs
          name: ingressgateway-ca-certs
          secretName: istio-ingressgateway-ca-certs
        type: LoadBalancer
    global:
      configValidation: true
      defaultNodeSelector: {}
      defaultPodDisruptionBudget:
        enabled: true
      defaultResources:
        requests:
          cpu: 10m
      imagePullPolicy: ""
      imagePullSecrets: []
      istioNamespace: istio-system
      istiod:
        enableAnalysis: false
      jwtPolicy: third-party-jwt
      logAsJson: false
      logging:
        level: default:info
      meshNetworks: {}
      mountMtlsCerts: false
      multiCluster:
        clusterName: ""
        enabled: false
      network: ""
      omitSidecarInjectorConfigMap: false
      oneNamespace: false
      operatorManageWebhooks: false
      pilotCertProvider: istiod
      priorityClassName: ""
      proxy:
        autoInject: enabled
        clusterDomain: cluster.local
        componentLogLevel: misc:error
        enableCoreDump: false
        excludeIPRanges: ""excludeInboundPorts:""
        excludeOutboundPorts: ""
        image: proxyv2
        includeIPRanges: '*'
        logLevel: warning
        privileged: false
        readinessFailureThreshold: 30
        readinessInitialDelaySeconds: 1
        readinessPeriodSeconds: 2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 128Mi
        statusPort: 15020
        tracer: zipkin
      proxy_init:
        image: proxyv2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 10m
            memory: 10Mi
      sds:
        token:
          aud: istio-ca
      sts:
        servicePort: 0
      tracer:
        datadog: {}
        lightstep: {}
        stackdriver: {}
        zipkin: {}
      useMCP: false
    istiodRemote:
      injectionURL: ""
    pilot:
      autoscaleEnabled: true
      autoscaleMax: 5
      autoscaleMin: 1
      configMap: true
      cpu:
        targetAverageUtilization: 80
      deploymentLabels: null
      enableProtocolSniffingForInbound: true
      enableProtocolSniffingForOutbound: true
      env: {}
      image: pilot
      keepaliveMaxServerConnectionAge: 30m
      nodeSelector: {}
      podLabels: {}
      replicaCount: 1
      traceSampling: 1
    telemetry:
      enabled: true
      v2:
        enabled: true
        metadataExchange:
          wasmEnabled: false
        prometheus:
          enabled: true
          wasmEnabled: false
        stackdriver:
          configOverride: {}
          enabled: false
          logging: false
          monitoring: false
          topology: false

GenerateConfig

配置局部总结如下，operator 将绝大部分面向用户的配置复杂度都暗藏在了内置 profile 中，其中 base profile 是根底演变项，演变过程为 base profile -> select profile -> user config -> set flag，越往后优先级越高。在具体实际中，内置 profile 能满足大多数需要，若有额定需要则应用 user config 解决，set flag 不便于管理。

RenderCharts

后面提到，operator 要应用 helm 的 chart rendering，因而通过 GenerateConfig 生成的 final.yaml 能够映射到 helm 的 values.yaml。并且，为了便于用户了解与保护 helm chart，istio 管制面的每一个组件都有一个 chart 与之对应，final.yaml 中各个组件的配置最终映射到各个 chart 的 values.yaml 进行渲染。charts 目录如下所示。

processRecursive

每个组件的 chart 渲染胜利后生成所需的 manifest，并利用 k8s 的 api 与 apiserver 进行交互，最初胜利创立或更新 istio 失常运行所需的所有资源。

Manifest

Manifest 命令根本与 Install 命令的流程相似，只是通过 helm 渲染后不须要与 apiserver 交互并创立资源，因而不再赘述。