关于kubernetes:HPA使用自定义指标进行扩缩容

kubernetes 引入 HPA 进行扩缩容，HPA 能够依据指标(metrics)，主动对 workload 进行弹性的扩缩容。

HPA 资源在 autocaling 这个 group 上面，HPA 分为 v1 和 v2 版本：

• HPA v1 仅反对基于 CPU 指标的伸缩；
• HPA v2beta1 引入了基于 Memory 和自定义指标的伸缩；
• HPA v2beta2 引入了自定义策略 (behavior) 的伸缩；
• HPA v2 将 v2beta2 变成 GA 性能；

# kubectl api-versions|grep auto
autoscaling/v1
autoscaling/v2beta1
autoscaling/v2beta2
autoscaling/v2

一.HPA 的指标类型

• Resouce: 提供 Pod 级别的 CPU/Memory 使用量；

  • 通过 metrics-server 提供指标，调用 metrics.k8s.io；

• ContainerResource: 提供 Container 级别的 CPU/Memory 使用量；

  • 通过 metrics-server 提供指标，调用 metrics.k8s.io；

• Object: 提供 k8s 集群内任意资源的相干指标；

  • 通过 prometheus-adapter 提供指标，调用 custom.metrics.k8s.io；

• Pods: 提供 k8s 集群内任意资源的相干指标；

  • 通过 prometheus-adapter 提供指标，调用 custom.metrics.k8s.io；
• External: 提供 k8s 集群外的指标数据；

二.HPA 自定义指标的伸缩

1. 部署 prometheus-adapter

自定义指标由 prometheus-adapter 查问 prometheus 失去，所以须要部署 prometheus 和 prometheus-adatper 组件。

这里应用 helm chart 的形式部署：

• prometheus: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
• prometheus-adatper: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-adapter

部署实现当前：

# kubectl get pod -n monitoring
NAME                                  READY   STATUS    RESTARTS   AGE
alertmanager-main-0                   2/2     Running   0          4d20h
kube-state-metrics-796659d8b6-vr5w8   3/3     Running   0          4d20h
node-exporter-rn9nj                   2/2     Running   0          4d20h
prometheus-adapter-5f4df978c8-4fxjq   1/1     Running   0          46s
prometheus-k8s-0                      3/3     Running   1          4d20h
prometheus-operator-7bc7bd7d7-jp754   2/2     Running   0          4d20h

2. 创立 deploy 和 service

deploy: 初始为 1 个正本

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - image: luxas/autoscale-demo:v0.1.2
        name: metrics-provider
        ports:
        - name: http
          containerPort: 8080

service:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: sample-app
  name: sample-app
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: sample-app
  type: ClusterIP

该 sample-app 仅裸露了一个指标：http_requests_total:

# curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP}')/metrics
# HELP http_requests_total The amount of requests served by the server in total
# TYPE http_requests_total counter
http_requests_total 6179

3. 创立 serviceMonitor

创立 serviceMonitor，以便让 prometheus 发现 sample-app，拉到它的 metrics:

kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  selector:
    matchLabels:
      app: sample-app
  endpoints:
  - port: http

4. 创立 prometheusRule 自定义指标

通过 prometheusRule 创立自定义指标：

• 创立了自定义指标：sample_app_http_requests_rate = rate(http_requests_total[5m])

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: sample-app-rules
  namespace: monitoring
spec:
  groups:
    - name: sample-app.expr.rules
      rules:
        - expr: |
            rate(http_requests_total[5m])
          record: sample_app_http_requests_rate
          labels:
            namespace: default
            service: sample-app

能够通过 kubectl 查问自定义指标：

# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/sample_app_http_requests_rate"|python -m json.tool
{
    "apiVersion": "custom.metrics.k8s.io/v1beta1",
    "items": [
        {
            "describedObject": {
                "apiVersion": "/v1",
                "kind": "Pod",
                "name": "sample-app-7cfb596f98-jmswq",
                "namespace": "default"
            },
            "metricName": "sample_app_http_requests_rate",
            "selector": null,
            "value": "33m"
        }
    ],
    "kind": "MetricValueList",
    "metadata": {"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/sample_app_http_requests_rate"}
}

5. 创立 HPA

应用 sample_app_http_requests_rate 指标，创立 HPA：

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: sample-app
spec:
  scaleTargetRef:
    # point the HPA at the sample application
    # you created above
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  # autoscale between 1 and 10 replicas
  minReplicas: 1
  maxReplicas: 10
  metrics:
  # use a "Pods" metric, which takes the average of the
  # given metric across all pods controlled by the autoscaling target
  - type: Pods
    pods:
      # use the metric that you used above: pods/sample_app_http_requests_rate
      metricName: sample_app_http_requests_rate
      # target 500 milli-requests per second,
      # which is 1 request every two seconds
      targetAverageValue: 500m

6. 加大流量察看 HPA

通过 curl 命令，加大流量：

# while true; do curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP}')/metrics; sleep 0.1; done

期待一会后，察看 hpa 对象，能够看到 deploy 已扩容为 5 个正本：

# kubectl get hpa sample-app
NAME         REFERENCE               TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
sample-app   Deployment/sample-app   697m/500m   1         10        5          45h

再次通过 kubectl 查问自定义指标：

# kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/sample_app_http_requests_rate?selector=app%3Dsample-app | python -m json.tool
{
    "apiVersion": "custom.metrics.k8s.io/v1beta1",
    "items": [
        {
            "describedObject": {
                "apiVersion": "/v1",
                "kind": "Pod",
                "name": "sample-app-7cfb596f98-9vkrb",
                "namespace": "default"
            },
            "metricName": "sample_app_http_requests_rate",
            "selector": null,
            "value": "263m"
        },
        {
            "describedObject": {
                "apiVersion": "/v1",
                "kind": "Pod",
                "name": "sample-app-7cfb596f98-f4dqr",
                "namespace": "default"
            },
            "metricName": "sample_app_http_requests_rate",
            "selector": null,
            "value": "553m"
        },
        {
            "describedObject": {
                "apiVersion": "/v1",
                "kind": "Pod",
                "name": "sample-app-7cfb596f98-jmswq",
                "namespace": "default"
            },
            "metricName": "sample_app_http_requests_rate",
            "selector": null,
            "value": "1740m"
        },
        {
            "describedObject": {
                "apiVersion": "/v1",
                "kind": "Pod",
                "name": "sample-app-7cfb596f98-qg24g",
                "namespace": "default"
            },
            "metricName": "sample_app_http_requests_rate",
            "selector": null,
            "value": "234m"
        }
    ],
    "kind": "MetricValueList",
    "metadata": {"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/sample_app_http_requests_rate"}
}

参考：

1.github-doc:https://github.com/arielb135/HPA-with-prometheus-and-RabbitMQ/blob/master/README.md
2.HPA 指标类型：https://mp.weixin.qq.com/s/ul2G1WFPmHo5QR0xI143jg