关于asm:Flagger-on-ASM基于Mixerless-Telemetry实现渐进式灰度发布系列-2-应用级扩缩容

40次阅读

共计 5653 个字符，预计需要花费 15 分钟才能阅读完成。

简介：利用级扩缩容是绝对于运维级而言的。像监控 CPU/ 内存的利用率就属于利用无关的纯运维指标，针对这种指标进行扩缩容的 HPA 配置就是运维级扩缩容。而像申请数量、申请提早、P99 散布等指标就属于利用相干的，或者叫业务感知的监控指标。本篇将介绍 3 种利用级监控指标在 HPA 中的配置，以实现利用级主动扩缩容。
利用级扩缩容是绝对于运维级而言的。像监控 CPU/ 内存的利用率就属于利用无关的纯运维指标，针对这种指标进行扩缩容的 HPA 配置就是运维级扩缩容。而像申请数量、申请提早、P99 散布等指标就属于利用相干的，或者叫业务感知的监控指标。

本篇将介绍 3 种利用级监控指标在 HPA 中的配置，以实现利用级主动扩缩容。

Setup HPA
1 部署 metrics-adapter
执行如下命令部署 kube-metrics-adapter(残缺脚本参见：demo_hpa.sh)。：

helm –kubeconfig “$USER_CONFIG” -n kube-system install asm-custom-metrics \
$KUBE_METRICS_ADAPTER_SRC/deploy/charts/kube-metrics-adapter \
–set prometheus.url=http://prometheus.istio-syste…
执行如下命令验证部署状况：

验证 POD

kubectl –kubeconfig “$USER_CONFIG” get po -n kube-system | grep metrics-adapter

asm-custom-metrics-kube-metrics-adapter-6fb4949988-ht8pv 1/1 Running 0 30s

验证 CRD

kubectl –kubeconfig “$USER_CONFIG” api-versions | grep “autoscaling/v2beta”

autoscaling/v2beta1
autoscaling/v2beta2

验证 CRD

kubectl –kubeconfig “$USER_CONFIG” get –raw “/apis/external.metrics.k8s.io/v1beta1” | jq .

{
“kind”: “APIResourceList”,
“apiVersion”: “v1”,
“groupVersion”: “external.metrics.k8s.io/v1beta1”,
“resources”: []
}
2 部署 loadtester
执行如下命令部署 flagger loadtester：

kubectl –kubeconfig “$USER_CONFIG” apply -f $FLAAGER_SRC/kustomize/tester/deployment.yaml -n test
kubectl –kubeconfig “$USER_CONFIG” apply -f $FLAAGER_SRC/kustomize/tester/service.yaml -n test
3 部署 HPA
3.1 依据利用申请数量扩缩容
首先咱们创立一个感知利用申请数量 (istio_requests_total) 的 HorizontalPodAutoscaler 配置：

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: podinfo-total
namespace: test
annotations:

metric-config.external.prometheus-query.prometheus/processed-requests-per-second: |
  sum(rate(istio_requests_total{destination_workload_namespace="test",reporter="destination"}[1m]))

spec:
maxReplicas: 5
minReplicas: 1
scaleTargetRef:

apiVersion: apps/v1
kind: Deployment
name: podinfo

metrics:

- type: External
  external:
    metric:
      name: prometheus-query
      selector:
        matchLabels:
          query-name: processed-requests-per-second
    target:
      type: AverageValue
      averageValue: "10"

执行如下命令部署这个 HPA 配置：

kubectl –kubeconfig “$USER_CONFIG” apply -f resources_hpa/requests_total_hpa.yaml
执行如下命令校验：

kubectl –kubeconfig “$USER_CONFIG” get –raw “/apis/external.metrics.k8s.io/v1beta1” | jq .
后果如下：

{
“kind”: “APIResourceList”,
“apiVersion”: “v1”,
“groupVersion”: “external.metrics.k8s.io/v1beta1”,
“resources”: [

{
  "name": "prometheus-query",
  "singularName": "","namespaced": true,"kind":"ExternalMetricValueList","verbs": ["get"]
}

]
}
相似地，咱们能够应用其余维度的利用级监控指标配置 HPA。举例如下，不再冗述。

3.2 依据均匀提早扩缩容
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: podinfo-latency-avg
namespace: test
annotations:

metric-config.external.prometheus-query.prometheus/latency-average: |
  sum(rate(istio_request_duration_milliseconds_sum{destination_workload_namespace="test",reporter="destination"}[1m]))
  /sum(rate(istio_request_duration_milliseconds_count{destination_workload_namespace="test",reporter="destination"}[1m]))

spec:
maxReplicas: 5
minReplicas: 1
scaleTargetRef:

apiVersion: apps/v1
kind: Deployment
name: podinfo

metrics:

- type: External
  external:
    metric:
      name: prometheus-query
      selector:
        matchLabels:
          query-name: latency-average
    target:
      type: AverageValue
      averageValue: "0.005"

3.3 依据 P95 散布扩缩容
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: podinfo-p95
namespace: test
annotations:

metric-config.external.prometheus-query.prometheus/p95-latency: |
  histogram_quantile(0.95,sum(irate(istio_request_duration_milliseconds_bucket{destination_workload_namespace="test",destination_canonical_service="podinfo"}[5m]))by (le))

spec:
maxReplicas: 5
minReplicas: 1
scaleTargetRef:

apiVersion: apps/v1
kind: Deployment
name: podinfo

metrics:

- type: External
  external:
    metric:
      name: prometheus-query
      selector:
        matchLabels:
          query-name: p95-latency
    target:
      type: AverageValue
      averageValue: "4"

验证 HPA
1 生成负载
执行如下命令产生试验流量，以验证 HPA 配置主动扩容失效。

alias k=”kubectl –kubeconfig $USER_CONFIG”
loadtester=$(k -n test get pod -l “app=flagger-loadtester” -o jsonpath='{.items..metadata.name}’)
k -n test exec -it ${loadtester} -c loadtester — hey -z 5m -c 2 -q 10 http://podinfo:9898
这里运行了一个继续 5 分钟、QPS=10、并发数为 2 的申请。

hey 命令具体参考如下：

Usage: hey [options…] <url>

Options:
-n Number of requests to run. Default is 200.
-c Number of workers to run concurrently. Total number of requests cannot

  be smaller than the concurrency level. Default is 50.

-q Rate limit, in queries per second (QPS) per worker. Default is no rate limit.
-z Duration of application to send requests. When duration is reached,

  application stops and exits. If duration is specified, n is ignored.
  Examples: -z 10s -z 3m.

-o Output type. If none provided, a summary is printed.

  "csv" is the only supported alternative. Dumps the response
  metrics in comma-separated values format.

-m HTTP method, one of GET, POST, PUT, DELETE, HEAD, OPTIONS.
-H Custom HTTP header. You can specify as many as needed by repeating the flag.

  For example, -H "Accept: text/html" -H "Content-Type: application/xml" .

-t Timeout for each request in seconds. Default is 20, use 0 for infinite.
-A HTTP Accept header.
-d HTTP request body.
-D HTTP request body from file. For example, /home/user/file.txt or ./file.txt.
-T Content-type, defaults to “text/html”.
-a Basic authentication, username:password.
-x HTTP Proxy address as host:port.
-h2 Enable HTTP/2.

-host HTTP Host header.

-disable-compression Disable compression.
-disable-keepalive Disable keep-alive, prevents re-use of TCP

                    connections between different HTTP requests.

-disable-redirects Disable following of HTTP redirects
-cpus Number of used cpu cores.

                    (default for current machine is 4 cores)

2 主动扩容
执行如下命令察看扩容状况：

watch kubectl –kubeconfig $USER_CONFIG -n test get hpa/podinfo-total
后果如下：

Every 2.0s: kubectl –kubeconfig /Users/han/shop_config/ack_zjk -n test get hpa/podinfo East6C16G: Tue Jan 26 18:01:30 2021

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
podinfo Deployment/podinfo 10056m/10 (avg) 1 5 2 4m45s
另外两个 HPA 相似，命令如下：

kubectl –kubeconfig $USER_CONFIG -n test get hpa

watch kubectl –kubeconfig $USER_CONFIG -n test get hpa/podinfo-latency-avg
watch kubectl –kubeconfig $USER_CONFIG -n test get hpa/podinfo-p95
3 监控指标
同时，咱们能够实时在 Prometheus 中查看相干的利用级监控指标的实时数据。示意如下：


原文链接
本文为阿里云原创内容，未经容许不得转载。

正文完
asm
发表至： asm
2021-04-20
 0
关于asm:Flagger-on-ASM基于Mixerless-Telemetry实现渐进式灰度发布系列-3-渐进式灰度发布
关于asm:汇编语言中的寻址方式
关于asm:汇编格式-ATT-与-Intel
关于asm:直播预告阿里云服务网格-ASM-产品易用性改善实践与思考