乐趣区

关于阿里云:应用场景系列之1流量管理下的熔断场景

jiezi

在采纳 Istio 服务网格技术的很多客户案例中, 熔断是其中一个十分广泛的流量治理场景。在启用网格技术之前, 在一些应用 Resilience4j 的 Java 服务中客户曾经应用了熔断性能, 但相比之下, Istio 可能在网络级别反对启用熔断能力，无需集成到每个服务的利用程序代码中。

深刻了解熔断器在不同场景下的行为, 是将其利用到线上环境之前的要害前提。

介绍

启用熔断性能，须要创立一个指标规定来为指标服务配置熔断。其中, connectionPool 下定义了与熔断性能相干的参数, 相干配置参数为：

  • tcp.maxConnections: 到指标主机的最大 HTTP1 /TCP 连接数。默认值为 2³²-1。
  • http.http1MaxPendingRequests：期待就绪的连接池连贯时，最多能够排队的申请数量。默认值为 1024。
  • http.http2MaxRequests：对后端服务的最大沉闷申请数。默认值为 1024。

这些参数在一个简略的场景中, 如一个客户端和一个指标服务实例（在 Kubernetes 环境中，一个实例相当于一个 pod）的状况下是清晰的。然而, 在生产环境中，比拟可能呈现的场景是：

  • 一个客户端实例和多个指标服务实例
  • 多个客户端实例和单个指标服务实例
  • 客户端和指标服务的多个实例

咱们创立了两个 python 脚本——一个用于示意指标服务，另一个用于调用服务的客户端。服务器脚本是一个简略的 Flask 应用程序，它公开一个休眠 5 秒的 API 服务端点，而后返回一个“hello world!”字符串。示例代码如下所示：

#! /usr/bin/env python3
from flask import Flask
import time

app = Flask(__name__)

@app.route('/hello')
def get():
    time.sleep(5)
    return 'hello world!'

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port='9080', threaded = True)

客户端脚本以 10 个为一组调用服务器端点，即 10 个并行申请，而后在发送下一批 10 个申请之前休眠一段时间。它在有限循环中执行此操作。为了确保当咱们运行客户端的多个 pod 时，它们都同时发送批处理，咱们应用零碎工夫（每分钟的第 0、20 和 40 秒）发送批处理。

#! /usr/bin/env python3
import requests
import time
import sys
from datetime import datetime
import _thread

def timedisplay(t):
  return t.strftime("%H:%M:%S")

def get(url):
  try:
    stime = datetime.now()
    start = time.time()
    response = requests.get(url)
    etime = datetime.now()
    end = time.time()
    elapsed = end-start
    sys.stderr.write("Status:" + str(response.status_code) + ", Start:" + timedisplay(stime) + ", End:" + timedisplay(etime) + ", Elapsed Time:" + str(elapsed)+"\n")
    sys.stdout.flush()
  except Exception as myexception:
    sys.stderr.write("Exception:" + str(myexception)+"\n")
    sys.stdout.flush()

time.sleep(30)

while True:
  sc = int(datetime.now().strftime('%S'))
  time_range = [0, 20, 40]

  if sc not in time_range:
    time.sleep(1)
    continue

  sys.stderr.write("\n----------Info----------\n")
  sys.stdout.flush()

  # Send 10 requests in parallel
  for i in range(10):
    _thread.start_new_thread(get, ("http://circuit-breaker-sample-server:9080/hello",))

  time.sleep(2)

部署示例

利用应用如下 YAML 部署示例利用：

##################################################################################################
#  circuit-breaker-sample-server services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: circuit-breaker-sample-server
  labels:
    app: circuit-breaker-sample-server
    service: circuit-breaker-sample-server
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: circuit-breaker-sample-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: circuit-breaker-sample-server
  labels:
    app: circuit-breaker-sample-server
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: circuit-breaker-sample-server
      version: v1
  template:
    metadata:
      labels:
        app: circuit-breaker-sample-server
        version: v1
    spec:
      containers:
      - name: circuit-breaker-sample-server
        image: registry.cn-hangzhou.aliyuncs.com/acs/istio-samples:circuit-breaker-sample-server.v1
        imagePullPolicy: Always
        ports:
        - containerPort: 9080
---
##################################################################################################
#  circuit-breaker-sample-client services
##################################################################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: circuit-breaker-sample-client
  labels:
    app: circuit-breaker-sample-client
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: circuit-breaker-sample-client
      version: v1
  template:
    metadata:
      labels:
        app: circuit-breaker-sample-client
        version: v1
    spec:
      containers:
      - name: circuit-breaker-sample-client
        image: registry.cn-hangzhou.aliyuncs.com/acs/istio-samples:circuit-breaker-sample-client.v1
        imagePullPolicy: Always

启动之后, 能够看到为客户端和服务器端别离启动了对应的 pod, 相似如下：

> kubectl get po |grep circuit       
circuit-breaker-sample-client-d4f64d66d-fwrh4   2/2     Running   0             1m22s
circuit-breaker-sample-server-6d6ddb4b-gcthv    2/2     Running   0             1m22s

在未定义指标规定限度的状况下, 服务器端能够满足解决并发的 10 个客户端申请, 因而在服务器端的响应后果始终是 200。客户端侧的日志该当相似如下：

----------Info----------
Status: 200, Start: 02:39:20, End: 02:39:25, Elapsed Time: 5.016539812088013
Status: 200, Start: 02:39:20, End: 02:39:25, Elapsed Time: 5.012614488601685
Status: 200, Start: 02:39:20, End: 02:39:25, Elapsed Time: 5.015984535217285
Status: 200, Start: 02:39:20, End: 02:39:25, Elapsed Time: 5.015599012374878
Status: 200, Start: 02:39:20, End: 02:39:25, Elapsed Time: 5.012874364852905
Status: 200, Start: 02:39:20, End: 02:39:25, Elapsed Time: 5.018714904785156
Status: 200, Start: 02:39:20, End: 02:39:25, Elapsed Time: 5.010422468185425
Status: 200, Start: 02:39:20, End: 02:39:25, Elapsed Time: 5.012431621551514
Status: 200, Start: 02:39:20, End: 02:39:25, Elapsed Time: 5.011001348495483
Status: 200, Start: 02:39:20, End: 02:39:25, Elapsed Time: 5.01432466506958

启用熔断规定

通过服务网格技术启用熔断规定, 只须要针对指标服务定义对应的指标规定 DestinationRule 即可。创立并利用以下指标规定：

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: circuit-breaker-sample-server
spec:
  host: circuit-breaker-sample-server
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 5

它将与指标服务的 TCP 连接数限度为 5。让咱们看看它在不同场景中的工作形式。

场景 #1：一个客户端实例和一个指标服务实例

在这种状况下，客户端和指标服务都只有一个 pod。当咱们启动客户端 pod 并监控日志时（倡议重启下客户端以取得更直观的统计后果），咱们会看到相似以下内容：

----------Info----------
Status: 200, Start: 02:49:40, End: 02:49:45, Elapsed Time: 5.0167787075042725
Status: 200, Start: 02:49:40, End: 02:49:45, Elapsed Time: 5.011920690536499
Status: 200, Start: 02:49:40, End: 02:49:45, Elapsed Time: 5.017078161239624
Status: 200, Start: 02:49:40, End: 02:49:45, Elapsed Time: 5.018405437469482
Status: 200, Start: 02:49:40, End: 02:49:45, Elapsed Time: 5.018689393997192
Status: 200, Start: 02:49:40, End: 02:49:50, Elapsed Time: 10.018936395645142
Status: 200, Start: 02:49:40, End: 02:49:50, Elapsed Time: 10.016417503356934
Status: 200, Start: 02:49:40, End: 02:49:50, Elapsed Time: 10.019930601119995
Status: 200, Start: 02:49:40, End: 02:49:50, Elapsed Time: 10.022735834121704
Status: 200, Start: 02:49:40, End: 02:49:55, Elapsed Time: 15.02303147315979

能够看到所有申请都胜利。然而，每批中只有 5 个申请的响应工夫约为 5 秒，其余的要慢得多（大部分为 10 秒之多）。这意味着仅应用 tcp.maxConnections 会导致过多的申请排队，期待连贯开释。如后面所述，默认状况下，能够排队的申请数为 2³²-1。要真正具备熔断（即疾速失败）行为，咱们还须要设置 http.http1MaxPendingRequests 限度能够排队的申请数量。它的默认值为 1024。乏味的是，如果咱们将它的值设置为 0，它就会回落到默认值。所以咱们必须至多将它设置为 1。让咱们更新指标规定以仅容许 1 个待处理申请：

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: circuit-breaker-sample-server
spec:
  host: circuit-breaker-sample-server
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 5
      http:
        http1MaxPendingRequests: 1

并重启客户端 pod（记得肯定须要重启客户端, 否则统计后果会呈现偏差）, 并持续察看日志, 相似如下：

----------Info----------
Status: 503, Start: 02:56:40, End: 02:56:40, Elapsed Time: 0.005339622497558594
Status: 503, Start: 02:56:40, End: 02:56:40, Elapsed Time: 0.007254838943481445
Status: 503, Start: 02:56:40, End: 02:56:40, Elapsed Time: 0.0044133663177490234
Status: 503, Start: 02:56:40, End: 02:56:40, Elapsed Time: 0.008964776992797852
Status: 200, Start: 02:56:40, End: 02:56:45, Elapsed Time: 5.018309116363525
Status: 200, Start: 02:56:40, End: 02:56:45, Elapsed Time: 5.017424821853638
Status: 200, Start: 02:56:40, End: 02:56:45, Elapsed Time: 5.019804954528809
Status: 200, Start: 02:56:40, End: 02:56:45, Elapsed Time: 5.01643180847168
Status: 200, Start: 02:56:40, End: 02:56:45, Elapsed Time: 5.025975227355957
Status: 200, Start: 02:56:40, End: 02:56:50, Elapsed Time: 10.01716136932373

能够看到, 4 个申请立刻被限度，5 个申请发送到指标服务，1 个申请排队。这是预期的行为。能够看到客户端 Istio 代理与指标服务中的 pod 建设的沉闷连接数为 5：

kubectl exec $(kubectl get pod --selector app=circuit-breaker-sample-client --output jsonpath='{.items[0].metadata.name}') -c istio-proxy -- curl -X POST http://localhost:15000/clusters | grep circuit-breaker-sample-server | grep cx_active

outbound|9080||circuit-breaker-sample-server.default.svc.cluster.local::172.20.192.124:9080::cx_active::5

场景 #2：一个客户端实例和多个指标服务实例

当初让咱们在有一个客户端实例和多个指标服务实例 pod 的场景中运行测试。

首先，咱们须要将指标服务部署扩大到多个正本（比方 3 个）：

kubectl scale deployment/circuit-breaker-sample-server  --replicas=3

在这里要验证的两个场景别离是：

  • 连贯限度利用于 pod 级别：指标服务的每个 pod 最多 5 个连贯;
  • 或者它是否利用于服务级别：无论指标服务中的 pod 数量如何，总共最多 5 个连贯;

在 (1) 中，咱们应该看不到限度或排队，因为容许的最大连接数为 15（3 个 pod，每个 pod 5 个连贯）。因为咱们一次只发送 10 个申请，所有申请都应该胜利并在大概 5 秒内返回。

在 (2) 中，咱们应该看到与之前场景 #1 大致相同的行为。

让咱们再次启动客户端 pod 并监控日志：

----------Info----------
Status: 503, Start: 03:06:20, End: 03:06:20, Elapsed Time: 0.011791706085205078
Status: 503, Start: 03:06:20, End: 03:06:20, Elapsed Time: 0.0032286643981933594
Status: 503, Start: 03:06:20, End: 03:06:20, Elapsed Time: 0.012153387069702148
Status: 503, Start: 03:06:20, End: 03:06:20, Elapsed Time: 0.011871814727783203
Status: 200, Start: 03:06:20, End: 03:06:25, Elapsed Time: 5.012892484664917
Status: 200, Start: 03:06:20, End: 03:06:25, Elapsed Time: 5.013102769851685
Status: 200, Start: 03:06:20, End: 03:06:25, Elapsed Time: 5.016939163208008
Status: 200, Start: 03:06:20, End: 03:06:25, Elapsed Time: 5.014261484146118
Status: 200, Start: 03:06:20, End: 03:06:25, Elapsed Time: 5.01246190071106
Status: 200, Start: 03:06:20, End: 03:06:30, Elapsed Time: 10.021712064743042

咱们依然看到相似的限度和排队，这意味着减少指标服务的实例数量不会减少客户端的限度。因而, 咱们推导出该限度实用于服务级别。

运行一段时间之后, 能够看到客户端 Istio 代理与指标服务中的每个 pod 建设的连接数：

outbound|9080||circuit-breaker-sample-server.default.svc.cluster.local::172.20.192.124:9080::cx_active::2
outbound|9080||circuit-breaker-sample-server.default.svc.cluster.local::172.20.192.158:9080::cx_active::2
outbound|9080||circuit-breaker-sample-server.default.svc.cluster.local::172.20.192.26:9080::cx_active::2

客户端代理与指标服务中的每个 pod 有 2 个流动连贯, 不是 5，而是 6。正如 Envoy 和 Istio 文档中所提到的，代理在连贯数量方面容许一些回旋余地。

场景 #3：多个客户端实例和一个指标服务实例

在这种状况下，咱们有多个客户端实例 pod，而指标服务只有一个 pod。

相应地缩放正本：

kubectl scale deployment/circuit-breaker-sample-server --replicas=1 
kubectl scale deployment/circuit-breaker-sample-client --replicas=3

因为所有 Istio 代理都基于本地信息独立运行，无需互相协调，因而对这个测试的冀望是每个客户端 pod 都会体现出场景 #1 的行为，即每个 pod 将有 5 个申请被立刻发送到指标服务，1 个申请正在排队并受到限制。

让咱们看一下日志，看看理论产生了什么：Client 1

----------Info----------
Status: 503, Start: 03:10:40, End: 03:10:40, Elapsed Time: 0.008828878402709961
Status: 503, Start: 03:10:40, End: 03:10:40, Elapsed Time: 0.010806798934936523
Status: 503, Start: 03:10:40, End: 03:10:40, Elapsed Time: 0.012855291366577148
Status: 503, Start: 03:10:40, End: 03:10:40, Elapsed Time: 0.004465818405151367
Status: 503, Start: 03:10:40, End: 03:10:40, Elapsed Time: 0.007823944091796875
Status: 503, Start: 03:10:40, End: 03:10:40, Elapsed Time: 0.06221342086791992
Status: 503, Start: 03:10:40, End: 03:10:40, Elapsed Time: 0.06922149658203125
Status: 503, Start: 03:10:40, End: 03:10:40, Elapsed Time: 0.06859922409057617
Status: 200, Start: 03:10:40, End: 03:10:45, Elapsed Time: 5.015282392501831
Status: 200, Start: 03:10:40, End: 03:10:50, Elapsed Time: 9.378434181213379

Client 2

----------Info----------
Status: 503, Start: 03:11:00, End: 03:11:00, Elapsed Time: 0.007795810699462891
Status: 503, Start: 03:11:00, End: 03:11:00, Elapsed Time: 0.00595545768737793
Status: 503, Start: 03:11:00, End: 03:11:00, Elapsed Time: 0.013380765914916992
Status: 503, Start: 03:11:00, End: 03:11:00, Elapsed Time: 0.004278898239135742
Status: 503, Start: 03:11:00, End: 03:11:00, Elapsed Time: 0.010999202728271484
Status: 200, Start: 03:11:00, End: 03:11:05, Elapsed Time: 5.015426874160767
Status: 200, Start: 03:11:00, End: 03:11:05, Elapsed Time: 5.0184690952301025
Status: 200, Start: 03:11:00, End: 03:11:05, Elapsed Time: 5.019806146621704
Status: 200, Start: 03:11:00, End: 03:11:05, Elapsed Time: 5.0175628662109375
Status: 200, Start: 03:11:00, End: 03:11:05, Elapsed Time: 5.031521558761597

Client 3

----------Info----------
Status: 503, Start: 03:13:20, End: 03:13:20, Elapsed Time: 0.012019157409667969
Status: 503, Start: 03:13:20, End: 03:13:20, Elapsed Time: 0.012546539306640625
Status: 503, Start: 03:13:20, End: 03:13:20, Elapsed Time: 0.013760805130004883
Status: 503, Start: 03:13:20, End: 03:13:20, Elapsed Time: 0.014089822769165039
Status: 503, Start: 03:13:20, End: 03:13:20, Elapsed Time: 0.014792442321777344
Status: 503, Start: 03:13:20, End: 03:13:20, Elapsed Time: 0.015463829040527344
Status: 503, Start: 03:13:20, End: 03:13:20, Elapsed Time: 0.01661539077758789
Status: 200, Start: 03:13:20, End: 03:13:20, Elapsed Time: 0.02904224395751953
Status: 200, Start: 03:13:20, End: 03:13:20, Elapsed Time: 0.03912043571472168
Status: 200, Start: 03:13:20, End: 03:13:20, Elapsed Time: 0.06436014175415039

结果显示, 每个客户端上的 503 数量减少了。零碎仅容许来自所有三个客户端实例 pod 的 5 个并发申请。查看客户端代理日志，心愿能失去一些线索，并察看到两种不同类型的日志，用于被限度的申请 (503)。其中, 留神到 RESPONSE_FLAGS 包含了两个值：UO 和 URX。

  • UO：上游溢出（断路）
  • URX: 申请被回绝，因为已达到上游重试限度 (HTTP)或最大连贯尝试次数 (TCP)。
{"authority":"circuit-breaker-sample-server:9080","bytes_received":"0","bytes_sent":"81","downstream_local_address":"192.168.142.207:9080","downstream_remote_address":"172.20.192.31:44610","duration":"0","istio_policy_status":"-","method":"GET","path":"/hello","protocol":"HTTP/1.1","request_id":"d9d87600-cd01-421f-8a6f-dc0ee0ac8ccd","requested_server_name":"-","response_code":"503","response_flags":"UO","route_name":"default","start_time":"2023-02-28T03:14:00.095Z","trace_id":"-","upstream_cluster":"outbound|9080||circuit-breaker-sample-server.default.svc.cluster.local","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"python-requests/2.21.0","x_forwarded_for":"-"}

{"authority":"circuit-breaker-sample-server:9080","bytes_received":"0","bytes_sent":"81","downstream_local_address":"192.168.142.207:9080","downstream_remote_address":"172.20.192.31:43294","duration":"58","istio_policy_status":"-","method":"GET","path":"/hello","protocol":"HTTP/1.1","request_id":"931d080a-3413-4e35-91f4-0c906e7ee565","requested_server_name":"-","response_code":"503","response_flags":"URX","route_name":"default","start_time":"2023-02-28T03:12:20.995Z","trace_id":"-","upstream_cluster":"outbound|9080||circuit-breaker-sample-server.default.svc.cluster.local","upstream_host":"172.20.192.84:9080","upstream_local_address":"172.20.192.31:58742","upstream_service_time":"57","upstream_transport_failure_reason":"-","user_agent":"python-requests/2.21.0","x_forwarded_for":"-"}

带有 UO 标记的申请由客户端代理在本地进行限度。带有 URX 标记的申请被指标服务代理回绝。日志中其余字段的值（例如 DURATION、UPSTREAM_HOST 和 UPSTREAM_CLUSTER）也证实了这一点。

为了进一步验证，咱们还要查看指标服务端的代理日志：

{"authority":"circuit-breaker-sample-server:9080","bytes_received":"0","bytes_sent":"81","downstream_local_address":"172.20.192.84:9080","downstream_remote_address":"172.20.192.31:59510","duration":"0","istio_policy_status":"-","method":"GET","path":"/hello","protocol":"HTTP/1.1","request_id":"7684cbb0-8f1c-44bf-b591-40c3deff6b0b","requested_server_name":"outbound_.9080_._.circuit-breaker-sample-server.default.svc.cluster.local","response_code":"503","response_flags":"UO","route_name":"default","start_time":"2023-02-28T03:14:00.095Z","trace_id":"-","upstream_cluster":"inbound|9080||","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"python-requests/2.21.0","x_forwarded_for":"-"}
{"authority":"circuit-breaker-sample-server:9080","bytes_received":"0","bytes_sent":"81","downstream_local_address":"172.20.192.84:9080","downstream_remote_address":"172.20.192.31:58218","duration":"0","istio_policy_status":"-","method":"GET","path":"/hello","protocol":"HTTP/1.1","request_id":"2aa351fa-349d-4283-a5ea-dc74ecbdff8c","requested_server_name":"outbound_.9080_._.circuit-breaker-sample-server.default.svc.cluster.local","response_code":"503","response_flags":"UO","route_name":"default","start_time":"2023-02-28T03:12:20.996Z","trace_id":"-","upstream_cluster":"inbound|9080||","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"python-requests/2.21.0","x_forwarded_for":"-"}

正如预期的那样，这里有 503 响应码，这也是导致客户端代理上有 “response_code”:”503″ 以及 ”response_flags”:”URX”。

总而言之, 客户端代理依据它们的连贯限度（每个 pod 最多 5 个连贯）发送申请——排队或限度（应用 UO 响应标记）多余的申请。所有三个客户端代理在批处理开始时最多能够发送 15 个并发申请。然而，其中只有 5 个胜利，因为指标服务代理也在应用雷同的配置（最多 5 个连贯）限度。指标服务代理将仅承受 5 个申请并限度其余申请，这些申请在客户端代理日志中带有 URX 响应标记。

上述所产生状况的直观形容相似如下：

场景 #4：多个客户端实例和多个指标服务实例

最初一个也可能是最常见的场景，咱们有多个客户端实例 pod 和多个指标服务实例 pod。当咱们减少指标服务正本时，咱们应该会看到申请的成功率整体减少，因为每个指标代理能够容许 5 个并发申请。

  • 如果咱们将正本数减少到 2，咱们应该会看到所有 3 个客户端代理在一个批次中生成的 30 个申请中有 10 个申请胜利。咱们依然会察看到客户端和指标服务代理上的限度。
  • 如果咱们将正本减少到 3，咱们应该看到 15 个胜利的申请。
  • 如果咱们将数量减少到 4，咱们应该依然只能看到 15 个胜利的申请。为什么？这是因为无论指标服务有多少个正本，客户端代理上的限度都实用于整个指标服务。因而，无论有多少个正本，每个客户端代理最多能够向指标服务收回 5 个并发申请。

总结

在客户端：

  • 每个客户端代理独立利用该限度。如果限度为 100，则每个客户端代理在利用本地限度之前能够有 100 个未实现的申请。如果有 N 个客户端调用指标服务，则总共最多能够有 100*N 个未实现的申请。
  • 客户端代理的限度是针对整个指标服务，而不是针对指标服务的单个正本。即便指标服务有 200 个流动 pod, 限流依然会是 100。

在指标服务端：

每个指标服务代理也实用该限度。如果该服务有 50 个流动的 pod，则在利用限流并返回 503 之前，每个 pod 最多能够有 100 个来自客户端代理的未实现申请。

原文链接

本文为阿里云原创内容，未经容许不得转载。

退出移动版