乐趣区

关于istio:逆向工程与云原生现场分析-Part2-eBPF-跟踪-IstioEnvoy-之启动监听与线程负载均衡

承上

在上一篇 逆向工程与云原生现场剖析 Part1 —— eBPF 跟踪 Istio/Envoy 之学步 中,介绍了如何入门 bpftrace 跟踪 Envoy。这次咱们来次较深度的历险。trace 察看一下 envoy 的启动、worker 线程启动与初始化、socket 监听。

预报

开始前先做个预报,逆向工程与云原生现场剖析 系列(将)包含:

  • Part 1: eBPF 跟踪 Istio/Envoy 之学步
  • Part 2: eBPF 跟踪 Istio/Envoy 之启动、监听与线程负载平衡
  • Part 3: eBPF 跟踪 Istio/Envoy 之 downstream L3/4 连贯 accept、TLS 握手 与 filter_chain 的抉择
  • Part 4: eBPF 跟踪 Istio/Envoy 之 http filter
  • Part 5: eBPF 跟踪 Istio/Envoy 之 http router
  • Part 6: eBPF 跟踪 Istio/Envoy 之 cluster、connection pool 与 outbound load balance

在系列中,我将演示如何让 bpftrace “ 读 ” 懂运行期的由 C++ 11 编写成的 envoy 过程中的对象数据。我花了一个月的工夫去补上几个技术债:

  • LLVM (clang++) 下 C++ 对象的内存构造,包含为 virtual 函数而生的 vtable
  • gdb 小恶魔的入坑,剖析 C++ 对象构造

坐稳,扶好。为免吓跑人,还是老套图,多图少代码,不过有的图有点点简单。程序员大叔开始讲故事了。

为何

为何要 trace

这是个很好的问题。我在 逆向工程与云原生现场剖析 Part1 —— eBPF 跟踪 Istio/Envoy 之学步 中曾经尝试答复。如果抛开那些大话,简略来说:

  • 实在

    • 源码只通知了咱们程序的能力和可能性,而且很多可能性深深地暗藏于简单的代码逻辑中很难发现。而这又和程序运行的环境、压力、甚至 CPU 温度无关🤕。只有现场的 trace 能力通知咱们实在的运行期行为。
  • 无常

    • 无常不然而佛系的词,每个关注生产运行期的经验过劫难事变的程序员,都心明于此。问题呈现时,设计再良好的埋点和日志永远不够用。

为何要理解启动和监听

现代人很忙,所以做每件事,都要给本人一个很好的理由(jie kou)。如果我说 just for fun. 预计很多人就 bye bye 了。所以我先引几个问题:

  1. 运行期的启动行为会影响性能吗?如果会,怎么影响力?
  2. 咱们看了源码,但咱们真理解它在咱们环境的实在执行门路吗?还是靠教训猜?
  3. 每一行启动代码,都会应用于你的环境吗?还是咱们认为是或不是?
  4. 如何优化 Envoy 性能、或者深度定制配置、定制开发?—— 深刻源码,但源码不好推理或证实理论运行期行为。
  5. 你晓得 Envoy 在哪儿,哪个 listerner,哪个线程中 open socket fd、bind socket、listen socket 吗?又在什么中央配置了什么 socket opt ?
  6. Worker thread 和 listner 是什么关系?
  7. Worker thread 是如何做到 event 驱动的?
  8. 在 C++ 的 OOP 形象、多态与封装下,如何理解运行期的理论行为?

术语

开始前简略过一下术语,以缩小前面的误会:

  • upstream: 流量方向中的角色:[downstream] –> envoy –> [upstream]。这里我防止用中文词 上 / 上游,因为概念上没有对立,也容易和英文误会。
  • downstream: 流量方向中的角色:[downstream] –> envoy –> [upstream]

以上只是局部术语,其它术语我将在首次应用时作介绍。

须要留神的是,upstream 与 downstream 是个绝对于观察者的概念。

如场景: service A – 调用 –> service B – 调用 –> service C :

  • 如果站在 service C 上,咱们在把 service B 叫 downstream;
  • 如果站在 service A 上,咱们把 service B 叫 upstream。

例子阐明

我不是一个太喜爱只讲实践的人。也置信对于初学一个实践的人来说,再多的实践形象形容,还不如一个具体例子来得疾速了解。即所谓的能够意会不能言传。从这里开始,本文将应用一个简略的部署例子来阐明一系列的实践和术语。

例子软件版本

  • Istio proxy release-1.11

    • Envoy version: 1.19.1
  • bpftrace v0.14.1

例子部署脚本

见:https://github.com/labilezhu/…

例子服务调用关系


图:服务与调用关系图

有 3 个服务,自下而上地调用。

  1. client
  2. fortio-server:8080
  3. fortio-server-l2:8080

图这样排版也是想间接反映 upstream 和 downstream 字面义。图中须要解释的,或者只有 inbound / outbound 这两个术语。首先,什么是 bound

  • bound: 字面意为边界。有人译为 站(名词)。而在事实的 k8s + istio 环境中,能够了解为 pod / service
  • inbound: 有人译为 入站。而在事实的 k8s + istio 环境中,能够了解为流量从 pod 内部进入 pod。即服务的被调用流量
  • outbound: 有人译为 出站。而在事实的 k8s + istio 环境中,能够了解为流量从 pod 外部输入到 pod 内部。

须要留神的是,对于同一个调用申请。他能够是调用者 service 的 outbound,同时也是被调用者的 inbound。如上图

回到咱们的例子,其实调用关系就是:

client –> fortio-server:8080 –> fortio-server-l2:8080

例子操作系统层面的组件

咱们再看看 k8s/Istio 管制下的各组件,在操作系统层面的组件与构造:


图:操作系统层面的组件与构造

置信图中曾经阐明,不作文字复述了。上面开始联合例子介绍一下 Evnoy 的理念和术语。

简介启动相干的架构

这部分是简介,因为篇幅限度也不可能说得十分分明,我在思考未来写一个独立的系列去阐明。如果你是 Envoy 新手,跳过吧。

网上有很多 Envoy 的架构图,一个是 Envoy 原作者 Matt Klein, Lyft 的 [Envoy Internals Deep Dive – Matt Klein, Lyft (Advanced Skill Level)]:

图源:[Envoy Internals Deep Dive – Matt Klein, Lyft (Advanced Skill Level)]

一个是当初比拟沉闷的 来自 Tetrate 的 Lizan Zhou 的 [Deep Dive: Envoy]

图源:Lizan Zhou 的 [Deep Dive: Envoy]

两图区别不大,看你是否喜爱上色了。

本文只关注启动和监听,所以咱们只看:

  • Worker Thread
  • Listener
  • Listener Filter
  • Network (L3/L4) filters

Envoy 外部基本概念

Worker Thread

Envoy 的 worker thread 当然是事件驱动和 non-blocking 的了:


图源:[Envoy Internals Deep Dive – Matt Klein, Lyft (Advanced Skill Level)]

即一条线程会负责多个 downstream 的 socket 连贯。有同学问,和 node.js 一样,单个 loop worker thread 就足够,为何要反对多个?

起因起码有:

  • non-blocking 操作有时还是会 blocking 或 waiting。如内存缓和时的内存申请,甚至虚拟内存 swap-in/out。
  • 某 worker 线程有时会被操作系统调度到一个比较忙的 cpu core(Noisy neighbor),而刚好一个不定时忙碌的其它利用线程与也跑在这个 core 上。这就呈现 core 调度 waiting
  • NUMA 架构的机器中。单个过程要充分利用各个内存通道,必须多线程。这个 Envoy 作为 Istio gateway 部署在 NUMA 机器时,由为重要。

那么问题来了,Worker Thread 是什么时候启动?他是在下面执行 socket bind/listen 的吗?如何实现 event loop ?

Listener

顾名思义,就是监听流量的。是不是每个 Listener 都会 listen socket ? 前面再看。先浏览一下 Listener 相干的组件和启动程序:

Listener 相干的组件和启动程序

不好意思,间接由下面的 High Level 阐明,一下拉回高空,讲源码了。不过我尽量不贴源码进去吓唬人,而是先从类性能、构造、职责说说。
Envoy 只有两种类型的 Listener 实现。TCP 和 UDP 的。这里我只看 TCP 了。图中信息量不少,不必怕,我缓缓道来。

首先介绍一下外围类:

  • TcpListenerImpl – Listener 的外围主类。负责 listen socket 和 listen socket 事件处理(就是收到 TCP Client 的 SYN 触发,即能够 accept socket 了)。

    • 那么每个 Worker Thread 会为配置的每个 Listener 创立本人的专属 TcpListenerImpl 实例。

      • 如:咱们配置了两个 Listener, L1 和 L2。有两个 work thread: W0 和 W1。那么会有 4 个 TcpListenerImpl 实例。
    • 能够见到 TcpListenerImpl 中有bool bind_to_port 你就明确,真有不 bind/listen socket 的 TcpListenerImpl 了。
  • TcpListenSocket – 负责 Listner 的理论 socket 操作,包含 bind 和 listen
  • WorkerImpl – Worker 线程的主入口类
  • DispatcherImpl – 主事件循环和队列类
  • ListenerManagerImpl

    • 创立 和 bind TcpListenSocket。
    • 按配置参数创立 WorkerImpl
    • 触发创立 TcpListenerImpl

或者你和我一样,刚看 Envoy 的代码时,总会混同名字相近的类。如:TcpListenerImpl、TcpListenSocket。

仔细的同学如果看了图例,就晓得图中黑、红连线代表不同类型的线程的。说说图中主流程:

  1. 过程 main 间接调用 ListenerManagerImpl
  2. bind socket 绑定到 ip 和 port
  3. 启动新的 worker 线程
  4. 退出异步 task:add Listener task(每个 Worker + Listener 执行一次) 到 worker 的工作队列中。
  5. worker 线程取出工作队列,执行 add Listener task
  6. worker 线程 异步 listen socket,和注册事件处理器

仔细的同学会发现问题:

  • 为何要在主线程中 bind socket?

    • 能够在过程启动晚期就发现 socket 监听端口抵触等常见问题。具体解释在 Envoy 的源码文档中 https://github.com/envoyproxy…。
  • 两个 worker 线程能够 listen 同一个 socket?

    • 在旧版本默认不应用 reuse_port 状况下,是应用 duplicate socket/file descriptor 的办法为每个 work thread 复制一个文件描述符。
    • 在应用 reuse_port 状况下(新版本如同默认),是应用当然能够每个线程独立 create/bind/listen 雷同 port 的 socket 了。见我的文章:https://blog.mygraphql.com/zh…

bpftrace 跟踪剖析 Listener 的理论监听行为

下面咱们对源码进行动态剖析和推理,但为了证实推理,还是要 bpftrace 一下,看看 istio proxy evnoy 实在 socket 行为、线程行为、代码地位(堆栈)。

这可能是少有的讲 Envoy 源码但不贴 Envoy 源码的文章🤣。不过 bpftrace 的代码还是要贴的,不然会给人骂的。

bpftrace 脚本地址:https://github.com/labilezhu/…

bpftrace 脚本

#!usr/bin/bpftrace

#include <linux/in.h>
#include <linux/in6.h>

BEGIN
{
}

tracepoint:syscalls:sys_enter_execve
{if( str(args->argv[0]) == "/usr/local/bin/envoy" ) {@watchedpid[pid] = pid;
              printf("watched envoy pid: %d\n", pid);
       }
}

tracepoint:syscalls:sys_enter_setsockopt
/@watchedpid[pid]/
{
       // socket opts: https://elixir.bootlin.com/linux/v5.16.3/source/include/uapi/linux/tcp.h#L92     
       $level = args->level;
       $fd = args->fd;
       $optname = args->optname;
       $optval = args->optval;
       $optval_int = *$optval;
       $optlen = args->optlen;
       printf("\n########## setsockopt() ##########\n");
       printf("comm:%-16s: setsockopt: level=%d, fd=%d, optname=%d, optval=%d, optlen=%d. \n", comm, $level, $fd, $optname, $optval_int, $optlen);
       @fd2sockopt[$fd] = ($level, $optname, $optval_int);
}


tracepoint:syscalls:sys_enter_bind
/@watchedpid[pid]/
{$sa = (struct sockaddr *)args->umyaddr;
       $fd = args->fd;
       printf("\n########## bind() ##########\n");

       if ($sa->sa_family == AF_INET || $sa->sa_family == AF_INET6) {if ($sa->sa_family == AF_INET) { //IPv4
                     $s = (struct sockaddr_in *)$sa;
                     $port = ($s->sin_port >> 8) |
                         (($s->sin_port << 8) & 0xff00);
                     $bind_ip = ntop(AF_INET, $s->sin_addr.s_addr);                         
                     printf("comm:%-16s: bind AF_INET: ip:%-16s port:%-5d fd=%d \n", comm,
                         $bind_ip,
                         $port, $fd);
                     @fd2bind[$fd] = ($bind_ip, $port);
                     printf("stack: %s\n", ustack);
              } else { //IPv6
                     printf("not support ipv6\n");
              }

       }      
}

tracepoint:syscalls:sys_enter_listen
/@watchedpid[pid]/
{printf("\n########## listen() ##########\n");

       $fd = args->fd;
       $backlog = args->backlog;

       @fd2listen[$fd] = 1;
       $bind = @fd2bind[$fd];
       if($bind.1 == 0) {$old_fd = @dup_fd_new2old[$fd];
              if($old_fd) {printf("comm:%-16s: listen() on an dupliated old_fd=%d \n", comm, $old_fd);
                     $bind = @fd2bind[$old_fd];
              }
       }
       printf("comm:%-16s: listen() fd=%d, ip:%-16s, port:%-5d \n", comm, $fd, $bind.0, $bind.1);
       printf("stack: %s\n", ustack);
}

tracepoint:syscalls:sys_enter_dup
/@watchedpid[pid]/
{
       $fildes = args->fildes;
       @duping_fd_cache[tid]= $fildes;
}



tracepoint:syscalls:sys_exit_dup
/@watchedpid[pid] && @duping_fd_cache[tid]/
{$duping_fd = @duping_fd_cache[tid];
       delete(@duping_fd_cache[tid]);

       $new_fd = args->ret;
       @dup_fd_old2new[$duping_fd]=$new_fd;
       @dup_fd_new2old[$new_fd]=$duping_fd;

       printf("\n########## sys_enter_dup() ##########\n");
       printf("comm:%-16s: duping_fd=%d, new_fd=%d \n", comm, $duping_fd, $new_fd);
       printf("stack: %s\n", ustack);
}


END
{clear(@watchedpid);
       clear(@dup_fd_old2new);
}

有点 TLDR;

bpftrace 执行

环境阐明:我是在一个 k8s worknode 上执行 bpftrace 的。worknode 上跑着两个有 istio-proxy 的 pod。

见:例子部署脚本。
因为 worknode 上 Ubuntu 自带的 bpftrace 有 bug。我本人编译和打包了最新 bpftrace v0.14.1 到 docker 镜像中。

执行步骤:

  1. 启动 bpftrace 脚本, 开启零碎级的探针。
➜  bpftrace git:(main) ✗ docker run -it --rm --init  --privileged --name bpftrace -h bpftrace \
    --pid host \
    --net host \
    -e SCRIPT_HOME=$SCRIPT_HOME \
    -v /etc/localtime:/etc/localtime:ro \
    -v /sys:/sys:rw \
    -v /usr/src:/usr/src:rw \
    -v /lib/modules:/lib/modules:ro \
    -v ${SCRIPT_HOME}:${SCRIPT_HOME}:rw \
    $bpftrace_image \
    bpftrace  ${SCRIPT_HOME}/trace-envoy-socket-listen.bt
  1. 重启其中一个 有 istio-proxy 的 pod,如:

    kubectl delete pod fortio-server
    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Pod
    metadata:
     name: fortio-server
     labels:
         app.kubernetes.io/name: fortio-server
         app: fortio-server
    spec:
     restartPolicy: Always
     imagePullSecrets:
     - name: docker-registry-key
     containers:
     - name: main-app
       image: docker.io/fortio/fortio
    ....

bpftrace 输入

老习惯,我先不讲原理,先贴出 trace 的输入(stack 有删减,倡议看原版本:https://github.com/labilezhu/…):

Attaching 8 probes...
watched envoy pid: 2062496

########## bind() ##########

########## setsockopt() ##########
comm:envoy           : setsockopt: level=1, fd=18, optname=2, optval=1, optlen=4. 

########## bind() ##########
comm:envoy           : bind AF_INET: ip:127.0.0.1        port:15000 fd=18 
stack: 
        bind+11
        Envoy::Network::IoSocketHandleImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+101
        Envoy::Network::SocketImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+383
        Envoy::Network::ListenSocketImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+77
        Envoy::Network::ListenSocketImpl::setupSocket(std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, bool)+76
        Envoy::Network::NetworkListenSocket<Envoy::Network::NetworkSocketTrait<(Envoy::Network::Socket::Type)0> >::NetworkListenSocket(std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, bool)+486
        Envoy::Server::InstanceImpl::initialize(Envoy::Server::Options const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::ComponentFactory&)+12555
        main+44


########## listen() ##########
comm:envoy           : listen() fd=18, ip:127.0.0.1       , port:15000 
stack: 
        listen+11
        Envoy::Network::IoSocketHandleImpl::listen(int)+69
        Envoy::Network::TcpListenerImpl::setupServerSocket(Envoy::Event::DispatcherImpl&, Envoy::Network::Socket&)+62
        Envoy::Network::TcpListenerImpl::TcpListenerImpl(Envoy::Event::DispatcherImpl&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Network::Socket>, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+195
        Envoy::Event::DispatcherImpl::createListener(std::__1::shared_ptr<Envoy::Network::Socket>&&, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+108
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+116
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Server::InstanceImpl::initialize(Envoy::Server::Options const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::ComponentFactory&)+13195
        main+44


########## setsockopt() ##########
comm:envoy           : setsockopt: level=1, fd=21, optname=2, optval=1, optlen=4. 

########## bind() ##########
comm:envoy           : bind AF_INET: ip:0.0.0.0          port:15090 fd=21 
stack: 
        bind+11
        Envoy::Network::IoSocketHandleImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+101
        Envoy::Network::SocketImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+383
        Envoy::Network::ListenSocketImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+77
        Envoy::Network::ListenSocketImpl::setupSocket(std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, bool)+76
        Envoy::Server::ListenSocketFactoryImpl::createListenSocketAndApplyOptions()+114
        Envoy::Server::ListenerManagerImpl::createListenSocketFactory(envoy::config::core::v3::Address const&, Envoy::Server::ListenerImpl&, bool)+133
        Envoy::Server::ListenerManagerImpl::setNewOrDrainingSocketFactory(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, envoy::config::core::v3::Address const&, Envoy::Server::ListenerImpl&, bool)+783
        Envoy::Server::ListenerManagerImpl::addOrUpdateListenerInternal(envoy::config::listener::v3::Listener const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+3172
        Envoy::Server::ListenerManagerImpl::addOrUpdateListener(envoy::config::listener::v3::Listener const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool)+409
        Envoy::Server::Configuration::MainImpl::initialize(envoy::config::bootstrap::v3::Bootstrap const&, Envoy::Server::Instance&, Envoy::Upstream::ClusterManagerFactory&)+2135
        Envoy::Server::InstanceImpl::initialize(Envoy::Server::Options const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::ComponentFactory&)+14470
        main+44


########## setsockopt() ##########
comm:envoy           : setsockopt: level=1, fd=22, optname=2, optval=1, optlen=4. 

########## bind() ##########
comm:envoy           : bind AF_INET: ip:0.0.0.0          port:15021 fd=22 
stack: 
        bind+11
        Envoy::Network::IoSocketHandleImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+101
        Envoy::Network::SocketImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+383
        Envoy::Network::ListenSocketImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+77
        Envoy::Network::ListenSocketImpl::setupSocket(std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, bool)+76
        Envoy::Server::ListenSocketFactoryImpl::createListenSocketAndApplyOptions()+114
        Envoy::Server::ListenerManagerImpl::createListenSocketFactory(envoy::config::core::v3::Address const&, Envoy::Server::ListenerImpl&, bool)+133
        Envoy::Server::ListenerManagerImpl::setNewOrDrainingSocketFactory(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, envoy::config::core::v3::Address const&, Envoy::Server::ListenerImpl&, bool)+783
        Envoy::Server::ListenerManagerImpl::addOrUpdateListenerInternal(envoy::config::listener::v3::Listener const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+3172
        Envoy::Server::ListenerManagerImpl::addOrUpdateListener(envoy::config::listener::v3::Listener const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool)+409
        Envoy::Server::Configuration::MainImpl::initialize(envoy::config::bootstrap::v3::Bootstrap const&, Envoy::Server::Instance&, Envoy::Upstream::ClusterManagerFactory&)+2135
        Envoy::Server::InstanceImpl::initialize(Envoy::Server::Options const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::ComponentFactory&)+14470
        main+44


########## setsockopt() ##########
comm:envoy           : setsockopt: level=1, fd=31, optname=2, optval=1, optlen=4. 

########## bind() ##########
comm:envoy           : bind AF_INET: ip:0.0.0.0          port:15001 fd=31 
stack: 
        bind+11
        Envoy::Network::IoSocketHandleImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+101
        Envoy::Network::SocketImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+383
        Envoy::Network::ListenSocketImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+77
        Envoy::Network::ListenSocketImpl::setupSocket(std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, bool)+76
        Envoy::Server::ListenSocketFactoryImpl::createListenSocketAndApplyOptions()+114
        Envoy::Server::ListenerManagerImpl::createListenSocketFactory(envoy::config::core::v3::Address const&, Envoy::Server::ListenerImpl&, bool)+133
        Envoy::Server::ListenerManagerImpl::setNewOrDrainingSocketFactory(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, envoy::config::core::v3::Address const&, Envoy::Server::ListenerImpl&, bool)+783
        Envoy::Server::ListenerManagerImpl::addOrUpdateListenerInternal(envoy::config::listener::v3::Listener const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+3172
        Envoy::Server::ListenerManagerImpl::addOrUpdateListener(envoy::config::listener::v3::Listener const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool)+409
        Envoy::Config::GrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DiscoveryResponse> >&&, Envoy::Config::ControlPlaneStats&)+3664
        Envoy::Grpc::AsyncStreamCallbacks<envoy::service::discovery::v3::DiscoveryResponse>::onReceiveMessageRaw(std::__1::unique_ptr<Envoy::Buffer::Instance, std::__1::default_delete<Envoy::Buffer::Instance> >&&)+161
        Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool)+199
        Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool)+354
        Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool)+230
        Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
        Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*)+1413
        Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_22::__invoke(nghttp2_session*, nghttp2_frame const*, void*)+32
        nghttp2_session_on_data_received+172
        nghttp2_session_mem_recv+7174
        Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1120
        virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&)+48
        Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool)+21
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        event_process_active_single_queue+1416
        event_base_loop+1953
        Envoy::Server::InstanceImpl::run()+747
        Envoy::MainCommonBase::run()+68
        Envoy::MainCommon::main(int, char**, std::__1::function<void (Envoy::Server::Instance&)>)+148
        main+44


########## setsockopt() ##########
comm:envoy           : setsockopt: level=1, fd=32, optname=2, optval=1, optlen=4. 

########## bind() ##########
comm:envoy           : bind AF_INET: ip:0.0.0.0          port:15006 fd=32 
stack: 
        bind+11
        Envoy::Network::IoSocketHandleImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+101
        Envoy::Network::SocketImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+383
        Envoy::Network::ListenSocketImpl::bind(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+77
        Envoy::Network::ListenSocketImpl::setupSocket(std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, bool)+76
        Envoy::Server::ListenSocketFactoryImpl::createListenSocketAndApplyOptions()+114
        Envoy::Server::ListenerManagerImpl::createListenSocketFactory(envoy::config::core::v3::Address const&, Envoy::Server::ListenerImpl&, bool)+133
        Envoy::Server::ListenerManagerImpl::setNewOrDrainingSocketFactory(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, envoy::config::core::v3::Address const&, Envoy::Server::ListenerImpl&, bool)+783
        Envoy::Server::ListenerManagerImpl::addOrUpdateListenerInternal(envoy::config::listener::v3::Listener const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+3172
        Envoy::Server::ListenerManagerImpl::addOrUpdateListener(envoy::config::listener::v3::Listener const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool)+409
        Envoy::Config::GrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DiscoveryResponse> >&&, Envoy::Config::ControlPlaneStats&)+3664
        Envoy::Grpc::AsyncStreamCallbacks<envoy::service::discovery::v3::DiscoveryResponse>::onReceiveMessageRaw(std::__1::unique_ptr<Envoy::Buffer::Instance, std::__1::default_delete<Envoy::Buffer::Instance> >&&)+161
        Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool)+199
        Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool)+354
        Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool)+230
        Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
        Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*)+1413
        Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_22::__invoke(nghttp2_session*, nghttp2_frame const*, void*)+32
        nghttp2_session_on_data_received+172
        nghttp2_session_mem_recv+7174
        Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1120
        virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&)+48
        Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool)+21
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        event_process_active_single_queue+1416
        event_base_loop+1953
        Envoy::Server::InstanceImpl::run()+747
        Envoy::MainCommonBase::run()+68
        Envoy::MainCommon::main(int, char**, std::__1::function<void (Envoy::Server::Instance&)>)+148
        main+44
        __libc_start_main+243


########## sys_enter_dup() ##########
comm:wrk:worker_0    : duping_fd=21, new_fd=33 
stack: 
        dup+11
        Envoy::Network::IoSocketHandleImpl::duplicate()+71
        Envoy::Network::ListenSocketImpl::duplicate()+54
        Envoy::Server::ListenSocketFactoryImpl::getListenSocket()+102
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+60
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## listen() ##########
comm:wrk:worker_0    : listen() on an dupliated old_fd=21 
comm:wrk:worker_0    : listen() fd=33, ip:0.0.0.0         , port:15090 
stack: 
        listen+11
        Envoy::Network::IoSocketHandleImpl::listen(int)+69
        Envoy::Network::TcpListenerImpl::setupServerSocket(Envoy::Event::DispatcherImpl&, Envoy::Network::Socket&)+62
        Envoy::Network::TcpListenerImpl::TcpListenerImpl(Envoy::Event::DispatcherImpl&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Network::Socket>, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+195
        Envoy::Event::DispatcherImpl::createListener(std::__1::shared_ptr<Envoy::Network::Socket>&&, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+108
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+116
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        std::__1::__function::__func<Envoy::Server::WorkerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&, std::__1::function<void (bool)>)::$_2, std::__1::allocator<Envoy::Server::WorkerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&, std::__1::function<void (bool)>)::$_2>, void ()>::operator()()+46
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## sys_enter_dup() ##########
comm:wrk:worker_0    : duping_fd=22, new_fd=34 
stack: 
        dup+11
        Envoy::Network::IoSocketHandleImpl::duplicate()+71
        Envoy::Network::ListenSocketImpl::duplicate()+54
        Envoy::Server::ListenSocketFactoryImpl::getListenSocket()+102
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+60
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## listen() ##########
comm:wrk:worker_0    : listen() on an dupliated old_fd=22 
comm:wrk:worker_0    : listen() fd=34, ip:0.0.0.0         , port:15021 
stack: 
        listen+11
        Envoy::Network::IoSocketHandleImpl::listen(int)+69
        Envoy::Network::TcpListenerImpl::setupServerSocket(Envoy::Event::DispatcherImpl&, Envoy::Network::Socket&)+62
        Envoy::Network::TcpListenerImpl::TcpListenerImpl(Envoy::Event::DispatcherImpl&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Network::Socket>, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+195
        Envoy::Event::DispatcherImpl::createListener(std::__1::shared_ptr<Envoy::Network::Socket>&&, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+108
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+116
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## sys_enter_dup() ##########
comm:wrk:worker_1    : duping_fd=21, new_fd=35 
stack: 
        dup+11
        Envoy::Network::IoSocketHandleImpl::duplicate()+71
        Envoy::Network::ListenSocketImpl::duplicate()+54
        Envoy::Server::ListenSocketFactoryImpl::getListenSocket()+102
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+60
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## listen() ##########
comm:wrk:worker_1    : listen() on an dupliated old_fd=21 
comm:wrk:worker_1    : listen() fd=35, ip:0.0.0.0         , port:15090 
stack: 
        listen+11
        Envoy::Network::IoSocketHandleImpl::listen(int)+69
        Envoy::Network::TcpListenerImpl::setupServerSocket(Envoy::Event::DispatcherImpl&, Envoy::Network::Socket&)+62
        Envoy::Network::TcpListenerImpl::TcpListenerImpl(Envoy::Event::DispatcherImpl&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Network::Socket>, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+195
        Envoy::Event::DispatcherImpl::createListener(std::__1::shared_ptr<Envoy::Network::Socket>&&, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+108
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+116
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## sys_enter_dup() ##########
comm:wrk:worker_1    : duping_fd=22, new_fd=36 
stack: 
        dup+11
        Envoy::Network::IoSocketHandleImpl::duplicate()+71
        Envoy::Network::ListenSocketImpl::duplicate()+54
        Envoy::Server::ListenSocketFactoryImpl::getListenSocket()+102
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+60
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## listen() ##########
comm:wrk:worker_1    : listen() on an dupliated old_fd=22 
comm:wrk:worker_1    : listen() fd=36, ip:0.0.0.0         , port:15021 
stack: 
        listen+11
        Envoy::Network::IoSocketHandleImpl::listen(int)+69
        Envoy::Network::TcpListenerImpl::setupServerSocket(Envoy::Event::DispatcherImpl&, Envoy::Network::Socket&)+62
        Envoy::Network::TcpListenerImpl::TcpListenerImpl(Envoy::Event::DispatcherImpl&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Network::Socket>, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+195
        Envoy::Event::DispatcherImpl::createListener(std::__1::shared_ptr<Envoy::Network::Socket>&&, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+108
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+116
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## sys_enter_dup() ##########
comm:wrk:worker_1    : duping_fd=31, new_fd=37 
stack: 
        dup+11
        Envoy::Network::IoSocketHandleImpl::duplicate()+71
        Envoy::Network::ListenSocketImpl::duplicate()+54
        Envoy::Server::ListenSocketFactoryImpl::getListenSocket()+102
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+60
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## listen() ##########
comm:wrk:worker_1    : listen() on an dupliated old_fd=31 
comm:wrk:worker_1    : listen() fd=37, ip:0.0.0.0         , port:15001 
stack: 
        listen+11
        Envoy::Network::IoSocketHandleImpl::listen(int)+69
        Envoy::Network::TcpListenerImpl::setupServerSocket(Envoy::Event::DispatcherImpl&, Envoy::Network::Socket&)+62
        Envoy::Network::TcpListenerImpl::TcpListenerImpl(Envoy::Event::DispatcherImpl&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Network::Socket>, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+195
        Envoy::Event::DispatcherImpl::createListener(std::__1::shared_ptr<Envoy::Network::Socket>&&, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+108
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+116
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## sys_enter_dup() ##########
comm:wrk:worker_0    : duping_fd=31, new_fd=38 
stack: 
        dup+11
        Envoy::Network::IoSocketHandleImpl::duplicate()+71
        Envoy::Network::ListenSocketImpl::duplicate()+54
        Envoy::Server::ListenSocketFactoryImpl::getListenSocket()+102
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+60
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## listen() ##########
comm:wrk:worker_0    : listen() on an dupliated old_fd=31 
comm:wrk:worker_0    : listen() fd=38, ip:0.0.0.0         , port:15001 
stack: 
        listen+11
        Envoy::Network::IoSocketHandleImpl::listen(int)+69
        Envoy::Network::TcpListenerImpl::setupServerSocket(Envoy::Event::DispatcherImpl&, Envoy::Network::Socket&)+62
        Envoy::Network::TcpListenerImpl::TcpListenerImpl(Envoy::Event::DispatcherImpl&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Network::Socket>, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+195
        Envoy::Event::DispatcherImpl::createListener(std::__1::shared_ptr<Envoy::Network::Socket>&&, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+108
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+116
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        std::__1::__function::__func<Envoy::Server::WorkerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&, std::__1::function<void (bool)>)::$_2, std::__1::allocator<Envoy::Server::WorkerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&, std::__1::function<void (bool)>)::$_2>, void ()>::operator()()+46
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## sys_enter_dup() ##########
comm:wrk:worker_0    : duping_fd=32, new_fd=39 
stack: 
        dup+11
        Envoy::Network::IoSocketHandleImpl::duplicate()+71
        Envoy::Network::ListenSocketImpl::duplicate()+54
        Envoy::Server::ListenSocketFactoryImpl::getListenSocket()+102
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+60
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## listen() ##########
comm:wrk:worker_0    : listen() on an dupliated old_fd=32 
comm:wrk:worker_0    : listen() fd=39, ip:0.0.0.0         , port:15006 
stack: 
        listen+11
        Envoy::Network::IoSocketHandleImpl::listen(int)+69
        Envoy::Network::TcpListenerImpl::setupServerSocket(Envoy::Event::DispatcherImpl&, Envoy::Network::Socket&)+62
        Envoy::Network::TcpListenerImpl::TcpListenerImpl(Envoy::Event::DispatcherImpl&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Network::Socket>, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+195
        Envoy::Event::DispatcherImpl::createListener(std::__1::shared_ptr<Envoy::Network::Socket>&&, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+108
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+116
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## sys_enter_dup() ##########
comm:wrk:worker_1    : duping_fd=32, new_fd=40 
stack: 
        dup+11
        Envoy::Network::IoSocketHandleImpl::duplicate()+71
        Envoy::Network::ListenSocketImpl::duplicate()+54
        Envoy::Server::ListenSocketFactoryImpl::getListenSocket()+102
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+60
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## listen() ##########
comm:wrk:worker_1    : listen() on an dupliated old_fd=32 
comm:wrk:worker_1    : listen() fd=40, ip:0.0.0.0         , port:15006 
stack: 
        listen+11
        Envoy::Network::IoSocketHandleImpl::listen(int)+69
        Envoy::Network::TcpListenerImpl::setupServerSocket(Envoy::Event::DispatcherImpl&, Envoy::Network::Socket&)+62
        Envoy::Network::TcpListenerImpl::TcpListenerImpl(Envoy::Event::DispatcherImpl&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Network::Socket>, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+195
        Envoy::Event::DispatcherImpl::createListener(std::__1::shared_ptr<Envoy::Network::Socket>&&, Envoy::Network::TcpListenerCallbacks&, bool, unsigned int)+108
        Envoy::Server::ActiveTcpListener::ActiveTcpListener(Envoy::Network::TcpConnectionHandler&, Envoy::Network::ListenerConfig&)+116
        Envoy::Server::ConnectionHandlerImpl::addListener(absl::optional<unsigned long>, Envoy::Network::ListenerConfig&)+671
        Envoy::Event::DispatcherImpl::runPostCallbacks()+317
        Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType)+44
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


########## setsockopt() ##########
comm:wrk:worker_0    : setsockopt: level=6, fd=41, optname=1, optval=1, optlen=4. 

########## setsockopt() ##########
comm:wrk:worker_0    : setsockopt: level=6, fd=42, optname=1, optval=1, optlen=4. 

########## setsockopt() ##########
comm:envoy           : setsockopt: level=6, fd=43, optname=1, optval=1, optlen=4. 

########## setsockopt() ##########
comm:wrk:worker_0    : setsockopt: level=6, fd=41, optname=1, optval=1, optlen=4. 

########## setsockopt() ##########
comm:wrk:worker_1    : setsockopt: level=6, fd=41, optname=1, optval=1, optlen=4. 

...
^C

@dup_fd_new2old[33]: 21
@dup_fd_new2old[35]: 21
@dup_fd_new2old[36]: 22
@dup_fd_new2old[34]: 22
@dup_fd_new2old[38]: 31
@dup_fd_new2old[37]: 31
@dup_fd_new2old[40]: 32
@dup_fd_new2old[39]: 32



@fd2bind[18]: (127.0.0.1, 15000)
@fd2bind[21]: (0.0.0.0, 15090)
@fd2bind[22]: (0.0.0.0, 15021)
@fd2bind[31]: (0.0.0.0, 15001)
@fd2bind[32]: (0.0.0.0, 15006)
@fd2listen[35]: 1
@fd2listen[39]: 1
@fd2listen[37]: 1
@fd2listen[18]: 1
@fd2listen[38]: 1
@fd2listen[34]: 1
@fd2listen[36]: 1
@fd2listen[40]: 1
@fd2listen[33]: 1
@fd2sockopt[18]: (1, 2, 1)
@fd2sockopt[21]: (1, 2, 1)
@fd2sockopt[22]: (1, 2, 1)
@fd2sockopt[31]: (1, 2, 1)
@fd2sockopt[32]: (1, 2, 1)
@fd2sockopt[41]: (6, 1, 1)
@fd2sockopt[42]: (6, 1, 1)
@fd2sockopt[43]: (6, 1, 1)
@fd2sockopt[44]: (6, 1, 1)
@fd2sockopt[45]: (6, 1, 1)

原理与剖析

原理

在分析家原理前,须要先简略理解一下 Linux server socket 编程的概念:

  • socket:本文中能够认为是 TCP 连贯在应用层的概念。
  • file descriptor:文件描述符,缩写是 fd(Windows 编程过去的同学叫:句柄 handle)。一个 socket 能够对应于多个 fd。不同的线程能够在不同的 fd 上 listen 同一个 socket。操作系统负责新连贯的负载平衡(只管做得不太好)。
  • dup() system call : 就是复制 fd。一个 socket 如果想对应多个 fd。能够 dup() 一下原 fd。为何要复制?因每个 worker 能够独立做 listen 和 close。详见:https://github.com/envoyproxy…
  • bind:把 socket fd 绑定到 ip 地址和 port。思考:不同 socket 能够反复绑定到雷同地址吗?如果能够,为何须要?见: https://blog.mygraphql.com/zh…
  • listen:开启 socket 监听,有同步线程期待和异步告诉两种模式,当然,envoy 用后者。

回到脚本。看看探针点(probe):

  • sys_enter_execve – 监听操作系统上的所有启动新过程行为。如果是 envoy 过程,就记下 pid。用于前面过滤其它监控点。
  • sys_enter_bind – 监听 fd 的 bind 操作。记下 fd 监听的 ip 地址和 port。打印出操作线程名与 stack。
  • sys_enter_listen – 监听 fd 的 listen 操作。记下 fd 监听的 ip 地址和 port。打印出操作线程名与 stack。
  • sys_enter_dup/sys_exit_dup – 监听 fddup 操作。记录下 fd 之间的父子关系。打印出操作线程名与 stack。
  • END 局部 – 脚本运行完结(用户按下 Ctrl+C)后,bpftrace 默认会打印所有记录的 bpf map。有的外部用,不须要打印,就革除。

剖析

本文只关怀 inbound 和 outbound 的 Listener 和相干 socket 与 port:

  • virtualOutbound Listener – 15001
  • virtualInbound Listener – 15006

阐明一下输入最初局部的 bpf map:

  • fd2bind – fd 和它 socket bind 的地址
  • dup_fd_new2old – dup() 进去的新 fd 和 原 fd
  • fd2listen – 执行了 listen 操作的 fd
  • fd2sockopt – 执行了 sockopt 操作的 fd,和 sockopt 的参数。

如果咱们这时在 POD 的 linux network namespace 上来执行:

$ sudo nsenter -t $POD_PID -n

$ ss -lnp | egrep '15001|15006'

tcp   LISTEN   0   4096   0.0.0.0:15001   0.0.0.0:*    users:(("envoy",pid=2062496,fd=37),("envoy",pid=2062496,fd=38),("envoy",pid=2062496,fd=31))
tcp   LISTEN   0   4096   0.0.0.0:15006   0.0.0.0:*    users:(("envoy",pid=2062496,fd=39),("envoy",pid=2062496,fd=40),("envoy",pid=2062496,fd=32))

注:我用的 Istio/Evnoy 版本默认没启用 reuse_port。下面数据也是基于这个收集和剖析的。

我想聪慧怯懦如你能保持读到这里,大略曾经明确了。剩下的就是看程序结构、流程、行为点了。

还记得后面的 High Level 流程吗?当初再联合下面的 stack 输入再次剖析:


Listener 相干的组件和启动程序 – 外围流程图

Listener 相干的组件和启动程序 – 外围流程图阐明以下几步:

  1. 过程 main 间接调用 ListenerManagerImpl
  2. bind socket 绑定到 ip 和 port
  3. 启动新的 worker 线程
  4. 退出异步 task:add Listener task(每个 Worker + Listener 执行一次) 到 worker 的工作队列中。
  5. worker 线程取出工作队列,执行 add Listener task
  6. worker 线程 异步 listen socket,和注册事件处理器

但从 stack 输入中,你应该能够晓得,事实必定比外围简单。我想,当初是回到简单事实的时候了。让说好的不贴代码见鬼去:


Listener 相干的组件和启动程序 – 代码流程图

如果你对照上图和 bpftrace 输入的 stack。置信,我曾经不必写多少文字剖析了。两显示器是必须的。

结尾

总感觉本 part 还有些货色未说分明:

  • reuse_port – https://blog.mygraphql.com/zh…
  • 说好的 uprobe / 剖析运行期 C++ 对象构造 / vtable – 留给下期吧。或者做个免费版本?💰
  • gdb 理解 Envoy 对象构造 – 这货色还是专门一篇来写吧。心急的同学能够先看:https://blog.mygraphql.com/zh…

谢谢浏览,记得上次的长文章曾经是去年年三十晚了。一个多月没新料了。学习 bpf / c++ vtable / gdb … / ELF 构造 /…,几次让人想放弃,还好,年纪大,对于我来说不是躺平,而是让我退无可退,保持到当初了。

同时也要多谢互联网精力让我至今未放弃深耕技术这条路。现世间接相干人,对技术的评定有时更多不是技术价值自身,而是对评定人自己的价值(和负价值)。但我置信,地球那么大,总有人只看技术价值,而非那一亩三分地的小山头和小站队。互联网精力,于我是凋谢、平等、主观。

我心愿这是个有温度的技术 Blog。让我用最近拍的一张片,完结本文。

原文:逆向工程与云原生现场剖析 Part2 —— eBPF 跟踪 Istio/Envoy 之启动、监听与线程负载平衡

退出移动版