关于istio:云原生现场分析-Part4-eBPF-跟踪-IstioEnvoy-之上下游事件驱动协作下的-HTTP-反向代理流程

注，原文来自 https://blog.mygraphql.com/zh... 。如你看到的转载图片不清，请回到原文。

为何要理解 upstream/downstream 事件驱动合作下的 HTTP 反向代理流程
逆向工程与云原生现场剖析系列介绍
HTTP 反向代理的总流程
- Downstream Read Request 模块合作
- Upstream Write Request 模块合作
- Upstream Read Response 模块合作
- Downstream Write Response 模块合作
BPF 脚本输入
BPF 脚本
结尾

为何要理解 upstream/downstream 事件驱动合作下的 HTTP 反向代理流程

想不到还真保持写到 Part4 了。不论你信不信，每一 part 的“为何”一节，是我感觉最难写的 :) 。如果你是第一次读这个系列，不必放心，每一节都是比拟独立的。

不出意外的话，你之前看到的 Envoy 简介是这样形容 Envoy 的个性的：

C++ 编写，原生底层，没有 GC stop the world，所以性能优异
异步事件驱动，多路复用，完满解决 C10k problem
因为单线程负责多连贯，在大连贯量时，缩小了大量线程的内存开销，和 CPU 上下文切换开销

这些形容，当然有其合理性。但很多远看很漂亮的事物，在微距放大后，可能有很多有意思，有价值的货色。我置信只有剖析得足够粗疏，总可能针对咱们理论运行环境和流量个性对作一些有意义的优化。可能只是批改一个 Envoy / Kernel 的配置，也可能是批改一行 Envoy 的代码。或你的利用的流程相干行为，如每次写 socket 的 buffer 数据包的大小。

而这些，都须要建设在理解实现细节的根底上。除非感觉运气或经验值特地好，能够瞎猜。

逆向工程与云原生现场剖析系列介绍

开始前先做个预报，逆向工程与云原生现场剖析系列（将）包含：

Part 1: eBPF 跟踪 Istio/Envoy 之学步
Part 2: eBPF 跟踪 Istio/Envoy 之启动、监听与线程负载平衡
Part 3: eBPF 跟踪 Istio/Envoy 之 downstream TCP 连贯 accept、TLS 握手与 filter_chain 的抉择
Part 4: eBPF 跟踪 Istio/Envoy 之 upstream/downstream 事件驱动合作下的 HTTP 反向代理流程（本文）
Part 5: eBPF 跟踪 Istio/Envoy 之 L3/4 Network Fitler 互动机制
Part 6: eBPF 跟踪 Istio/Envoy 之 http filter
Part 7: eBPF 跟踪 Istio/Envoy 之 http router
Part 8: eBPF 跟踪 Istio/Envoy 之 cluster、 connection pool 与 outbound load balance

在系列中，我将演示如何让 bpftrace "读" 懂运行期的由 C++ 11 编写成的 envoy 过程中的对象数据。
为免吓跑人，还是老套路，多图少代码，不过有的图有点点简单。程序员大叔开始讲故事了。

HTTP 反向代理的总流程

整体看，Socket 事件驱动的 HTTP 反向代理总流程如下：

图中看出，有4种事件驱动了整个流程。前面几节会一一剖析。

为免一下子进入各个步骤细节而让人迷途，当初先看看所有步骤的总流程：

图：Istio/Envoy 模块合作总图

Downstream Read Request 模块合作

图：Downstream Read-Ready 模块合作

大略阐明一下流程：

upstream socket 可读回调
Http::ConnectionManagerImpl 读取 socket，增量放入 Http1::ConnectionImpl
Http1::ConnectionImpl 调用 nghttp2 增量解释 HTTP 申请
如果 nghttp2 认为曾经残缺读取了 HTTP Request 申请，则调用 Http::ServerConnection::onMessageCompleteBase()
Http::ServerConnection::onMessageCompleteBase() 首先 进行 downstream ReadReady 监听
Http::ServerConnection::onMessageCompleteBase() 调用 Http::FilterManager ，发动 http filter chain 的 decodeHeaders 迭代流程
个别，http filter chain 的最初一个 http filter 是 Router::Filter ，Router::Filter::decodeHeaders() 被调用
Router::Filter::decodeHeaders() 的逻辑就见下图了。

图：Downstream Request Router 模块合作

大略阐明一下流程：

Router::Filter ，Router::Filter::decodeHeaders() 被调用
依据配置的 Router 规定，匹配到 Cluster
如 Cluster 连接池对象不存在，则新建
新建 Envoy::Router::UpstreamRequest 对象。
调用 Envoy::Router::UpstreamRequest::encodeHeaders(bool end_stream) ， encode HTTP header
通过一系列的负载平衡算法，匹配到 upstream 的 host(endpoint)
发现到选定的 upstream host 的连贯有余，则：
1. 关上一新的 socket fd(未连贯)
2. 注册 upstream socket FD 的 WriteReady / Connected 事件。筹备在事件回调时写 upstream request
3. 用 socket fd 发动到 upstream host 的异步连贯申请
关联 downstream 与 upstream fd

Upstream Write Request 模块合作

大略阐明一下流程：

upstream socket write ready 回调
发现是连贯胜利回调，关联 upstream socket 到 ConnectionPool::ActiveClient
upstream socket write ready 回调
发现是连贯可写，写入 upstream HTTP request

Upstream Read Response 模块合作

Downstream Write Response 模块合作

BPF 脚本输入

以上图片，不是只读源码就得出的。是须要通过 bpf 埋点察看：

记录 downstream FD(socket 的 file descriptor)，能够认为是过程中的 socket id
埋点过程级和 kernel 级的相干函数入口和进口，察看其出入参加 stack
关联起 downstream FD 与 upstream FD

其中当然有很多细节，不过我不打算一个个讲了，切实有须要用，想深刻理解的同学能够分割我探讨。下一节的 bpf 代码是最好的细节。

https://github.com/labilezhu/...

#### Downstream Read Request: accept downstream socket， 可见 downstream socket 的 FD=41OS handshaked TCP:01:22:11 4215   wrk:worker_0   172.30.207.129                          58708 172.21.206.232                          15006 0/4096sys_exit_accept4 fd=41        accept4+96        Envoy::Network::IoSocketHandleImpl::accept(sockaddr*, unsigned int*)+82        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+216        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217 ***** elapsed=1621361178: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__110__function6__funcIZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS2_7Network21ListenerFilterMatcherEEERNS2_6Server13Configuration22ListenerFactoryContextEEUlRNSD_21ListenerFilterManagerEE_NS_9allocatorISO_EEFvSN_EEclESN_***** elapsed=1621785490: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS3_7Network21ListenerFilterMatcherEEERNS3_6Server13Configuration22ListenerFactoryContextEEUlRNSE_21ListenerFilterManagerEE_SO_EEEvDpOT_***** elapsed=1630898092: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC1ENSt3__110shared_ptrINS2_6ConfigEEE***** elapsed=1630908037: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC2ENSt3__110shared_ptrINS2_6ConfigEEE***** elapsed=1631036930: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter8onAcceptERNS_7Network23ListenerFilterCallbacksE***** elapsed=1631045047: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter6onReadEv***** elapsed=1631070582: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_recvfrom, fd=41, ret=89        recv+108***** elapsed=1631085065: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter16parseClientHelloEPKvm#### Downstream Read Request: 抉择 network filter chain***** elapsed=1631145371: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterD0Evcomm:wrk:worker_0,tid:5327: Got setFilterChainName=0.0.0.0_8080, lenght=12        Envoy::StreamInfo::StreamInfoImpl::setFilterChainName(absl::string_view)+0        Envoy::Server::ActiveTcpSocket::newConnection()+377        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107#### Downstream Read Request: 监听 upstream FD 事件***** elapsed=1631199672: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)EPOLL_CTL_ADD/MOD EdgeTrigger        epoll_ctl+14        epoll_nochangelist_add+54        evmap_io_add_+421        event_add_nolock_+603        event_add+54        Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362        Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284        Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126        Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026        Envoy::Network::ServerConnectionImpl::ServerConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+107        Envoy::Event::DispatcherImpl::createServerConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&)+70        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+307        Envoy::Server::ActiveTcpSocket::newConnection()+377        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217comm:wrk:worker_0    : setsockopt: level=6, fd=41, optname=1, optval=1, optlen=4. comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl9addFilterENSt3__110shared_ptrINS0_6FilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41         Envoy::Network::FilterManagerImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+0        std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774        Envoy::Server::ActiveTcpSocket::newConnection()+377        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41         Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0        virtual thunk to Envoy::Network::ConnectionImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+62        std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774        Envoy::Server::ActiveTcpSocket::newConnection()+377        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41         Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0        std::__1::__function::__func<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7, std::__1::allocator<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+353        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774        Envoy::Server::ActiveTcpSocket::newConnection()+377        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41         Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774        Envoy::Server::ActiveTcpSocket::newConnection()+377        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 ***** elapsed=1631443267: tid=5327,comm=wrk:worker_0: END:EventFired******* WAKE-ROUND:END Summary ************ elapsed=1631456738: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=31393726, tid2epollNrFdReady=1*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=100 ***************************#### Downstream Read Request: 读 HTTP Request***** elapsed=1631475016: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26libevent: EV_READlibevent: EV_WRITElibevent: EV_ET***** elapsed=1631513821: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=89        readv+77        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136        Envoy::Network::ConnectionImpl::onReadReady()+753        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217***** elapsed=1631527775: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=-11        readv+77        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136        Envoy::Network::ConnectionImpl::onReadReady()+753        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl6onReadEv,FilterManagerImpl.this=0x559f98d38758,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41         Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 ***** elapsed=1631671079: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005#### Downstream Read Request: downstream 的 HTTP Reuest 曾经齐全读取（onMessageComplete()）。进行监听 downstream 的 ReadReady 事件***** elapsed=1631678652: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80002004EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)EPOLL_CTL_ADD/MOD EdgeTrigger        epoll_ctl+14        epoll_nochangelist_add+54        evmap_io_add_+421        event_add_nolock_+603        event_add+54        Envoy::Network::ConnectionImpl::readDisable(bool)+1077        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+86        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31        http_parser_execute+7959        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303        Envoy::Network::ConnectionImpl::onReadReady()+1622        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 #### Downstream Read Request: 监听新建设的 upstream socket 的 FD=44 ， 这里 socket 还处于未连贯状态。当初监听 upstream socket 的 所有事件。以备前面异步连贯胜利时收到回调***** elapsed=1632388633: tid=5327,comm=wrk:worker_0: register upstream event trigger:sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=44, events=0x80000005EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)EPOLL_CTL_ADD/MOD EdgeTrigger        epoll_ctl+14        epoll_nochangelist_add+54        evmap_io_add_+421        event_add_nolock_+603        event_add+54        Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362        Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284        Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126        Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026        Envoy::Network::ClientConnectionImpl::ClientConnectionImpl(Envoy::Event::Dispatcher&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+283        Envoy::Event::DispatcherImpl::createClientConnection(std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+75        Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterInfo const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, Envoy::Network::TransportSocketFactory&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>)+561        Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+126        non-virtual thunk to Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+21        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+112        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31        http_parser_execute+7959        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303        Envoy::Network::ConnectionImpl::onReadReady()+1622        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98f08478,fd=41         Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0        Envoy::Http::CodecClient::CodecClient(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&)+357        Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+70        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94        Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35        Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31        http_parser_execute+7959        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303        Envoy::Network::ConnectionImpl::onReadReady()+1622        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217comm:wrk:worker_0    : setsockopt: level=6, fd=44, optname=1, optval=1, optlen=4. #### Downstream Read Request: 发现异步 upstream 连贯connecting to 8080:01:23:36 4215     wrk:worker_0     127.0.0.6                               36383  172.21.206.232                          8080          connect+75        Envoy::Network::IoSocketHandleImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+104        Envoy::Network::SocketImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+64        Envoy::Network::ClientConnectionImpl::connect()+650        Envoy::Http::CodecClient::connect()+543        Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+694        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94        Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35        Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31        http_parser_execute+7959        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303        Envoy::Network::ConnectionImpl::onReadReady()+1622        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 ***** elapsed=1632608064: tid=5327,comm=wrk:worker_0: END:EventFired******* WAKE-ROUND:END Summary ************ elapsed=1632618881: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1145219, tid2epollNrFdReady=1*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=92 ******************************** elapsed=1632635526: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24libevent: EV_WRITElibevent: EV_ET***** elapsed=1632654517: tid=5327,comm=wrk:worker_0: END:EventFired#### Upstream Write Request: 异步 upstream 连贯胜利，回调。***** elapsed=1632659321: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()FileEventImpl*=0x559f98eb6540, fd=44, events=0x24libevent: EV_WRITElibevent: EV_ETcomm:wrk:worker_0,tid:5327: ConnPoolImplBase*attachStreamToClient: uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy14ConnectionPool16ConnPoolImplBase20attachStreamToClientERNS0_12ActiveClientERNS0_13AttachContextE         Envoy::ConnectionPool::ConnPoolImplBase::attachStreamToClient(Envoy::ConnectionPool::ActiveClient&, Envoy::ConnectionPool::AttachContext&)+0        Envoy::ConnectionPool::ConnPoolImplBase::onConnectionEvent(Envoy::ConnectionPool::ActiveClient&, absl::string_view, Envoy::Network::ConnectionEvent)+497        Envoy::Network::ConnectionImplBase::raiseConnectionEvent(Envoy::Network::ConnectionEvent)+59        Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+511        Envoy::Network::ConnectionImpl::onWriteReady()+1648        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217#### Upstream Write Request: 向 upstream socket 写 HTTP Request***** elapsed=1632741863: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=44, ret=262        writev+77        Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263        Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107        Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121        Envoy::Network::ConnectionImpl::onWriteReady()+1876        Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+529        Envoy::Network::ConnectionImpl::onWriteReady()+1648        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217***** elapsed=1632756069: tid=5327,comm=wrk:worker_0: END:EventFired******* WAKE-ROUND:END Summary ************ elapsed=1632764448: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=129648, tid2epollNrFdReady=2*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=60 ***************************#### Upstream Read Response: upstream socket ReadReady 回调***** elapsed=1637821708: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()FileEventImpl*=0x559f98eb6540, fd=44, events=0x26libevent: EV_READlibevent: EV_WRITElibevent: EV_ET#### Upstream Read Response: 读 upstream HTTP Response***** elapsed=1637898855: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=4511        readv+77        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136        Envoy::Network::ConnectionImpl::onReadReady()+753        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217***** elapsed=1637915998: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=-11        readv+77        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136        Envoy::Network::ConnectionImpl::onReadReady()+753        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217***** elapsed=1638220184: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80002004#### Upstream Read Response: 曾经残缺读取 upstream HTTP Response，开始监听 downstream 的 WriteReady 事件，筹备写 downstream HTTP Response***** elapsed=1638231319: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)EPOLL_CTL_ADD/MOD EdgeTrigger        epoll_ctl+14        epoll_nochangelist_add+54        evmap_io_add_+421        event_add_nolock_+603        event_add+54        Envoy::Network::ConnectionImpl::readDisable(bool)+938        Envoy::Http::Http1::StreamEncoderImpl::~StreamEncoderImpl()+112        non-virtual thunk to Envoy::Http::Http1::ServerConnectionImpl::onEncodeComplete()+54        Envoy::Http::Http1::StreamEncoderImpl::endEncode()+166        Envoy::Http::Http1::StreamEncoderImpl::encodeData(Envoy::Buffer::Instance&, bool)+340        Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeData(Envoy::Buffer::Instance&, bool)+679        Envoy::Http::FilterManager::encodeData(Envoy::Http::ActiveStreamEncoderFilter*, Envoy::Buffer::Instance&, bool, Envoy::Http::FilterManager::FilterIterationStartState)+2138        Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool)+230        Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59        Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59        Envoy::Http::Http1::ClientConnectionImpl::onMessageCompleteBase()+619        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31        http_parser_execute+7705        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151        Envoy::Http::Http1::ClientConnectionImpl::dispatch(Envoy::Buffer::Instance&)+29        Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&)+48        Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool)+21        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303        Envoy::Network::ConnectionImpl::onReadReady()+1622        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92***** elapsed=1638898463: tid=5327,comm=wrk:worker_0: END:EventFired******* WAKE-ROUND:END Summary ************ elapsed=1638934709: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1123611, tid2epollNrFdReady=1*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56 ******************************** elapsed=1638955421: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()FileEventImpl*=0x559f98eb6540, fd=44, events=0x24libevent: EV_WRITElibevent: EV_ET***** elapsed=1638975366: tid=5327,comm=wrk:worker_0: END:EventFired#### Downstream Write Response: downstream 的 WriteReady 事件回调。***** elapsed=1638980295: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24libevent: EV_WRITElibevent: EV_ET#### Downstream Write Response: 写 downstream HTTP Response***** elapsed=1639253752: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=41, ret=4631        writev+77        Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263        Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107        Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121        Envoy::Network::ConnectionImpl::onWriteReady()+1876        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217***** elapsed=1639293010: tid=5327,comm=wrk:worker_0: END:EventFired******* WAKE-ROUND:END Summary ************ elapsed=1639308346: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=354486, tid2epollNrFdReady=2*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56 ******************************** elapsed=1643785802: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26libevent: EV_READlibevent: EV_WRITElibevent: EV_ET***** elapsed=1643855427: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=0        readv+77        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136        Envoy::Network::ConnectionImpl::onReadReady()+753        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92        0x7fffffffe000        event_base_loop+1953        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19        start_thread+217***** elapsed=1643880789: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005sys_enter_close fd=41delete fd2filterManagerImpl, fd=41delete fd2filterManagerImpl=0x559f98d38758delete fd2filterManagerImpl, fd=41delete fd2filterManagerImpl=0x559f98996380***** elapsed=1644006858: tid=5327,comm=wrk:worker_0: END:EventFired******* WAKE-ROUND:END Summary ************ elapsed=1644056452: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=289169, tid2epollNrFdReady=1*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=52 ***************************^C@epoll_ctl_op[1]: EPOLL_CTL_ADD@epoll_ctl_op[2]: EPOLL_CTL_DEL@epoll_ctl_op[3]: EPOLL_CTL_MOD@fd2sockopt[44, 6, 1, 1]: 1@fd2sockopt[41, 6, 1, 1]: 1@filterManagerImpl2fd[94143954060408]: 41@filterManagerImpl2fd[94143950909952]: 41

BPF 脚本

https://github.com/labilezhu/...

#!usr/bin/bpftrace/*IMPORT-ENV: $ENVOY_PIDargs: $1=ENVOY_PID*//*#export PODID=`sudo crictl  pods | grep fortio-server | egrep '\bReady' | awk '{print $1}'`#export CONTAINERID=`sudo crictl ps | grep $PODID | grep istio-proxy |  awk '{print $1}'`kubectl -n idm-mark exec -it fortio-server -c  istio-proxy  -- bash -c 'ls -l /proc/`pgrep envoy`/ns/pid'lrwxrwxrwx 1 istio-proxy istio-proxy 0 Mar 31 09:18 /proc/15/ns/pid -> 'pid:[4026532799]'pgrep envoy | tee /dev/fd/2 | xargs -L 1 -I '{}' sudo ls -l /proc/'{}'/ns export PID=4215export SCRIPT_HOME=`pwd`export bpftrace_image=cndt-bcc-ubdocker run -it --rm --init  --privileged --name bpftrace -h bpftrace \    --pid host \    --net host \    -e SCRIPT_HOME=$SCRIPT_HOME \    -e PID=$PID \    -e ENVOY_PID=$PID \    -e BT=trace-envoy-filter-router.bt \    -v /etc/localtime:/etc/localtime:ro \    -v /sys:/sys:rw \    -v /usr/src:/usr/src:rw \    -v /lib/modules:/lib/modules:ro \    -v ${SCRIPT_HOME}:${SCRIPT_HOME}:rw \    $bpftrace_image \    ${SCRIPT_HOME}/warp-bt.sh $PID*//*1. BPF Map 阐明:1.1. range tid map- @watchedWakeRound[tid]=tid       - sys_exit_accept4       - FileEventImpl*assignEvents*- @fdFired[tid]=$fd;       - uprobe: *FileEventImpl*assignEvents*       - uretprobe: *FileEventImpl*assignEvents*1.2. fd map- @fd2sockpair[$fd]=@sockpair[tid]       - sys_exit_accept4       - sys_enter_close- @fd2sockopt[$fd, $level, $optname, $optval_int] = 1       - sys_enter_setsockopt- @filterManagerImpl2fd[arg0] = @fdFired[tid] // filterManagerImpl to downstream fd- @fd2filterManagerImpl[$fd] = arg0 - @fd2filterManagerImpl2[$fd] = arg0- @downFD2upFD[$downFD] = $fd; //downstream fd to upstream fd       - sys_enter_epoll_ctl- @upFD[$fd] = $fd; //upstream fd list       - sys_enter_epoll_ctl1.3. temp tid map- @sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);- @sockpair_exist[tid]=1- @watchedIo[tid]=fd*/#include <linux/in.h>#include <linux/in6.h>#include <linux/socket.h>#include <net/sock.h>// #include <sys/epoll.h>// The event argument describes the object linked to the file descriptor fd.  The struct epoll_event is defined as:// struct epoll_data_t {//     int          fd;// };// struct epoll_event {//     //a bit mask composed by ORing together zero or more of the following available event types//     uint32_t     events;      /* Epoll events *///     struct epoll_data_t data;        /* User data variable */// };BEGIN{    printf("Tracing Envoy. Hit Ctrl-C to end.\n");       printf("#define EPOLL_CTL_ADD 1\n #define EPOLL_CTL_DEL 2\n #define EPOLL_CTL_MOD 3\n");       @epoll_ctl_op[1]="EPOLL_CTL_ADD";       @epoll_ctl_op[2]="EPOLL_CTL_DEL";       @epoll_ctl_op[3]="EPOLL_CTL_MOD";}/*获取 accept 连贯时的本地和对端地址。并标记本轮 wakeup 须要监控 */kretprobe:inet_csk_accept/pid==$1 /{    $sk = (struct sock *)retval;    $inet_family = $sk->__sk_common.skc_family;    if ($inet_family == AF_INET || $inet_family == AF_INET6) {        // initialize variable type:        $daddr = ntop(0);        $saddr = ntop(0);        if ($inet_family == AF_INET) {            $daddr = ntop($sk->__sk_common.skc_daddr);            $saddr = ntop($sk->__sk_common.skc_rcv_saddr);        } else {            printf("not support IPv6.\n");            return;        }                $lport = $sk->__sk_common.skc_num;              // printf("accept(), port=%d\n", $lport);              //only watch listen port 15006        if( 15006 != $lport ) {             return;        }        $dport = $sk->__sk_common.skc_dport;        $qlen  = $sk->sk_ack_backlog;        $qmax  = $sk->sk_max_ack_backlog;        // Destination port is big endian, it must be flipped        $dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);              printf("OS handshaked TCP:\n");        time("%H:%M:%S ");        printf("%-6d %-14s ", pid, comm);        printf("%-39s %-5d %-39s %-5d ", $daddr, $dport, $saddr,            $lport);        printf("%d/%d\n", $qlen, $qmax);        @sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);        @sockpair_exist[tid]=1;              @watchedWakeRound[tid]=tid;    }}/** * 获取新连贯的 FD 。并标记本轮 libevent 回调须要监控 */tracepoint:syscalls:sys_exit_accept4/pid==$1 && @sockpair_exist[tid] /{    $fd = args->ret;    if( $fd < 0 ) {        return;    }    printf("sys_exit_accept4 fd=%d\n", $fd);    @fd2sockpair[$fd]=@sockpair[tid];       @fdFired[tid]=$fd;    delete(@sockpair[tid]);    delete(@sockpair_exist[tid]);       printf("%s \n", ustack());}/** * 记录 FD 的 sockopt */tracepoint:syscalls:sys_enter_setsockopt/pid==$1/{       // socket opts: https://elixir.bootlin.com/linux/v5.16.3/source/include/uapi/linux/tcp.h#L92            $level = args->level;       $fd = args->fd;       if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {              $optname = args->optname;              $optval = args->optval;              $optval_int = *$optval;              $optlen = args->optlen;              // printf("\n########## setsockopt() ##########\n");              printf("comm:%-16s: setsockopt: level=%d, fd=%d, optname=%d, optval=%d, optlen=%d. \n", comm, $level, $fd, $optname, $optval_int, $optlen);              @fd2sockopt[$fd, $level, $optname, $optval_int] = 1;       }}/*记录 epoll 监听的 FDcat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_ctl/format */tracepoint:syscalls:sys_enter_epoll_ctl/pid==$1/{       $fd=args->fd;       $epollEvents = (uint32) (args->event->events);       $op = args->op;       if( @fd2sockpair[$fd].0 ) {              printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n",                      elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);              if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event                     if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {                            printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");                     }                     if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {                            printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");                     }                     if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {                            printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");                     }                     printf("%s\n", ustack(30));              }                     } else {              $downFD = @fdFired[tid];              if( $downFD ) {//in downstream fd event callback, register upstream event trigger                     @upFD[$fd] = $fd;                     @downFD2upFD[$downFD] = $fd;                     printf("***** elapsed=%d: tid=%d,comm=%s: register upstream event trigger:sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n",                             elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);                     if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event                            if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {                                   printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");                            }                            if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {                                   printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");                            }                            if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {                                   printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");                            }                            printf("%s\n", ustack(60));                     }                                   }       }}/*完结本轮的 wakeRound/runnableRound，并期待下一轮cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_wait/format */tracepoint:syscalls:sys_enter_epoll_wait/pid==$1/{       if( @watchedWakeRound[tid] ) {              $runnableStartTime=@tid2Waketime[tid];              if( $runnableStartTime ) {                     $runableDuaration = elapsed - $runnableStartTime;                     printf("\n******* WAKE-ROUND:END Summary *******\n");                     printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait, runableDuaration=%d, tid2epollNrFdReady=%d\n",                             elapsed, tid, comm, $runableDuaration, @tid2epollNrFdReady[tid]);                     $tid_last_epoll_wait_args = @last_epoll_wait_args[tid];                     if( $tid_last_epoll_wait_args.0  ) {                            printf("*** last_epoll_wait_args: epfd=%d, events=%d, maxevents=%d, timeout=%d \n",                                    $tid_last_epoll_wait_args.0, $tid_last_epoll_wait_args.1, $tid_last_epoll_wait_args.2, $tid_last_epoll_wait_args.3);                     }                     printf("***************************\n\n");              }       }       delete(@tid2Waketime[tid]);       delete(@watchedWakeRound[tid]);       delete(@tid2epollNrFdReady[tid]);       //read in next sys_enter_epoll_wait       @last_epoll_wait_args[tid]=(args->epfd, args->events, args->maxevents, args->timeout);}/*开始本轮 wakeRound/runnableRoundcat /sys/kernel/debug/tracing/events/syscalls/sys_exit_epoll_wait/format */tracepoint:syscalls:sys_exit_epoll_wait/pid==$1/{       // printf("\n***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait\n", elapsed, tid, comm);       // printf("epfd: 0x%08lx, events: 0x%08lx, maxevents: 0x%08lx, timeout: 0x%08lx \n",        //        ((args->epfd)), ((args->events)), ((args->maxevents)), ((args->timeout)) );       @tid2Waketime[tid]=elapsed;       @tid2epollNrFdReady[tid]=args->ret;}/*记录本轮 epoll 事件触发的 FD 级回调开始Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int fd, short events, void* fileEventImplThis)  !!!NOT!!: FileEventImpl::assignEvents()C++11 Lambda expressions:event_assign( &raw_event_, base, fd_, xxx, [](evutil_socket_t, short what, void* arg)-> void {}, this );event_assign(): https://libevent.org/doc/event_8h.html#a3e49a8172e00ae82959dfe64684eda11       event_assign    (    struct event *     ev,              struct event_base *     base,              evutil_socket_t     fd,              short     events,              event_callback_fn     callback,              void *     callback_arg        )           https://libevent.org/doc/event_8h.html#aed2307f3d9b38e07cc10c2607322d758       typedef void(* event_callback_fn) (evutil_socket_t, short, void *)              fd    An fd or signal              events    One or more EV_* flags              arg    A user-supplied argument.*/uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*/pid == $1/ {       $fd = arg0;       if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {              $libevent_events = arg1;              printf("\n***** elapsed=%d: tid=%d,comm=%s: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()\n", elapsed, tid, comm);              printf("FileEventImpl*=%p, fd=%d, events=0x%x\n",arg2, $fd, $libevent_events);              if( $libevent_events & (uint16)0x01 /*EV_TIMEOUT*/ ) {                     printf("libevent: EV_TIMEOUT\n");              }              if( $libevent_events & (uint16)0x02 /*EV_TIMEOUT*/ ) {                     printf("libevent: EV_READ\n");              }              if( $libevent_events & (uint16)0x04 /*EV_TIMEOUT*/ ) {                     printf("libevent: EV_WRITE\n");              }              if( $libevent_events & (uint16)0x20 /*EV_TIMEOUT*/ ) {                     printf("libevent: EV_ET\n");              }              if( $libevent_events & (uint16)0x80 /*EV_TIMEOUT*/ ) {                     printf("libevent: EV_CLOSED\n");              }                            printf("%s\n", kstack);              @fdFired[tid]=$fd;              @watchedWakeRound[tid]=tid;       }}/*记录本轮 epoll 事件触发的 FD 级回调完结*/uretprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*/pid == $1 && @fdFired[tid] / {       printf("\n***** elapsed=%d: tid=%d,comm=%s: END:EventFired\n", elapsed, tid, comm);       delete(@fdFired[tid]);}/*埋点 socket 读*/ tracepoint:syscalls:sys_enter_read,tracepoint:syscalls:sys_enter_readv,tracepoint:syscalls:sys_enter_recv*/pid == $1 && (@fd2sockpair[args->fd].0 || @upFD[args->fd] ) / {       @watchedIo[tid]=args->fd;}tracepoint:syscalls:sys_exit_read,tracepoint:syscalls:sys_exit_readv,tracepoint:syscalls:sys_exit_recv*/pid == $1 && @watchedIo[tid]/ {       printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_read, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid],args->ret);       delete(@watchedIo[tid]);       printf("%s\n", ustack(20));}/*埋点 socket 写*/tracepoint:syscalls:sys_enter_write,tracepoint:syscalls:sys_enter_writev/pid == $1 && ( @fd2sockpair[args->fd].0 || @upFD[args->fd] ) / {       @watchedIo[tid]=args->fd;}tracepoint:syscalls:sys_exit_write,tracepoint:syscalls:sys_exit_writev/pid == $1 && @watchedIo[tid]/ {       printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_write, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid], args->ret);       delete(@watchedIo[tid]);       printf("%s\n", ustack(20));}/*打印 TlsInspector 的所有函数调用*/uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*TlsInspector*/pid == $1 && @fdFired[tid]/ {       printf("\n***** elapsed=%d: tid=%d,comm=%s: TlsInspector*, probe=%s\n", elapsed, tid, comm, probe);}/*打印连贯确认协后的函数调用*/uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnectionSocketImpl*setRequestedApplicationProtocols*/pid == $1 && @fdFired[tid]/ {       printf("\n***** elapsed=%d: tid=%d,comm=%s: ConnectionSocketImpl::setRequestedApplicationProtocols\n", elapsed, tid, comm);       // printf("%s", ustack);}//SslSocket::SslSocketuprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocketC2ENSt3__110shared_ptrINS_3Ssl7ContextEEENS2_12InitialStateERKNS5_IKNS_7Network22TransportSocketOptionsEEENS4_8functionIFNS5_INS6_10HandshakerEEENS4_10unique_ptrI6ssl_stN4bssl8internal7DeleterISK_EEEEiPNS6_18HandshakeCallbacksEEEE/pid == $1 && @fdFired[tid]/ {       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);       printf("%s\n", ustack(10));}// setTransportSocketCallbacksuprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket27setTransportSocketCallbacksERNS_7Network24TransportSocketCallbacksE/pid == $1 && @fdFired[tid]/ {       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);       printf("%s\n", ustack(10));}// SslSocket::doReaduprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket6doReadERNS_6Buffer8InstanceE/pid == $1 && @fdFired[tid]/ {       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);       printf("%s\n", ustack(10));}// SslSocket::doWriteuprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket7doWriteERNS_6Buffer8InstanceEb/pid == $1 && @fdFired[tid]/ {       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);       printf("%s\n", ustack(10));}// SslSocket::onSuccessuprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_N5Envoy10Extensions16TransportSockets3Tls9SslSocket9onSuccessEP6ssl_st/pid == $1 && @fdFired[tid]/ {       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);       printf("%s\n", ustack(10));}// SslSocket::connectionuprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv/pid == $1 && @fdFired[tid]/ {       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);       printf("%s\n", ustack(10));}// SslSocket::connectionuprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv/pid == $1 && @fdFired[tid]/ {       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);       printf("%s\n", ustack(10));}/*打印匹配到的 Network Fitler Chain 名字void setFilterChainName(absl::string_view filter_chain_name)*/uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10StreamInfo14StreamInfoImpl18setFilterChainNameEN4absl11string_viewE /pid == $1 && @fdFired[tid]/ { /**[Assembly 2: Calling convention](https://cs61.seas.harvard.edu/site/2018/Asm2/)1. A structure argument that fits in a single machine word (64 bits/8 bytes) is passed in a single register.       Example: `struct small { char a1, a2; }`2. A structure that fits in two to four machine words (16–32 bytes) is passed in sequential registers, as if it were multiple arguments.       Example: `struct medium { long a1, a2; }`*/    $filterName = str(reg("si"));    $filterNameLength = reg("dx");    // printf("tid:%d: Got setFilterChainName=%s, lenght=%d\n %s \n", tid, $filterName, $filterNameLength, ustack );     printf("comm:%s,tid:%d: Got setFilterChainName=%s, lenght=%d\n", comm, tid, $filterName, $filterNameLength );     if( $filterNameLength > 0 ) {        printf("%s\n", ustack(3));    }}/*埋点 FilterManagerImpl::add*Filter()*/uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*add*Filter*,uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*FilterManagerImpl*/pid == $1 && @fdFired[tid]/ {        if( @upFD[@fdFired[tid]] ) {              return;       }       $fd = @fdFired[tid];       @filterManagerImpl2fd[arg0] = $fd;       if( @fd2filterManagerImpl[$fd] ) {              @fd2filterManagerImpl2[$fd] = arg0;       } else {              @fd2filterManagerImpl[$fd] = arg0;       }       printf("comm:%s,tid:%d: FilterManagerImpl.add*Filter|FilterManagerImpl::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd);       printf("%s\n", ustack(40));}/*埋点 FilterManagerImpl::on*()*/uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*on*/pid == $1 && @fdFired[tid]/ {        if( @upFD[@fdFired[tid]] ) {              return;       }       $fd = @fdFired[tid];       @filterManagerImpl2fd[arg0] = $fd;       if( @fd2filterManagerImpl[$fd] ) {              @fd2filterManagerImpl2[$fd] = arg0;       } else {              @fd2filterManagerImpl[$fd] = arg0;       }       printf("comm:%s,tid:%d: FilterManagerImpl.on*::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd); }/*埋点 ConnPoolImplBase::attachStreamToClient()*/uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnPoolImplBase*attachStreamToClient*/pid == $1 && @fdFired[tid]/ {        printf("comm:%s,tid:%d: ConnPoolImplBase*attachStreamToClient: %s \n", comm, tid, probe);        printf("%s\n", ustack(40));}/*埋点 socket connect upstream*/kprobe:tcp_connect/pid == $1/{       $sk = ((struct sock *)arg0);       $inet_family = $sk->__sk_common.skc_family;       if ($inet_family == AF_INET || $inet_family == AF_INET6)       {              if ($inet_family == AF_INET)              {                     $daddr = ntop($sk->__sk_common.skc_daddr);                     $saddr = ntop($sk->__sk_common.skc_rcv_saddr);              }              else              {                     $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);                     $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);              }              $lport = $sk->__sk_common.skc_num;              $dport = $sk->__sk_common.skc_dport;              // Destination port is big endian, it must be flipped              $dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);              // only watch listen port 8080              if (8080 != $dport)              {                     return;              }              printf("connecting to 8080:\n");              time("%H:%M:%S ");              printf("%-8d %-16s ", pid, comm);              printf("%-39s %-6d %-39s %-6d\n", $saddr, $lport, $daddr, $dport);              printf("%s\n", ustack(40));       }}/*清理敞开的 FD 相干的 Mapcat /sys/kernel/debug/tracing/events/syscalls/sys_enter_close/format*/tracepoint:syscalls:sys_enter_close/pid==$1/{    $fd = args->fd;    if( $fd < 0 ) {        return;    }       delete(@upFD[$fd]);       if( @fd2sockpair[$fd].0 ) {              printf("sys_enter_close fd=%d\n", $fd);       }       else {              return;       }    delete(@fd2sockpair[$fd]);       /////       $filterManagerImpl = @fd2filterManagerImpl[$fd];       if( $filterManagerImpl ) {              printf("delete fd2filterManagerImpl, fd=%d\n", $fd );              delete( @fd2filterManagerImpl[$fd] );       }       if( @filterManagerImpl2fd[$filterManagerImpl] ) {              printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );              delete(@filterManagerImpl2fd[$filterManagerImpl]);       }       $filterManagerImpl = @fd2filterManagerImpl2[$fd];       if( $filterManagerImpl ) {              printf("delete fd2filterManagerImpl, fd=%d\n", $fd );              delete( @fd2filterManagerImpl2[$fd] );       }       if( @filterManagerImpl2fd[$filterManagerImpl] ) {              printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );              delete(@filterManagerImpl2fd[$filterManagerImpl]);       }       delete(@upFD[@downFD2upFD[$fd]]);       delete(@downFD2upFD[$fd]);}END{       clear(@last_epoll_wait_args);       clear(@tid2Waketime);       clear(@tid2epollNrFdReady);}

结尾

这 part，从 socket 事件驱动的角度，理解了 envoy 作为反向代理的次要流程。我本人感觉是学习到货色的，你呢？
写博客的初憧是记录本人。想不到也能通过博客意识到很多气味相投，对技术认真之人（不能叫同志？）。谢谢你们，我能力保持写下去，也认为这样的写作有肯定价值！

这是几年前拍的老照片了，共勉。

为何要理解 upstream/downstream 事件驱动合作下的 HTTP 反向代理流程

逆向工程与云原生现场剖析 系列介绍

HTTP 反向代理的总流程

Downstream Read Request 模块合作

Upstream Write Request 模块合作

Upstream Read Response 模块合作

Downstream Write Response 模块合作

BPF 脚本输入

BPF 脚本

结尾

逆向工程与云原生现场剖析系列介绍