TLS interception breaks tunneled http requests and is not activated reliably based on the https request target authority

[JJ-Author]

Describe the bug

• A) CONNECT requests to an http target URL are falsely attempted to be intercepted by interception mode. proxypy ssl exception is triggered and request fails.

• B) do_intercept() does not intercept request target that are supposed to be intercepted when CONNECT request use IP as authority.

To Reproduce

Steps to reproduce the behavior:

1. the issue can be reproduced with running poetry install and then poetry shell and then python proxy-test/custom_proxy.py in the root dir of https://github.com/jj-author/https-interception-proxypy/ or alternatively setting up your own version of proxypy on port 8897 that TLS intercepts requests for example.org but not all other request targets (e.g. not for www.example.org , google.com and all other)
2. run test for problem A and problem B below

A)
run http request (plaintext) command via CONNECT (curl -p option)

curl -p -I -x http://tools.dbpedia.org:8897 http://example.org

HTTP/1.1 200 Connection established 
curl: (52) Empty reply from server 

http messages

CONNECT example.org:80 HTTP/1.1\r
Host: example.org:80\r
User-Agent: curl/7.81.0\r
Proxy-Connection: Keep-Alive\r
\r
< 2024/10/14 14:50:11.129890  length=39 from=0 to=38
HTTP/1.1 200 Connection established\r
\r
> 2024/10/14 14:50:11.164106  length=76 from=112 to=187
HEAD / HTTP/1.1\r
Host: example.org\r
User-Agent: curl/7.81.0\r
Accept: */*\r
\r

proxypy exception

024-10-14 14:50:10,909 - pid:1143139 [D] handler.handle_readables:222 - Client is read ready, receiving...
2024-10-14 14:50:10,909 - pid:1143139 [D] connection.recv:62 - received 112 bytes from client
2024-10-14 14:50:10,909 - pid:1143139 [D] server.connect_upstream:578 - Connecting to upstream example.org:80
2024-10-14 14:50:11,014 - pid:1143139 [D] server.connect_upstream:613 - Connected to upstream example.org:80
Do intercept triggered: {'state': 6, 'type': 1, 'protocol': None, 'host': b'example.org', 'port': 80, 'path': None, 'method': b'CONNECT', 'code': None, 'reason': None, 'version': b'HTTP/1.1', 'total_size': 112, 'buffer': None, 'headers': {b'host': (b'Host', b'example.org:80'), b'user-agent': (b'User-Agent', b'curl/7.81.0'), b'proxy-connection': (b'Proxy-Connection', b'Keep-Alive')}, 'body': None, 'chunk': None, '_url': <proxy.http.url.Url object at 0x7f3f39053a60>, '_is_chunked_encoded': False, '_content_expected': False, '_is_https_tunnel': True}

##### Self.client vars: {'tag': 'client', 'buffer': [<memory at 0x7f3f393004c0>], 'closed': False, '_reusable': False, '_num_buffer': 1, '_conn': <socket.socket fd=107, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 8897), raddr=('127.0.0.1', 56922)>, 'addr': ('127.0.0.1', 56922)}
2024-10-14 14:50:11,127 - pid:1143139 [E] server.wrap_server:802 - SSLError when wrapping client for upstream: example.org
Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/proxy-test-f6SwClI5-py3.10/lib/python3.10/site-packages/proxy/http/proxy/server.py", line 776, in wrap_server
    self.upstream.wrap(
  File "/root/.cache/pypoetry/virtualenvs/proxy-test-f6SwClI5-py3.10/lib/python3.10/site-packages/proxy/core/connection/server.py", line 63, in wrap
    self._conn = ctx.wrap_socket(
  File "/root/.pyenv/versions/3.10.14/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/root/.pyenv/versions/3.10.14/lib/python3.10/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/root/.pyenv/versions/3.10.14/lib/python3.10/ssl.py", line 1375, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1007)
2024-10-14 14:50:11,128 - pid:1143139 [D] tcp_server.handle_readables:212 - Implementation signaled shutdown for client 127.0.0.1:56922
2024-10-14 14:50:11,129 - pid:1143139 [D] tcp_server.handle_readables:218 - Client 127.0.0.1:56922 has pending buffer, will be flushed before shutting down
2024-10-14 14:50:11,129 - pid:1143139 [D] threadless._update_work_events:166 - fd#107 modified for mask#2 by work#107
2024-10-14 14:50:11,129 - pid:1143139 [D] handler.handle_writables:193 - Client is write ready, flushing...
2024-10-14 14:50:11,129 - pid:1143139 [D] tcp_server.handle_writables:173 - Flushing buffer to client 127.0.0.1:56922
2024-10-14 14:50:11,129 - pid:1143139 [D] connection.flush:101 - flushed 39 bytes to client
2024-10-14 14:50:11,129 - pid:1143139 [D] threadless._cleanup:311 - fd#107 unregistered by work#107
2024-10-14 14:50:11,130 - pid:1143139 [I] server.access_log:393 - 127.0.0.1:56922 - CONNECT example.org:80 - 0 bytes - 220.20ms
2024-10-14 14:50:11,130 - pid:1143139 [D] server.on_client_connection_close:384 - Closed server connection, has buffer False
2024-10-14 14:50:11,130 - pid:1143139 [D] handler.shutdown:89 - Closing client connection 127.0.0.1:56922 has buffer False
2024-10-14 14:50:11,130 - pid:1143139 [D] handler.shutdown:99 - Client connection shutdown successful
2024-10-14 14:50:11,130 - pid:1143139 [D] handler.shutdown:113 - Client connection closed

B)

issue CONNECT request that is using IP as authority but uses SNI/servername that is supposed to be intercepted => bug is that original certificate (see issuer) is returned

printf "HEAD / HTTP/1.1\r\nHost: example.org\r\nConnection: close\r\n\r\n" | openssl s_client -servername example.org -connect 93.184.215.14:443 -proxy localhost:8897 2>/dev/null | openssl x509 -noout -issuer -subject
issuer=C=US, O=DigiCert Inc, CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
subject=C=US, ST=California, L=Los Angeles, O=Internet Corporation for Assigned Names and Numbers, CN=www.example.org

http messages

> 2024/10/15 13:53:46.631581  length=68 from=0 to=67                                                                                                                          
CONNECT 93.184.215.14:443 HTTP/1.0\r                                                                                                                                          
Proxy-Connection: Keep-Alive\r                                                                                                                                                
\r                                                                                                                                                                            
< 2024/10/15 13:53:46.744051  length=39 from=0 to=38                                                                                                                          
HTTP/1.1 200 Connection established\r                                                                                                                                         
\r                                                                                                                                                                            
> 2024/10/15 13:53:46.744490  length=322 from=68 to=389                                                                                                                       
....=...9........X.@o                                                                                                                                                         
r.\r~0.A\vw\b#.25?.G.._ _h|".T......[.4,}.....H.E.V.<....<.......,.0.........+./...$.(.k.#.'.g.                                                                               
...9.   ...3.....=.<.5./.................\vexample.org.\v.......
.........................#...........\r.0........\b\a\b\b\b.\b.\b.\b    \b
\b\v\b.\b.\b....................+........-.....3.&.$... X.....1.n..B1A.....|.&....L....]< 2024/10/15 13:53:46.843873  length=99 from=39 to=137
....X...T...!.t..a......e......z..^\a.  ...3. _h|".T......[.4,}.....H.E.V.<.......\f.+.....3..........> 2024/10/15 13:53:46.844423  length=361 from=390 to=750
..........^...Z........X.@o 
r.\r~0.A\vw\b#.25?.G.._ _h|".T......[.4,}.....H.E.V.<....<.......,.0.........+./...$.(.k.#.'.g.
...9.   ...3.....=.<.5./.................\vexample.org.\v.......
.........................#...........\r.0........\b\a\b\b\b.\b.\b.\b    \b
\b\v\b.\b.\b....................+........-.....3.G.E...A.N..\\L......k...=..%.....g.....KK      ...{..Z.7=.pG.3TVZ.
"_A.......< 2024/10/15 13:53:46.945856  length=3722 from=138 to=3859
.............b..-|r,..6../%.,...w\fD.....%.
 _h|".T......[.4,}.....H.E.V.<.......O.+.....3.E...A...3....D{..K.S....r.SV.:W..5.'4....<!......`.K..8<Z.a..O<.D..}..........n......&,..b...F4 .;...\fa........l.hKrLHR.@7$vZ.
t8..L....}4T-|d..Nk.+...u..j....N5....j..%..\b.7|..k......N

Expected behavior

A) I expect the curl request to succeed or more specifically I expect that plaintext http tunnels are not attempted to be TLS-intercepted. In other words proxypy should only try to intercept a connect request if a TLS hello is sent by the client. I think the correct behavior would be for non-tls tunnels either to not trigger do_intercept or at least ignore the return value of it. As a sidenote: I would expect 'request._is_https_tunnel' to be false in this case because that is obviously not true.

B) I expect proxypy to correctly intercept example.org and to see my local certificate issuer for that request proofing interception. On implementation level do_intercept method needs fixing in a way that the decision to intercept is being made based on the actual TLS connection hostname (SNI) and not on the CONNECT target/authority/hostname - which doesn't and won't work.
So in other words expected is that the decision is being (or at least can be) made based on the SNI header if present. For reasons of plugin backward compatibility probably the easiest way to fix it would be to add a connect_tunnel object information to the client variable/object. that tunnel information would carry for now the field tsl_client_sni (that contains the SNI name null if none detected) tls_client_hello_version (version set to null if no tls detected). This would allow that by default there is the "faulty" behavior but new plugins could implement it correctly. Additionally adding the destination port tunnel_dest_port to that tunnel object would make a lot of sense would help to mitigate the port bug of proxypy explained in #1437

I filed both bugs in the same issue because fixing both is achieved by parsing and taking actions based on the TLS client hello.

Note that fixing this would also fix broken websocket requests #1432 (comment) and also another aspect of issue #1437 where there is no easy way to identify whether it is an https or http request

Version information

• OS: Ubuntu 20.04 with dual stack (IPv4 and IPv6)
• clients: curl
• Device: same server
• proxy.py Version 2.4.8

Additional context

compare with output of B) - when connect target-authority and tunneled request-authority/SNI match selective interception works
Please note that there seems another issue about the encoding of the original values lets just hope for now that does not break anything...

printf "HEAD / HTTP/1.1\r\nHost: example.org\r\nConnection: close\r\n\r\n" | openssl s_client -connect example.org:443 -servername example.org -proxy localhost:8894 2>/dev/null | openssl x509 -noout -issuer -subject
issuer=CN=localhost
subject=CN=www.example.org, C=US, ST=California, L=Los Angeles, O=Internet Corporation for Assigned Names and Numbers