Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Certain SSL failures can result in worker thread termination #599

Closed
1 of 3 tasks
J-W-C opened this issue Jan 28, 2023 · 0 comments
Closed
1 of 3 tasks

Certain SSL failures can result in worker thread termination #599

J-W-C opened this issue Jan 28, 2023 · 0 comments
Labels
bug Something is broken triage

Comments

@J-W-C
Copy link

J-W-C commented Jan 28, 2023 •
edited by webknjaz
Loading

❓ I'm submitting a ...

  • 🐞 bug report
  • 🐣 feature request
  • ❓ question about the decisions made in the repository

🐞 Describe the bug. What is the current behavior?

My Internet-facing web server would become unresponsive to HTTPS requests after, I believe, losing enough worker threads to un-handled exceptions coming out of the SSL library.

💡 To Reproduce

Not easily reproducible but logged on my server. Happens when HTTPConnection::communicate() gets an exception from req.parse_request() that causes it to call _conditional_error() from exception handling code, which can result in an un-handled exception if req.simple_response(response) throws an exception other than FatalSSLAlert or NoSSLError.

💡 Expected behavior

Exception handled and logged.

📋 Details

Sample log entries:

Dec 29 22:12:40 Po python3[29548]: [29/Dec/2022:22:12:40] ENGINE socket.error 1
Dec 29 22:12:40 Po python3[29548]: Traceback (most recent call last):
Dec 29 22:12:40 Po python3[29548]: File "/usr/local/lib/python3.7/site-packages/cheroot/server.py", line 1277, in communicate
Dec 29 22:12:40 Po python3[29548]: req.parse_request()
Dec 29 22:12:40 Po python3[29548]: File "/usr/local/lib/python3.7/site-packages/cheroot/server.py", line 710, in parse_request
Dec 29 22:12:40 Po python3[29548]: success = self.read_request_line()
Dec 29 22:12:40 Po python3[29548]: File "/usr/local/lib/python3.7/site-packages/cheroot/server.py", line 751, in read_request_line
Dec 29 22:12:40 Po python3[29548]: request_line = self.rfile.readline()
Dec 29 22:12:40 Po python3[29548]: File "/usr/local/lib/python3.7/site-packages/cheroot/server.py", line 301, in readline
Dec 29 22:12:40 Po python3[29548]: data = self.rfile.readline(256)
Dec 29 22:12:40 Po python3[29548]: File "/usr/lib64/python3.7/_pyio.py", line 512, in readline
Dec 29 22:12:40 Po python3[29548]: b = self.read(nreadahead())
Dec 29 22:12:40 Po python3[29548]: File "/usr/lib64/python3.7/_pyio.py", line 491, in nreadahead
Dec 29 22:12:40 Po python3[29548]: readahead = self.peek(1)
Dec 29 22:12:40 Po python3[29548]: File "/usr/lib64/python3.7/_pyio.py", line 1085, in peek
Dec 29 22:12:40 Po python3[29548]: return self._peek_unlocked(size)
Dec 29 22:12:40 Po python3[29548]: File "/usr/lib64/python3.7/_pyio.py", line 1092, in _peek_unlocked
Dec 29 22:12:40 Po python3[29548]: current = self.raw.read(to_read)
Dec 29 22:12:40 Po python3[29548]: File "/usr/lib64/python3.7/socket.py", line 589, in readinto
Dec 29 22:12:40 Po python3[29548]: return self._sock.recv_into(b)
Dec 29 22:12:40 Po python3[29548]: File "/usr/lib64/python3.7/ssl.py", line 1071, in recv_into
Dec 29 22:12:40 Po python3[29548]: return self.read(nbytes, buffer)
Dec 29 22:12:40 Po python3[29548]: File "/usr/lib64/python3.7/ssl.py", line 929, in read
Dec 29 22:12:40 Po python3[29548]: return self._sslobj.read(len, buffer)
Dec 29 22:12:40 Po python3[29548]: ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:2570)
Dec 29 22:12:40 Po python3[29548]: During handling of the above exception, another exception occurred:
Dec 29 22:12:40 Po python3[29548]: Traceback (most recent call last):
Dec 29 22:12:40 Po python3[29548]: File "/usr/local/lib/python3.7/site-packages/cheroot/server.py", line 1352, in _conditional_error
Dec 29 22:12:40 Po python3[29548]: req.simple_response(response)
Dec 29 22:12:40 Po python3[29548]: File "/usr/local/lib/python3.7/site-packages/cheroot/server.py", line 1115, in simple_response
Dec 29 22:12:40 Po python3[29548]: self.conn.wfile.write(EMPTY.join(buf))
Dec 29 22:12:40 Po python3[29548]: File "/usr/local/lib/python3.7/site-packages/cheroot/makefile.py", line 68, in write
Dec 29 22:12:40 Po python3[29548]: res = super().write(val, *args, **kwargs)
Dec 29 22:12:40 Po python3[29548]: File "/usr/local/lib/python3.7/site-packages/cheroot/makefile.py", line 24, in write
Dec 29 22:12:40 Po python3[29548]: self._flush_unlocked()
Dec 29 22:12:40 Po python3[29548]: File "/usr/local/lib/python3.7/site-packages/cheroot/makefile.py", line 33, in _flush_unlocked
Dec 29 22:12:40 Po python3[29548]: n = self.raw.write(bytes(self._write_buf))
Dec 29 22:12:40 Po python3[29548]: File "/usr/lib64/python3.7/socket.py", line 607, in write
Dec 29 22:12:40 Po python3[29548]: return self._sock.send(b)
Dec 29 22:12:40 Po python3[29548]: File "/usr/lib64/python3.7/ssl.py", line 1003, in send
Dec 29 22:12:40 Po python3[29548]: return self._sslobj.write(data)
Dec 29 22:12:40 Po python3[29548]: ssl.SSLError: [SSL: UNKNOWN_STATE] unknown state (_ssl.c:2423)
Dec 29 22:12:40 Po python3[29548]: 70.93.14.177 - - [29/Dec/2022:22:12:40] "GET / HTTP/1.1" 200 6627 "" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0"

📋 Environment

cheroot 9.0.0
CherryPy 18.8.0
Python 3.7.16
OS: Linux 4.14.301-224.520.amzn2.x86_64
Browser: unknown (bots probing for vulnerabilities?)
@J-W-C J-W-C added bug Something is broken triage labels Jan 28, 2023
webknjaz added a commit that referenced this issue Mar 31, 2024
@webknjaz
Merge branch 'bugfixes/358-unhandled-worker-thread-exc'
9720c31 
A DoS would happen in many situations, including TLS errors and
attempts to close the underlying sockets erroring out.

This patch aims to prevent a situation when the worker threads are
killed by arbitrary exceptions that bubble up to their entry point
layers that aren't handled properly or at all.

PR #649

Fixes #358
Fixes #354

Ref #310
Ref #346
Ref #375
Ref #599
Ref #641

Resolves #365
@J-W-C J-W-C closed this as completed Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something is broken triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@J-W-C