Response header overflow leads to buffer corruptions #4936
Comments
nathankooij
changed the title
Signed-off-by: Joakim Erdfelt <[email protected]>
Signed-off-by: Joakim Erdfelt <[email protected]>
|
We've been able to replicate with a variant of your test case. (It proved very useful! Thanks!) We have a general fix working currently, but are reviewing other buffer pool usages for similar issues. |
Signed-off-by: Joakim Erdfelt <[email protected]>
If the response buffer is too large, the header buffer was released but not nulled, then an exception thrown, which again released the not nulled buffer. The buffer thus ends up in the buffer pool twice! Signed-off-by: Greg Wilkins <[email protected]>
|
Thanks for the detailed bug report, test case and initial analysis. Your good report has enabled a quick fix and we'll get a release out asap! |
|
Thanks for the quick turnaround and clear explanation! Looking forward to the release. |
removed old comment Signed-off-by: Greg Wilkins <[email protected]>
* Issue #4936 - Adding LargeHeaderTest to replicate issue Signed-off-by: Joakim Erdfelt <[email protected]> * Issue #4936 - Updating LargeHeaderTest to use ServerConnector Signed-off-by: Joakim Erdfelt <[email protected]> * Issue #4936 - Fail LargeHeaderTest if client detects issues. Signed-off-by: Joakim Erdfelt <[email protected]> * Issue #4936 large response header buffer corruption If the response buffer is too large, the header buffer was released but not nulled, then an exception thrown, which again released the not nulled buffer. The buffer thus ends up in the buffer pool twice! Signed-off-by: Greg Wilkins <[email protected]> * Issue #4936 large response header buffer corruption removed old comment Signed-off-by: Greg Wilkins <[email protected]> Co-authored-by: Joakim Erdfelt <[email protected]>
|
Merged into |
joakime
changed the title
|
We have identified that the bug mentioned here actually has security implications as well. It turned out that corrupt HTTP response buffer unfortunately is sent back in a lot of cases to different clients when the server is under heavy load and ends up sending incorrect responses to different clients. We had a few cases, where due to this particular flaw, clients were able to jump sessions thereby having cross account access. This happens since sometimes the buffer is not completely corrupted and ends up there by sending authentication cookies from one user's response to another user thereby allowing user A to jump in user B's session. We were running version 9.4.28 when this started occuring. As a result we have rolled back to an earlier version where we did not see the bug. I think opening up a CVE for this bug indicating the potential for security implications might be a good idea.. |
|
Good idea. |
This upgrade fixed critical error: [1], that was a follow-up of the previous fix: [2]. Response header overflow leads to buffer corruptions Jetty server always allocates maximum response header size [1] jetty/jetty.project#4936 [2] jetty/jetty.project#4541 Bug: Issue 12846 Change-Id: Ibc47c8b332f433afb6fdee8e78452380996c1dcb
|
A CVE was filed for this issue, CVE-2019-17638, and the issue was resolved in 9.4.30.v20200611. |
|
Why does the linked CVE indicate that "in case of too large response headers, Jetty throws an exception to produce an HTTP 431 error." ?
|
|
@hossman, you are correct. We return a 500 and did so before the CVE and fix as well. |
As the max request header size and response heder size are configurable in airlift, CVE-2019-17638 (https://nvd.nist.gov/vuln/detail/CVE-2019-17638) might affect it. Jetty version update to 9.4.30.v20200611 will fix the issue in lights of CVE-2019-17638 and jetty/jetty.project#4936.
As the max request header size and response heder size are configurable in airlift, CVE-2019-17638 (https://nvd.nist.gov/vuln/detail/CVE-2019-17638) might affect it. Jetty version update to 9.4.30.v20200611 will fix the issue in lights of CVE-2019-17638 and jetty/jetty.project#4936.
As the max request header size and response heder size are configurable in airlift, CVE-2019-17638 (https://nvd.nist.gov/vuln/detail/CVE-2019-17638) might affect it. Jetty version update to 9.4.30.v20200611 will fix the issue in lights of CVE-2019-17638 and jetty/jetty.project#4936.
As the max request header size and response heder size are configurable in airlift, CVE-2019-17638 (https://nvd.nist.gov/vuln/detail/CVE-2019-17638) might affect it. Jetty version update to 9.4.30.v20200611 will fix the issue in lights of CVE-2019-17638 and jetty/jetty.project#4936.
As the max request header size and response heder size are configurable in airlift, CVE-2019-17638 (https://nvd.nist.gov/vuln/detail/CVE-2019-17638) might affect it. Jetty version update to 9.4.30.v20200611 will fix the issue in lights of CVE-2019-17638 and jetty/jetty.project#4936.
|
@gregw @joakime @sbordet I have recently facing similar issue, when multiple request are triggered at once then we see the following issue but request individually process just fine. org.eclipse.jetty.http.BadMessageException: 500: Response header too large I see similar issue was fixed in 9.4.30 version , currently we are on 9.4.38.v20210224. I haven't specified any value for response buffer size and the header in the request doesn't have any information , all the data is passed as body content. Can you please advice. |
|
@ritesh3d then you likely have an honest scenario of response headers that are too big. pointing to a possible bug in your code. Setup a debug break-point here ... And look at what is currently in the |
As the max request header size and response heder size are configurable in airlift, CVE-2019-17638 (https://nvd.nist.gov/vuln/detail/CVE-2019-17638) might affect it. Jetty version update to 9.4.30.v20200611 will fix the issue in lights of CVE-2019-17638 and jetty/jetty.project#4936.
As the max request header size and response heder size are configurable in airlift, CVE-2019-17638 (https://nvd.nist.gov/vuln/detail/CVE-2019-17638) might affect it. Jetty version update to 9.4.30.v20200611 will fix the issue in lights of CVE-2019-17638 and jetty/jetty.project#4936.
Jetty version
9.4.29(but traced it back to9.4.27)Java version
OS type/version
Debian GNU/Linux 10 (buster)Description
We run several Spring Boot 2 applications, originally deployed as WAR files on a standalone Jetty server (
9.4.15). Recently we changed these to use an embedded Jetty server, which included an upgrade to Jetty9.4.28. Subsequently, we intermittently observed errors in one of our applications, serving an admin page using webjars. The logs indicated errors such as:org.eclipse.jetty.server.HttpChannel () : /path-to-a-webjar org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the HTTP method "HTGET" was not included within the whitelist [HEAD, DELETE, POST, GET, OPTIONS, PATCH, PUT]This was a valid request from the client side, but somehow the HTTP method got corrupted.
o.e.j.u.thread.strategy.EatWhatYouKill (): java.lang.IllegalArgumentException: newPosition > limit: (212 > 0)Truncated this one, see below for an example of a stacktrace.
org.eclipse.jetty.http.BadMessageException: 500: Response header too largeo.e.j.u.thread.strategy.EatWhatYouKill () : java.nio.BufferOverflowException: nullThe admin page itself would also become highly unresponsive, and some files would fail to be served. A restart of the application would fix the errors, but it would also resolve itself "eventually". Moreover, the errors' frequency grew with load, but we couldn't reproduce it consistently. This application in particular was affected the most, but we also observed the errors in other applications.
We managed to reproduce the issue by creating an endpoint in which we would explicitly send a response header larger than the configured maximum size (
8KB), after which an application would immediately become unresponsive and all aforementioned errors would start appearing. We confirmed this behavior wasn't present in our previous Jetty version9.4.15, and found it was first introduced in9.4.27. (Perhaps related to #4541?)This might also be the same issue as observed in #4828, but
9.4.29which included the potential fix didn't resolve the issue for us.We have for now resolved this internally by allowing bigger response headers, as we do have a use case for these internally.
I have provided below a test that can be used to trigger the behavior. It consists of sending a (too) large response header, after which we concurrently trigger many requests in order to reproduce the behavior. From my testing concurrency is necessary, which could explain why the error was more frequently observed in our webpage (it serves many webjars separately rather than bundling).
Lastly, from my brief foray into exploring this issue, in one of the cases I noticed that when the header overflow was triggered, which released the
HttpConnection#_headerbuffer (but before any error handling was done), this same buffer was used byHttpParser#quickStart, but its limit was set to 0. These are two different requests, and could explain the HTTP method corruption for instance. So it seems that somehow the buffer cleanup in case of a header overflow is perhaps not correctly handled.If you need any more information, please do let me know.
Example test
Please look at the logs to observe the triggered errors. Further below I have also provided some sample stacktraces that can be reproduced by this test.
Example stacktraces
The text was updated successfully, but these errors were encountered: