-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incomplete uploads with HTTP/2 client & server #4948
Comments
@mklinger it's unlikely that your problems are due to #4855, since those failure happens only when the client is sending to the server nasty failures to verify the behavior against the HTTP/2 specification. The two major changes between .28 and .29 are stream creation and channel recycling. Do you have an intermediary between client and server? Can you please give us a server dump (see https://www.eclipse.org/jetty/documentation/current/jetty-dump-tool.html) on both client and server when this happens? Any chance that you can run the test with DEBUG logs enabled? |
@sbordet thank you for the quick response!
Client and server are communicating directly without any (Reverse-)Proxy or similar. TLS termination happens within Jetty using Conscrypt 2.4.0. My integration tests run in local Docker containers using a Docker network.
Gzip is enabled for server responses, but currently not for client requests. The problem only appears for requests, not responses.
I will do that, but it will take some time for me to trigger the dump programmatically at the right moment.
Yes, I already did that. I will try to filter the log for the problematic request/response cycle (as otherwise the log gets huge) and attach it to this ticket. |
If you enabled this via the Jetty When you said "but currently not for client requests" did you mean that you didn't configure Jetty Client for compressing its requests? |
@joakime I was wrong. Gzip compression is not enabled on both sides. I never see any |
I ran the tests with DEBUG logs enabled and here are the condensed log lines for server and client for the request that fails: The request body should have 47682 bytes, but only 16375 bytes end up in the JSON parser. If this doesn't help, I can also provide the full log file (which is > 100 MB) |
@mklinger seems a buffer corruption at TLS level: we pass 4 buffers with 9, 16384, 9 16384 bytes each. Can you try without Conscrypt? I would like to verify whether it's a Conscrypt bug. |
@sbordet Wow, thanks for the information. My first try getting it running without Conscrypt did not succeed. I will try again tomorrow morning and come back to you. Thanks! |
@mklinger also are you serving mixed HTTP/1.1 and HTTP/2 requests? Are you able to try the latest code in the `jetty-9.4.x' branch? We can help you build Jetty from that branch if you need help. |
@sbordet Unfortunately, using a build from the |
|
I tried. The application is a Spring Boot application and I had to find out that current Spring Boot versions force the usage of Conscrypt for Jetty HTTP/2 setup. I browsed the code but did not find a way to change that in a reasonable time.
The application instances under test only do HTTP/2 requests. When double-checking, I found that the test itself uses two Jetty clients, one doing HTTP/2 and another doing HTTP/1.1. These clients are used for requests against the application instances. The test code runs in another process. The actual test failure happens for requests between application instances.
I did that and got the following output when the test environment is shut down. It looks to me like this output is produced by the test process. Application instances don't produce such output. (I did that using the 9.4.29 version)
|
@sbordet: Unfortunately, the problem still persists using a SNAPSHOT build from branch |
@mklinger can you please attach the DEBUG logs for the run with branch Also, did you try without Conscrypt? |
@sbordet here is the debug log. I tried to filter the log for the interesting parts. I hope, I got it right. |
I wasn't able to use something else than Conscrypt on the server side, but I managed to use the SunJSSE provider on the client side. The result was the same, the test failed with the same error. I will dig into Spring Boot's Jetty setup code again to find a way to get the server running without Conscrypt, but I'm not very optimistic about that. |
I added an issue at spring-boot as a result of the discussion on this issue. |
@mklinger the client logs look ok, writing to the server 69078 bytes of PUT content. On the server, we see a first read of 16375 bytes that is added to the
The Then the application is called, it goes through a number of filters and then lands on Spring's This is incorrect but we cannot find any trace in your logs of when the state was moved to EOF and it should be there. Also, client and server times are a little off (the server seems to be few ms behind) - can you please confirm that? The good news is that we don't see anymore the buffer corruption that we were seeing before - these last logs seem to show a different problem. Can you please check what application code is doing the read of just 1 byte? Is it some filter or it's your application? I'm baffled! |
@mklinger what's the status on this? I guess we can close it? |
Hi @mklinger, @sbordet! It seems, that we experience somewhat similar problem with jetty embedded in hadoop hdfs namenode (jetty version 9.4.20.v20190813 as reported by maven). Upload silently fails on server side during upload of file, and debug log shows, that last filled ByteBuffer somehow is not applied to HttpInput, but no errors or warns is produced. Clients behaves itself in two ways - it either tells that all is ok, but file is effectively not uploaded, or shows error like 'Unexpected end of file from server'. But only sometimes, and it is impossible to predict when. I've attached two logs - one with debug info for successful upload and another one (unfortunately only as jpg image) that shows completion of failed upload. Hope it helps. |
@DenisDudinski the image you attached shows that HTTP/1.1 is in use. This issue is about HTTP/2, so please open a new issue. |
This issue has been automatically marked as stale because it has been a |
Jetty version
9.4.29.v20200521
Java version
OpenJDK 11.0.7
OS type/version
Ubuntu 18.04.4
Description
After upgrading to 9.4.29 in a project using HTTP/2 Jetty client and server I experienced problems with request bodies around 30k - 40k of size. It seems that the last chunk (last h2 frame?) is not transferred to the server sometimes.
In my case, this results in invalid JSON received on the server side. The size of the body read from the servlet input stream is smaller than the value given in the Content-Length request header. I made sure that my code around the client invocation produces correct JSON with the correct size. I'm using a byte-array for my JSON payload, so I can rule out I/O problems.
I had a hard time nailing down the source of the problem as I can only reproduce it in a quite big integration test of my application. This test runs 3 application processes that all communicate with each other concurrently and asynchronous. Within the test, a few hundred requests are made and usually one of them fails.
I tried to create a smaller test-case but could not reproduce the problem there. I am even unsure whether the problem is caused on the client or server side, as all traffic is encrypted and the debug log is not very clear about the number of bytes sent over the wire.
Downgrading Jetty client and server to 9.4.28 made the problem disappear.
After having a look at the HTTP/2 related changes between 9.4.28 and 9.4.29 I tried to set the
recycleHttpChannels
flags tofalse
, on both, server and client side, without any effect (my tests where still failing).May this be related to #4855 (comment) ?
I would like to kindly ask you to review the HTTP/2 related changes between 9.4.28 and 9.4.29. If I can provide any further information please let me know.
Thank you.
The text was updated successfully, but these errors were encountered: