Avoid content if original request has no content and avoid Transfer-Encoding: chunked if Content-Length is known #1971
Conversation
Co-authored-by: Raman Maksimchuk <[email protected]>
Co-authored-by: Raman Maksimchuk <[email protected]>
…ithub.com/ThreeMammals/Ocelot into 1844-more-open-customization-of-polly-use
…-customization-of-polly-use ThreeMammals#1844 More open customization of Polly use
…ncoding: chunked if Content-Length is known
|
@ggnaegi Please, review! |
|
@alexreinert Hi Alexander! And thank you for the bug reporting! In my opinion, this user scenario is quite complex, but it definitely needs to be tested. |
raman-m
added
Routing
|
It is not just the iis. A GET request should not have content, but the Transfer-Encoding header indicates, that there is content (even if it is just zero bytes). For a DELETE request there must be no content, but the issue will send the Transfer-Encoding header there, too. So this could be also an issue with some WAF, which enforce the RFC rules. |
|
@alexreinert Yes, unfortunately the waf we are using didn't complain about this and a colleague of us has put the feature on production for a good month without any issues. But it's probably a case for a hot fix, you're right. Could you maybe add a few test cases or give me the permissions on your branch? Thanks. |
TomPallister
force-pushed
the
develop
branch
from
f88bd51 to
108bede
Compare
|
I try to add some test cases tomorrow. |
|
@TomPallister You changed my fork, was this intended? |
* Added tests for mapping requests
|
I added some tests. The unit tests can not cover the whole story as the Transfer-Encoding is not applied until real sending the content, therefor I added some acceptence tests to cover it. |
@alexreinert Ok, I will review the code. Are you sure that returning null when no Content isn't sufficent? It would do the trick, and .net wouldn't add the content headers. You have made some changes that enforce the content size in the new HttpContent class where we are using the Headers property. We could think that this value is never set, but that's not the case. I can understand that, then you will not have TryComputeLength returning false when size is known and the content-encoding: "chunked" header will not be set. And finally you're adding the content-encoding to the the unsupported headers. Could you explain why? Thanks |
This is exactly what I do: I return null, if there is no content in the incoming request. But I changed the detection of content. Request.Body will be not null, even if there is no content. So I used as detection what is stated in RFC 2616 (HTTP 1.1) as detection mechanism.
In my tests, the content will be chunked and the Transfer-Encoding header is added, if TryComputeLength return false. In that case, the Content-Length header will be removed, too. (Having both Content-Length and Transfer-Encoding headers is strictly forbidden and .net ensures that) Of course,it would be possible to just send the content chunked without Content-Length and this would work in most cases, but some backends could respond with status "411 content length required".
No, I added Transfer-Encoding to the unsupported headers, as this header should be used between two systems and not end to end like Content-Encoding (that is stated in RFC2616). And if you think of Transfer-Encoding: gzip or deflate: This Header needs to be removed as reading the Body stream of the request will implicitly unzip the content and send uncompressed to the backend server (but chunked). |
Yes, it was! I've asked him to do that. According to GitHub Forking policy head repo owner has a right to apply limited set of operations in forked repo. One of them is notification to forked repo admin about changes in the head repo protected branch that top commits are changed. |
|
@alexreinert Ok, fine for the first part, we shouldn't have the headers if no content. And we should ensure that. About the 411 status, I need to investigate that a bit more. We need to support streaming. I'm a bit concerned about the changes you made, possibly breaking this feature. As of GET and DELETE without body, it's a large debate, as I worked with some APIs expecting a body with the GET method (eg. Elasticsearch). |
|
@alexreinert
Thanks! @ggnaegi This PR will be closed! |
|
Follow-up in #1972 |
… Transfer-Encoding: chunked if Content-Length is known (#1972) * Avoid content if original request has no content and avoid Transfer-Encoding: chunked if Content-Length is known * * Optimized mapping of empty Content * Added tests for mapping requests * Added tests to verify that mapped content is streamed * Changes requested by review * Changes requested by review * Be a little more conservative in streaming test and use 25GB instead if 100GB test data * Reduced streaming test content to 1GB as requested * Convert to file-scoped namespace * Move `Dispose` closer to the constructor * Move `ChunkedContent` class out * IDE1006: Naming rule violation * More private helpers to setup a test * Code review: round 2 * Move common helpers to `Steps` --------- Co-authored-by: Alexander Reinert <[email protected]> Co-authored-by: Raman Maksimchuk <[email protected]>
With the changes in Ocelot 23.0 the
RequestMappersets the PropertyContentusing the newStreamHttpContent. Because the Content-Length is not computed inStreamHttpContentthe header Transfer-Encoding: chunked will be set on all upstream requests even if there is no content in the downstream request (e.g. on GET requests). This causes issues when the upstream is an IIS working as reverse proxy (using ARR), in that case all GET requests are failing with status code 502.7 inside ARR.Additionally in some cases the Content-Length is required by the upstream server, it will send status code 411 if it is missing, behind Ocelot all request will fail with status code 411, as the Content-Length was not transferred to the upstream server and instead Transfer-Encoding: chunked is sent to the upstream server.
Both issue are addressed with this PR.