Understanding and controlling chunk size

[hangvane]

Hello everyone,

I'm using reqwest to retrieve large HTTP responses in a streaming manner. The demo code I'm using is as follows:

use tokio_stream::StreamExt;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut stream = reqwest::get("http://www.baidu.com/")
        .await?
        .bytes_stream();
    while let Some(item) = stream.next().await {
        println!("chunk length: {}", item?.len());
    }
    Ok(())
}

For http://www.baidu.com/, the output is:

chunk length: 6041
chunk length: 3467

For https://www.baidu.com/, the output is:

chunk length: 220
chunk length: 2358
chunk length: 559
chunk length: 4716
chunk length: 1655

1. When I make a GET request to an HTTPS site, this method appears to be able to fetch partial data before the request is completed. Is this because the TLS encryption is also chunk-level that can be decrypted early?
2. The sizes and the number of chunks when sending GET requests to www.baidu.com differ between HTTP and HTTPS. I'm unsure how the item?.len() is determined - by the server, the client, or some protocol parameters in Linux?
3. Is there any way to control chunk length?

I apologize for bringing up this question here, but my attempts at finding meaningful answers have been overwhelmed by results related to Transfer-Encoding: chunked from search engines, and the source code/doc of reqwest doesn't explain much about chunk.

If anyone could provide some insights, I would greatly appreciate it.

[seanmonstar]

The size of each "chunk" is not really specified on purpose, but I can explain some of the current implementation details. The size of any chunk is based on these criteria:

1. A single read is performed on the IO transport, and however much is read (from the TCP socket, or decrypted from TLS, etc) makes up the chunk.
2. If the encoding is HTTP/1.1 "chunked", then whatever was read is checked for chunk delimiters, and if smaller than the bytes read, then it is sliced and only that part is give as a chunk. The remaining parts will be checked for the same thing on next poll.
3. In HTTP/2, chunks come from DATA frames, which is kind of like chunked encoding.

[hangvane]

Thank you for the detailed explanation!