Regression found in 1.67: Timeout downloading crates from custom registry

[nbigaouette]

Problem

I am in the process of updating to 1.69 in some repos and found what I believe to be a regression.

The CI process started to fail trying to download the crates from a custom registry (named internal-registry here):

warning: spurious network error (1 tries remaining): [28] Timeout was reached (download of `mycrate v0.1.0 (registry `internal-registry`)` failed to transfer more than 10 bytes in 30s)
error: failed to download from `https://example.com/api/v1/crates/mycrate/0.1.0/download`

Caused by:
  [28] Timeout was reached (download of `mycrate v0.1.0 (registry `internal-registry`)` failed to transfer more than 10 bytes in 30s)

At first a connection issue was though to be causing this, but the problem was constant over many weeks. Adding a curl call to download the crate right before the cargo call was able to download the crate too so it isn't caused by a network issue.

Unfortunately the problem occurs inside Jenkins (using a kubernetes agent) and I cannot reproduce this locally (macOS). Jobs were ran with all Rust versions since 1.60 to see if it could be caused by the toolchain update.

All versions from 1.60 to 1.66 succeeded in CI, while 1.67 stopped working with the error above.

Looking at cargo CHANGELOG for 1.67 the only line that jumps to me is the last one of the Changed section:

Updated the internal HTTP library libcurl with various fixes and updates. #11307 #11326

Looking at #11307 in particular mentions the update:

libcurl from 7.83.1 to 7.86.0

A lot has changed between those two libcurl versions so I'm not sure what to look for.

[ehuss]

Can you include the output of cargo -Vv?

What server are you running (what is the host OS, the web server, the registry implementation, etc.)?

Does it still happen on nightly?

[nbigaouette]

Note that I do not control the Jenkins CI environment. I do control the docker image being used in there though. I just tried in the docker image locally but couldn't reproduce... :(

Here's the info from inside the docker container:

# cargo +1.67.0 -Vv
cargo 1.67.0 (8ecd4f20a 2023-01-10)
release: 1.67.0
commit-hash: 8ecd4f20a9efb626975ac18a016d480dc7183d9b
commit-date: 2023-01-10
host: x86_64-unknown-linux-gnu
libgit2: 1.5.0 (sys:0.16.0 vendored)
libcurl: 7.86.0-DEV (sys:0.4.59+curl-7.86.0 vendored ssl:OpenSSL/1.1.1q)
os: CentOS 7.0.0 [64-bit]

The registry is Alexandrie.

Suprisingly it worked using nightly!

# cargo +nightly -vV
cargo 1.71.0-nightly (9e586fbd8 2023-04-25)
release: 1.71.0-nightly
commit-hash: 9e586fbd8b931494067144623b76c37d213b1ab6
commit-date: 2023-04-25
host: x86_64-unknown-linux-gnu
libgit2: 1.6.3 (sys:0.17.0 vendored)
libcurl: 8.0.1-DEV (sys:0.4.61+curl-8.0.1 vendored ssl:OpenSSL/1.1.1t)
ssl: OpenSSL 1.1.1t  7 Feb 2023
os: CentOS 7.0.0 [64-bit]

[weihanglo]

We've got another update in 1.70.0 (#11871). Probably somehow related to #11746.

Could you try the current beta (1.70.0) as well? We are all good if beta is good.

[nbigaouette]

Yes, beta worked too.

# cargo +beta -Vv
cargo 1.70.0-beta.2 (84b7041fd 2023-04-13)
release: 1.70.0-beta.2
commit-hash: 84b7041fd2745ee6b3b4a150314f81aabb78e6b2
commit-date: 2023-04-13
host: x86_64-unknown-linux-gnu
libgit2: 1.6.3 (sys:0.17.0 vendored)
libcurl: 8.0.1-DEV (sys:0.4.61+curl-8.0.1 vendored ssl:OpenSSL/1.1.1t)
os: CentOS 7.0.0 [64-bit]

I'm looking at the issue/PR for some possible workaround to my issue.

[weihanglo]

Thanks for helping the confirmation. Go ahead and close this, as Rust the project doesn't have a policy of backporting older version of toolchain, and this issue has been resolved in the current beta channel.

If you find something useful, feel free to share here or elsewhere :)

[nbigaouette]

I was able to make it work with Rust 1.69 by clearing both http_proxy and https_proxy environment variables in the CI job (haven't tried to see if clearing just one worked).

Thanks for the help into identifying the issue!

Now into fighting Jenkins to clear the variable...

[nbigaouette]

So I couldn't clear the http_proxy and https_proxy variables in Jenkins, but I could overwrite no_proxy with the custom registry hostname. cargo respected that and was then able to download the crates.
Will remove this workaround when updating to 1.70.
Thanks again!