Cargo-update: exceedingly slow progress on update when local registry/index is used and git-fetch-with-cli is false

[datdenkikniet]

Problem

When using a local registry located at /home/user/rust/crates.io-index (which is simply a clone of https://github.com/rust-lang/crates.io-index.git) , specified in ~/.cargo/config using the following config:

[source]

[source.mirror]
registry = "file:///home/user/rust/crates.io-index"

[source.crates-io]
replace-with = "mirror"

is used, cargo update progresses incredibly slowly, consumes 1 CPU and slowly increases its RAM usage, given that the project has at least one dependency. That's to say: when running cargo update the process takes very long.

The total update time was 21 minutes.

However, when ~/.cargo/config is changed to

[net]
git-fetch-with-cli = true

[source]

[source.mirror]
registry = "file:///home/jona/projects/rust/crates.io-index"

[source.crates-io]
replace-with = "mirror"

the update actually succeeds relatively quickly, which is the expected behaviour. With this option, the update took about 21 seconds.
Steps

1. Install rust
2. Clone https://github.com/rust-lang/crates.io-index.git
3. Edit/add the local source and overriding of the crates-io source to ~/.cargo/config using the first of the above examples. Point the mirror source to the directory into which crates.io-index was cloned.
4. Create an empty/basic project
5. Add a dependency to this project (I used time = "0.1.0", but as far as I can tell any dependency that triggers an update of the registry will cause this issue)
6. Attempt to run cargo update

Possible Solution(s)

Use

[net]
git-fetch-with-cli = true

in cargo configuration.

Notes

Output of cargo version:
I tested found the issue on 1.46.0, 1.49.0 (d00d64d 2020-12-05), and 1.50.0.

[ehuss]

Thanks for the report. I have noticed this, too, and it can be extremely slow. Unfortunately I think this is an issue with libgit2 (libgit2/libgit2#2836). For now, git-fetch-with-cli is probably the best workaround.

[romecode]

Super slow on https git pulls on macos here. Spent a while trying different things... this worked for me.

[source.crates-io]
registry = "git://github.com/rust-lang/crates.io-index.git"

[datdenkikniet]

@romecode good to see that you have fixed your problem, but I don't think your solution is relevant to this issue.

I've edited the issue itself to clearly include the workaround for the problem.

[kozik47]

[source.crates-io]
registry = "git://github.com/rust-lang/crates.io-index.git"

Thanks. This fixed extremely slow 'Updating crates.io index' for me on Arch

[asottile-sentry]

in my situation, in addition to taking a bunch of CPU the update also allocated ~8.6GB of ram which was not freed after the update (eventually leading to OOM). using the workaround above worked great

[RouHim]

It even leads to out of memory exceptions, when cross compiling in an alpine arm container in github actions:
https://github.com/RouHim/this-week-in-past/runs/7892722833

(fixed with git-fetch-with-cli = true)

[weihanglo]

FWIW, there is an ongoing project on integrating gitoxide¹ to replace some uses of libgit2 in Cargo. I wouldn't say it will be landed soon but we can keep a eye on it, and maybe help @Byron if needed.

Footnotes

1. It is a project sponsored by Rust Foundation. Here is a recent discussion: https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/.60gitoxide.60.20integration.20for.20shallow.20clone.20support ↩

[Byron]

gitoxide at this time supports git-aware transport exclusively so will do a 'proper' clone with file:///home/… and /home/…, and in doing that it is faster than git itself (when it is also using git-aware transport). Thus I'd also expect this issue to be fixed once the crates-io indices are cloned and fetched with gitoxide.

Edit: It also turns out that part of the slowness comes from libgit2 implementing its own slow upload-pack, which gitoxide doesn't do either - instead it calls git-upload-pack to deal with the remote side. That should help with performance too until gitoxide implements the server side as well.

[dxspro28]

Set your source to this mirror. It worked for me
[source.crates-io]
replace-with = 'tuna'

[source.tuna]
registry = "https://mirrors.tuna.tsinghua.edu.cn/git/crates.io-index.git"

[datdenkikniet]

Thanks for thinking along @dxspro28, but that solution is irrelevant to the problem described here. Configuring the registry to be some other online source is not a solution when one wishes to use a local registry.

Additionally, unless the registry located at https://mirrors.tuna.tsinghua.edu.cn/git/crates.io-index.git contains some crates that are not available through crates.io, there is no real reason to use that over the default.