feat(cargo): support sparse registry indices

[tofay]

Changes

Allow Renovate to update dependencies from private sparse Rust crate registries.

Context

Sparse registries, documented at https://doc.rust-lang.org/nightly/cargo/reference/registry-index.html#sparse-protocol, use a HTTP file based index rather than git. Per https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-sparse-protocol.html they will stabilize in the next Rust release (1.68). I use them at work, and have started using renovate to keep Rust dependencies up to date.

The directory structure of sparse registries (in particular the index suffixing e.g fo/ob/foobar) is identical to existing registries. Given that renovate already uses HTTP file based access to the crates.io index on github, this PR mainly consists of detecting the sparse+` prefix of index URLs and reusing the crates-io code path.

(I've not touched on authentication here as sparse registries aren't necessarily authenticated. I am running renovate self hosted and have this code working against authenticated sparse registries by:

• configuring a host URL for renovate's HTTP calls to the index
• configuring cargo to use authentication via environment variable
  )

I found #18941 when looking for related issues, though the changes in that draft PR looked specific to JFrog artifactory.

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

I've been running this self-hosted against a few dozen private Azure DevOps repositories that use dependencies from a sparse registry for a couple of weeks, and had PRs raised succesfully.

[HonkingGoose]

Suggested change

	Renovate can only update Cargo lockfiles for projects using dependencies from sparse registries if the local Rust toolchain has the sparse-registry feature enabled (i.e beta/nightly/or 1.68 or later - 1.68 is due for release on March 9th 2023).
	Before Renovate can update Cargo lockfiles for projects using dependencies from sparse registries you must:
	- Use Rust `1.68` or later
	- Enable the `sparse-registry` feature in your local Rust toolchain

How about something like this?

The current sentence is too long, and has too much information in one go. So we should split things up into smaller parts. 😉

[tofay]

Perhaps omit all mention of the sparse-registry feature, which is actually a compilation flag so unlikely something most users can control?

So:
"Before Renovate can update Cargo lockfiles for projects using dependencies from sparse registries you must use Rust 1.68 or later"

The only oddity with that is that Rust 1.68 isn't out yet, but anyone using sparse registries before then is likely using a beta/nightly toolchain, or a self-built with sparse-registry built in (I fall into this camp :)), and can figure out how to get it working.

[HonkingGoose]

Perhaps omit all mention of the sparse-registry feature, which is actually a compilation flag so unlikely something most users can control?

It sounds we have two groups of people here:

1. General public who uses Renovate to update Rust dependencies
2. Maintainers of Rust repositories who want Renovate to work properly for their published packages

Sounds like the compilation flag sparse-registry is aimed at number 2 then?

I'll let the Renovate maintainers decide what to keep in the docs and what to drop. 😉

The only oddity with that is that Rust 1.68 isn't out yet, but anyone using sparse registries before then is likely using a beta/nightly toolchain, or a self-built with sparse-registry built in (I fall into this camp :)), and can figure out how to get it working.

I'm okay with mentioning the Rust version 1.68 ahead of time. Are you sure that the sparse-registry feature will be included in version 1.68?

[tofay]

Are you sure that the sparse-registry feature will be included in version 1.68?

Yes, at least that's still the current plan: https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-sparse-protocol.html

[rarkins]

@Turbo87 could you please review?

[viceice]

related discussion

• why is allowCustomCrateRegistries self-hosted only? #18872

[Turbo87]

looks alright to me on first glance. I haven't tried it out locally though and my experience is limited to the sparse implementation for crates.io itself.

[tofay]

closing as #21187 has landed