why is allowCustomCrateRegistries self-hosted only?

[rbtcollins]

We have a private Cargo repository, conceptually the same as a private Maven repo: there is an encrypted token in a HostRule for the http://HOST/ that the registry is stored on.

Accessing the registry metadata could be done with either the sparse API or a shallow git clone : the credentials we have will do that correctly. As far as I can tell renovate itself isn't dependent on cargo - it parses and processes the data itself, so it should be possible to do that cheaply without having to muck around with the cargo [net] configuration etc - though if you do call out to cargo, environment variables control this and allow overriding things without state management.

If there is no fundamental reason, and it is just a code gap, then I'd be happy enough to put together a PR to enable this, since it would be super useful and avoid having a github action setup just to thunk across.

[rarkins]

I believe this was just a code gap, and a PR would be welcome.

If I remember correctly, private crate registries were implemented using git, and naturally we wanted to cache the clone instead of cloning once per package, and then we worried about accidentally leaking caches between users. Result: make it self hosted only initially.

[rbtcollins]

If ensuring hygiene is important - is there some containment already in place for Renovate aaS ? (E.g. a chroot / cgroup around the processing of each repository?) If so, contained caching for the git repos should be very easy: use a strong hash of the url and the credentials to the git repo as a cache location. After each run git push -f the origin branch from the isolated run of cargo back to the cache location, seeding the cache for the next run. I could implement this rather than the stateless isolation I described below, but it does depend on containment - without that there is too much room for someone using e.g. npm or pip or something to cause arbitrary code to run and access the cached repos, disclosing package names and versions and download urls.

[rbtcollins]

Ok so a few notes so far.

Artifactory (the private registry we're using) does support the sparse index protocol, so thats good.

However, it supports it at a different URL space than the git index.

e.g. if the git index is at
https://COMPANY.jfrog.io/artifactory/git/NAME.git then the sparse index is at https://COMPANY.jfrog.io/artifactory/api/cargo/NAME/index.

I think I need to add registryAliases support for cargo to do that? And then we need to resolve by direct HTTP call rather than looking up on disk, but that should be about it.

I'd also like to add documentation to guide others on this path.

[rbtcollins]

However that will only make it possible to detect the presence of the change. The requirement to call 'cargo update' when there is a lockfile will cause this to fail today because of the missing authentication support.

I think the following should work

0. use registryAliases to detect if we need to do this code path
1. we create a temporary sandbox dir to isolate cargo.
2. We copy the top level structure from your existing CARGO_HOME to it: config.toml, credentials.toml,
3. We create registry/index/{github.com-1ecc6299db9ec823,index.crates.io-93a9e06d8945f2c0} as symlinks to the original Cargo structure (these are the public crates.io). So new indices will be downloaded into this sandbox structure, but public ones are shared. The global lock will not be visible to other cargo instance, but the crates-io git index is concurrent writer safe (its git); and for a sparse crates.io index sparse index files are written in-place with truncate, so concurrent cargo runs might stomp on each other but it will self correct - (see Summaries::parse) - with the last cargo completing the final write out of the file and so the system will recover without intervention.
4. We write the hostRules values we've received to a .netrc file in another temp dir - see below - because git uses https://curl.se/libcurl/c/CURLOPT_NETRC.html to enable HTTP authentication, we can project the git CLI authentication to disk, sandboxed to the one invocation of cargo, safely.
5. when running cargo update we export CARGO_HOME and HOME to the temp dirs.
6. delete the temp dirs when finished

the file structure would be something like

• cargo-temps/{tempdir}/cargo/config.toml
• cargo-temps/{tempdir}/cargo/credentials.toml
• cargo-temps/{tempdir}/registry/cache/github.com-1ecc6299db9ec823 -> ~/.cargo/registry/cache/github.com-1ecc6299db9ec823
• cargo-temps/{tempdir}/registry/cache/index.crates.io-93a9e06d8945f2c0 -> ~/.cargo/registry/index/index.crates.io-93a9e06d8945f2c0
• cargo-temps{tempdir}/home/.netrc

It might be good to facilitate easy cleanup to add a date in there:

• cargo-temps/yyyy-mm-dd/{tempdir}/cargo/config.toml
• etc

And then include a background task that looks for not-today directories and deletes them, Just In Case something interrupts renovate and a directory is leaked.

What do you think?

[rarkins]

The first step would be making sure Renovate can lookup the sparse index over HTTP, including populating the hostRules (but the hostRules should be the easy part). Can Renovate detect the custom registry during the cargo extract phrase, or would it need to be configured using registryUrls?

The second step would be making sure cargo update is passed the necessary credentials it needs, by converting them from hostRules to a format cargo understands.

Right now I'm not sure why there's a need for any sandbox concept, or are you worried about Cargo leaking the details somewhere outside the repo directory that's not cleaned up?

[rarkins]

Example to copy:

renovate/lib/modules/manager/pip_requirements/artifacts.ts

Lines 78 to 80 in 4abd0ab

	extraEnv: {
	PIP_CACHE_DIR: await ensureCacheDir('pip'),
	},

[rbtcollins]

It looks like privateCacheDir would be more suitable? Then just needs an async helper like ensureCacheDir for it?

[rarkins]

In the hosted app, the cache dir is cleared out after each repo anyway.

If you prefer it to be cleared after each command (meaning if you run multiple cargo artifact updating commands in the same repo, they don't share cache) then that should be the default behavior today.

So I'm not sure if a private cache dir helps in any way.

[rbtcollins]

Ok. Ideally the cache would live for the whole repo, so that multiple crates in a single workspace reuse the cache.

[rarkins]

This might be enough:

 extraEnv: { 
   CARGO_HOME: await ensureCacheDir('cargo'), 
 }, 

[rbtcollins]

What is getDependencyURL used for? It seems like cargo dependencies urls for dependency grouping are extracted from the packages, not from the datasource. Is it perhaps a fallback?

[rarkins]

dependencyUrl is optional and populated if there's a public HTTP url for the package on the registry. e.g. https://npmjs.com/package/lodash which is different from the sourceUrl which is https://github.com/lodash/lodash and different from homepage which is https://lodash.com.

[rbtcollins]

so a separate question then, but does it really make sense to have a depedencyUrl for private registries?

[rarkins]

The only purpose of it is to provide a link to the URL for users. It's entirely optional, and you don't need it if you don't think anyone would want to click on it

[rbtcollins]

So, I think I have something approximating a branch that should work, but it has a caveat I need to check on :) .

Your servers will need to run nightly cargo with two feature flags - -Z registry-auth and -Z sparse-registry.

I could:

• hard code the use of this, which will keep working until the features are stable and then start warning, but be a trivial fix.
• make it opt-in for users of Cargo via a new setting in the manager

What do you think?

[rarkins]

When you say "need to run", do you mean for the cargo update commands which are used to update the lock file?

[samrogerson]

These features are now stable in cargo.

I've recently started experimenting with renovate. We make heavy use rust + sparse registries on artifactory, so I ended up implementing this feature, locally, and have verified it on our real repos. Happy to contribute a PR, but realise @rbtcollins also has a branch (only just came across this discussion!)

[rbtcollins]

I am just crazy bandwidth constrained at work. I say put your patch up - great! If its missing things we need I can adjust my patch to match.

What we need is sparse + authenticated registry support.

[samrogerson]

Sounds good, will get something up :)

[samrogerson]

#21187