Index parsing errors are swallowed

[jonhoo]

Problem

When using a private registry, I recently ran into a case where a crate version that is in the index (confirmed by cURLing the registry directly) could not be resolved by Cargo, giving me an error like

error: no matching package named `something` found
location searched: `private-registry` index

I quickly suspected that the index entry was invalid somehow, but Cargo gave no errors or warnings, even with CARGO_LOG=trace. Furthermore, by tweaking http_remote.rs, I found that its query to the registry (with an if-none-match header) returned NotModified, indicating Cargo did have the etag of the latest index file, which did include the crate/version in question. By removing the etag header, I also confirmed that the contents Cargo got from its request contained the version in question.

Long story short, I discovered that Cargo ends up swallowing errors related to invalid index entries. The first time Cargo downloads an index file with an invalid entry, it hits

cargo/src/cargo/core/summary.rs

Lines 345 to 348 in 05f54fd

	bail!(
	"optional dependency `{dep}` is not included in any feature\n\
	Make sure that `dep:{dep}` is included in one of features in the [features] table."
	);

However, that gets turned into a tracing::info in

cargo/src/cargo/sources/registry/index/mod.rs

Lines 604 to 619 in 05f54fd

	Err(e) => {
	// This should only happen when there is an index
	// entry from a future version of cargo that this
	// version doesn't understand. Hopefully, those future
	// versions of cargo correctly set INDEX_V_MAX and
	// CURRENT_CACHE_VERSION, otherwise this will skip
	// entries in the cache preventing those newer
	// versions from reading them (that is, until the
	// cache is rebuilt).
	tracing::info!(
	"failed to parse {:?} registry package: {}",
	relative,
	e
	);
	continue;
	}

The user is likely to miss this since Cargo tracing logs are off by default, so they will just observe that Cargo claims the version doesn't exist.

Unfortunately, the issue doesn't stop there. If the user then tries to run with CARGO_LOG=trace, they will then see no errors or warnings from Cargo. In fact, the logs will make it seem like the invalid entry doesn't exist at all. There will be no trace of it anywhere. This is because when Cargo parses an index entry, it caches the parsed lines so that it doesn't have to parse them again

cargo/src/cargo/sources/registry/index/mod.rs

Line 622 in 05f54fd

cache.versions.push((version.clone(), line));

cargo/src/cargo/sources/registry/index/mod.rs

Lines 627 to 631 in 05f54fd

	let cache_bytes = cache.serialize(index_version.as_str());
	// Once we have our `cache_bytes` which represents the `Summaries` we're
	// about to return, write that back out to disk so future Cargo
	// invocations can use it.
	cache_manager.put(name, &cache_bytes);

However, when a line can't be parsed, it doesn't cache that line (note the continue where the error gets turned into tracing::info!). It also doesn't leave that line as "unparsed" somewhere. So on subsequent invocations, Cargo grabs the version list from the cache manager to avoid parsing lines again, doesn't observe the (invalid) index entry for the version in question so it doesn't warn, then downloads the file again because it doesn't find the requested version, that download yields and empty response because the etag does match the latest index file, and so Cargo gives up.

So, two requests:

1. Cargo should be louder when it finds an invalid index entry.
2. Cargo should repeat the warning about an invalid index entry even if it's in an up-to-date cached version of the index file.

Steps

1. Inject a new version with an invalid index entry into a registry.
2. Take a dependency on said version.
3. Run cargo build.
4. Observe that Cargo only says "version not found" with no further warnings.
5. Re-run cargo build with CARGO_LOG=trace.
6. Observe that there is no information in the logs about the invalid index entry which caused the version not to be found.

Possible Solution(s)

No response

Notes

In my case, the index entry parsing error I was eventually able to extract by disabling the if-none-match logic and then running with CARGO_LOG=trace was:

failed to parse "so/me/something" registry package: optional dependency `uom_0_35` is not included in any feature
Make sure that `dep:uom_0_35` is included in one of features in the [features] table.

I suspect, but am not quite sure, that this may be an erroneous index entry manipulation by the private registry provider we're using. Although it could also be that cargo publish allowed publishing something with a nonsensical optional dependency in the first place?

Version

cargo 1.85.0 (05f54fdc3 2024-12-03)

[epage]

This is heavily related to #10623.

Ideally we track the error state through Cargo. We've added IndexSummary to better track state. That now tracks newer versions which can help with #10623 (though we should also track the MSRV if present) though we still need to pipe this through up to the user. Also, unsure if that is only on successful parses or also on failed parses which we should also support.

What is distinct about this Issue compared to #10623 is adding an error variant for when the parse fails for a reason other than a newer index version.

Granted, this will only help if we can extract the name and version. If its too corrupted to even do that, we'll need to consider other means of reporting.

[ehuss]

Thanks for the report! The failures are silenced somewhat on purpose, since this is intended to handle the scenario where an older version doesn't understand the format of newer index entries. We wanted to avoid the situation where older versions would spam warnings on the console whenever they are used. I think this is somewhat related or a duplicate of #7929 where we could have a controlled method of presenting this information. Or maybe similarly to #4309, where there could be a separate command to display that.

[epage]

I think there are two separate cases

• When a package/version isn't found at all (cargo <1.60 failed to select a version of a dependency due to either ? (weak) or dep: (namespaced) features #10623)
• When a version is found but its not the one the user expected (Provide a way to describe why newer versions are not selected #7929)

The work we have planned for #10623 would make it so we have more information available to show for #7929.