pip resolver installing incompatible versions

[hsheth2]

Description

See reproduce steps for setup.

It installed certifi==2021.5.30, and then reported an error for a version mismatch. Instead, pip should have looked at the requirements and installed 2020.12.5 to work with all the other dependencies.

It seems that passing --upgrade makes it work.

Ref: datahub-project/datahub#2717

Expected behavior

It should have installed certifi==2020.12.5, since the main requirements are certifi<2021.0.0 and greater than a version from 2017.

pip version

21.1.2

Python version

3.8.6

OS

MacOS

How to Reproduce

The following steps:

python3.8 -m venv venv
source venv/bin/activate
pip install --upgrade pip wheel setuptools  # sanity check
pip freeze  # empty, since I'm in a fresh venv
PIP_RESOLVER_DEBUG=1 pip install 'acryl-datahub[snowflake]==0.8.1.2'

Yields:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
snowflake-connector-python 2.4.5 requires certifi<2021.0.0, but you have certifi 2021.5.30 which is incompatible.

Output

Edited by @uranusjr: I put this into a Gist instead since the log is way too long to reasonably read and reference in comments.

https://gist.github.com/uranusjr/74ec344ce72f9c9cbe91b262bcd49dd5

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

The error message explains why that occurs. Changing this behaviour is tracked in #9094. Consolidating this issue into that.

[hsheth2]

@pradyunsg I think this one might be a bit different, since there’s only one pip install command and nothing was installed previously (fresh venv).

[uranusjr]

I can reproduce this in a fresh virtual environment. This is very weird. I suspect there is a package doing funky things to bypass the resolver and install its own things.

[uranusjr]

Hmm. So the resolver has no trouble understanding certifi needs to be pinned.

It first fetches 2021.5.30 because requests needs certifi>=2017.4.17 (line 161):

Reporter.adding_requirement(SpecifierRequirement('certifi>=2017.4.17'), LinkCandidate('https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl#sha256=c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e (from https://pypi.org/simple/requests/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*)'))
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)

And then correctly triggers a downgrade when snowflake-connector-python needs certifi<2021.0.0 (line 245):

Reporter.adding_requirement(SpecifierRequirement('certifi<2021.0.0'), LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)

Nothing afterwards invalidated certifi 2020.12.5, so the resolver runs to the end…

and proceeds to pick the wrong version of certifi!? (line 384)

('certifi', LinkCandidate('https://files.pythonhosted.org/packages/05/1b/0a0dece0e8aa492a6ec9e4ad2fe366b511558cdc73fd3abc82ba7348e875/certifi-2021.5.30-py2.py3-none-any.whl#sha256=50b1e4f8446b06f41be7dd6338db18e0990601dce795c2b1686458aa7e8fa7d8 (from https://pypi.org/simple/certifi/)'))

So this is probably a bug in the resolver when it collects the final results. Still need to investigate more. Unfortunately(?) pip install requests snowflake-connector-python gives me the correct pin (even though it went through basically all the same steps regarding certifi).

[pradyunsg]

AHA. Apologies for my confusion here!

---

and proceeds to pick the wrong version of certifi!? (line 384)

Yea, the resolver is marking the wrong requirement against the snowflake-connector-python candidate for some reason.

(SpecifierRequirement('certifi>=2017.4.17'), via=LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))

From line 245:

Reporter.adding_requirement(SpecifierRequirement('certifi<2021.0.0'), LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))

From line 249:

Reporter.adding_requirement(SpecifierRequirement('certifi>=2017.4.17'), LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))

That's... suboptimal at best. :/

[uranusjr]

I dug into for like two hours and finally think I know what’s going on.

The trigger to this issue is that snowflake-connector-python specifies certifi twice. From the wheel’s metadata:

Metadata-Version: 2.1
Name: snowflake-connector-python
Version: 2.4.5
...
Requires-Dist: certifi (<2021.0.0)
...
Requires-Dist: certifi (>=2017.4.17)
...

A wrikle to this is that there’s a reproducibility issue, because the issue only happen when certifi<2021.0.0 is fed into the resolver before certifi>=2017.4.17. But pkg_resources shuffles the Requires-Dist values on calculation, so to reproduce this deterministically, you need to change line 3041 of pkg_resources.DistInfoDistribution._compute_dependencies() to something like:

dm[None].extend(reqs_for_extra(None))

so the order of requirements don’t change between runs.

Now onto the actual resolver stuff. When the resolver pins through a candidate, it converts all of the candidate’s dependencies into criteria so they can be merged into existing criteria:

pip/src/pip/_vendor/resolvelib/resolvers.py

Lines 200 to 205 in 7c3abcc

	def _get_criteria_to_update(self, candidate):
	criteria = {}
	for r in self._p.get_dependencies(candidate=candidate):
	name, crit = self._merge_into_criterion(r, parent=candidate)
	criteria[name] = crit
	return criteria

But it uses a dict for the job. So when a candidate specifies a package multiple times (like snowflake-connector-python), the latter specification would override the previous one. Since certifi<2021.0.0 is specified first (after we patch pkg_resources; without the patch the resolver has 50% chance to get the “right” one), it got lost during resolution.

I have to say, I haven’t had such an exercise debugging an issue like this for quite a while. This kind of things is why I’m doing open source work, so thanks for that, in a way 😆 I’m unfortunately quite busy tomorrow (it’s late Saturday here), but will try to find some time to fix asap.

[uranusjr]

I posted #10083 which should fix this issue. It would be awesome if you could give it a try.

[hsheth2]

@uranusjr yup seems to be working for me!

Thanks for digging in - seems like it turned out to be quite the bug :)

[uranusjr]

Awesome! I’ll merge the resolvelib fix, and will cut 0.7.1 when we release the next pip version for inclusion.