Use pip 20.2 feature 2020-resolver

[jwhitlock]

Install python libraries with pip 20.2 and the 2020-resolver planned for the 20.3 release. See the changes to the pip dependency resolver in 20.2 (2020)

This fails for me when installing in the docker image, and when installing on macOS in a virtualenv. It succeeds when the default resolver in pip 20.2 is used.

The error with --use-feature=2020-resolver is:

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    idna<3,>=2.5 from https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl#sha256=b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 (from requests[security]==2.24.0->-r requirements/shared.txt (line 9))

However, idna==2.10, with hashes, is defined in requirements/constraints.txt.

I'm going to submit this to the pip developers as a potential use case for the 2020-resolver, so here's some background. There are two scenarios when we install requirements:

• For development and production, we install all the requirements. Some of these require binary development libraries, such as the MySQL client libraries.
• To build documentation on https://ichnaea.readthedocs.io, we just need a subset of the requirements, and none of the binary development libraries.

To handle this, we have 4 requirements files:

• requirements/default.txt - The top-level requirements for production and development
• requirements/docs.txt -The top-level documentation requirements (Sphinx and a Sphinx theme)
• requirements/shared.txt - Top-level requirements used by both production and docs (certifi for certs, everett for configuration, requests[security], and urllib3[secure])
• requirements/constraints.txt - Second-level requirements without [extra] specifiers

We specify hashes in all four files, using hashin to populate them.

Our requirements files include other files:

• default.txt: -c constraints.txt, -r docs.txt, -r shared.txt
• docs.txt: -c constraints.txt, -r shared.txt
• shared.txt: None
• contraints.txt: None

This allows us to define each requirement exactly once, and ensure that docs are built with the same versions in the development environment and CI as on ReadTheDocs.

This allows us to install requirements with pip install -r requirements/default.txt for development or for creating Docker images, and pip install -r requirements/docs.txt on ReadTheDocs. It works with the default resolver through pip 20.2. With pip --use-feaature=2020-resolver, it seems that the hash checker gets confused about some constraints.

It seems to be related to constraints that are required in two paths from two different files (determined with pipdeptree). For example, idna is included twice:

• Sphinx (docs.txt)
  • requires requests (shared.txt)
    • requires idna (constraints.txt)
• geoip2 (default.txt)
  • requires aiohttp (constraints.txt)
    • requires yarl (constraints.txt)
      • requires idna (constraints.txt)

I've also seen chardet have the --require-hashes mode when installing in a virtualenv (system packages allowed) on macOS. It has similar multiple requires paths:

• Sphinx (docs.txt)
  • requires requests (shared.txt)
    • requires chardet (constraints.txt)
• geoip2 (default.txt)
  • requires aiohttp (constraints.txt)
    • requires chardet (constraints.txt)

[jwhitlock]

The 2020 resolver is OK with multiple requirements files, but it does not consider hashes at all in constraints files. This issue is being discussed in pypa/pip#8792, where there is general agreement that hashes in constraints files should be loaded. The questions are around what happens when there are multiple declarations for the same version but with different sets of hashes.

Update 1 - Also, we could work around the issue by using -r constraints.txt instead. This would allow us to use the new resolver, but "unused" constraints would now be installed instead of skipped.

[jwhitlock]

A fix for pypa/pip#8792 has merged, but has not been released. My test project at https://github.com/jwhitlock/pip-resolver-demo continues to fail. I'll wait for the fix to get into a released version to re-file.

ReadTheDocs.org started adding --use-feature=2020-resolver in readthedocs/readthedocs.org#7412, breaking the docs build. The -r constraints.txt work-around may work, but might require splitting documentation constraints from full install constraints. There may also be a way to disable this feature in RTD.

[jwhitlock]

Read The Docs was randomly setting --use-feature=2020-resolver, they turned it off for the Ichnaea project, docs are happy again.

[jwhitlock]

Force push to rebase on origin/main, and then add @pradyunsg's suggested change. The test now keeps pip 20.2.3, but this release doesn't have the fixes from pypa/pip#8839, so the install continues to fail in the same way.

[jwhitlock]

I filed the new issue pypa/pip#9020 last week, and got some feedback. The devs say that the restriction is that constraints plus hashes must be an exact match. For example, Django 3.0 has a requirement of asgiref~=3.2.10, and the constraint asgiref==3.2.10 is not an exact match, so it doesn't match for hash checking. It's a regression, but probably will be an intentional regression, like package[extras] in constraints, and constraints will no longer work well with hashes.

I'm going to follow @willkg's strategy in mozilla-services/socorro#5595 and switch to pip-compile instead. I also need to adjust my advocacy for constraints as requirements of requirements. They are a lot less useful with the new resolver.

[jwhitlock]

Moved to issue #1407.