python::pip's 'latest' not compatible with latest 'pip' (version 20.3) due to changed output

[kBite]

Affected Puppet, Ruby, OS and module versions/distributions

• Module version: v5.0.0

How to reproduce (e.g Puppet code you use)

Starting with 20.3 pip is using a new dependency resolver which does not produce the same output as before and breaks puppet-python's logic to determine latest available package version.

Upgrade to pip 20.3 and try installing latest of any package via python::pip.

python::pyvenv { $venv_path:
  ensure     => 'present',
  systempkgs => true,
  owner      => 'agent',
  group      => 'agent',
  mode       => '0755',
}
-> python::pip { 'agent' :
  ensure       => 'latest',
  pkgname      => 'agent',
  pip_provider => 'pip',
  virtualenv   => $venv_path,
  owner        => 'agent',
  group        => 'agent',
  timeout      => 1800,
  require      => Package['python36-devel'],
  notify       => Service['agent'],
}

What are you seeing / What behaviour did you expect instead

With pip 20.2.4 installing a non-existing version returns all available versions (see below).

# /home/agent/venv/bin/pip3 --version
pip 20.2.4 from /home/agent/venv/lib64/python3.6/site-packages/pip (python 3.6)

# /home/agent/venv/bin/pip3 --log /home/agent/venv/pip.log install     agent==notreallyaversion 2>&1
Looking in indexes: https://artifactory.internal-domain.net/artifactory/api/pypi/python-pypi.python.org-remote/simple
ERROR: Could not find a version that satisfies the requirement agent==notreallyaversion (from versions: 0.2.5, 0.2.6, 0.2.7, 0.2.8, 0.2.9, 0.2.10, 0.2.13, 0.2.14)
ERROR: No matching distribution found for agent==notreallyaversion
WARNING: You are using pip version 20.2.4; however, version 20.3 is available.
You should consider upgrading via the '/home/agent/venv/bin/python3.6 -m pip install --upgrade pip' command.

Starting with version 20.3 this information is not returned anymore.

# /home/agent/venv/bin/pip3 --version
pip 20.3 from /home/agent/venv/lib64/python3.6/site-packages/pip (python 3.6)

# /home/agent/venv/bin/pip3 --log /home/agent/venv/pip.log install     agent==notreallyaversion 2>&1
Looking in indexes: https://artifactory.internal-domain.net/artifactory/api/pypi/python-pypi.python.org-remote/simple
ERROR: Could not find a version that satisfies the requirement agent==notreallyaversion
ERROR: No matching distribution found for agent==notreallyaversion

relevant Puppet code

This snippet is from pip.pp and fails due to the missing from versions: string that was dropped with the new dependecy resolver.

'latest': {
  # Unfortunately this is the smartest way of getting the latest available package version with pip as of now
  # Note: we DO need to repeat ourselves with "from version" in both grep and sed as on some systems pip returns
  # more than one line with paretheses.
  $latest_version = join(["${pip_install} ${pypi_index} ${proxy_flag} ${install_args} ${install_editable} ${real_pkgname}==notreallyaversion 2>&1",
      ' | grep -oP "\(from versions: .*\)" | sed -E "s/\(from versions: (.*?, )*(.*)\)/\2/g"',
  ' | tr -d "[:space:]"'])

reference

• http://pyfound.blogspot.com/2020/11/pip-20-3-new-resolver.html

[bastelfreak]

Hi,
I put your code, and a few variations, into an acceptance test and that looks quite good at the moment: #588

[bastelfreak]

I managed to reproduce the error in #591

[vchepkov]

There is a huge discussion on pypa/pip#9139 to bring functionality back, but for now, one can use

# /opt/python3/bin/pip install --use-deprecated=legacy-resolver agent== 
ERROR: Could not find a version that satisfies the requirement agent== (from versions: 0.1.0, 0.1.1, 0.1.2)
ERROR: No matching distribution found for agent==

notice that you don't have to add anything after ==

[uranusjr]

Hi, a resolution has since been proposed and merged to resolver pypa/pip#9139 by a contributor. Are there still things missing to resolve this?

[vchepkov]

I have the latest pip 21.0.1 and feature is not included there yet

[uranusjr]

Yes, the change will be the 21.1 release due this month. I was asking whether there are any more things you’d need, or maybe some people to test 21.1 against the current main branch to make sure it works for you, so we don’t need to wait for another feature release to get this resolved.

[bhundven-skytap]

should be fixed in newer version? pypa/pip#9405

[kBite]

pip did it again ... see #695