Python 3.10: pip installs wheel for ancient Python versions

[albertosottile]

Description

I am the current maintainer of py2exe https://github.com/py2exe/py2exe and I still have not released a version of the package for Python 3.10,

However, users reported that they were able to install the package on their system. When questioned, they reported that pip first tries to install from some old sources (py2exe-0.10.0.2.tar.gz). I am sure this is my fault and setup.py is not proper there, and I intend to delete this rogue release.

Unfortunately, after this failed attempt, pip picks an even older wheel which is not compatible at all with 3.10 (py2exe-0.9.2.2-py33.py34-none-any.whl) and claims that the package was successfully installed.

I can delete the rogue 0.10.0.2 source archive, but I would like not to delete also all the releases before that one. Could you tell me why this is happening and if is there anything, besides deleting all past releases, that I can do?

Thank you for your assistance

Expected behavior

pip should either say that no version compatible with the platform was found or try to install from the source archive and fail due to the lack of dependencies/compiler.

pip version

21.3.1

Python version

3.10.0

OS

Microsoft Windows [Version 10.0.17763.2183]

How to Reproduce

pip install py2exe

Output

C:\>pip install py2exe
Collecting py2exe
  Downloading py2exe-0.10.0.2.tar.gz (104 kB)
     |████████████████████████████████| 104 kB 6.8 MB/s
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'C:\Python\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\ContainerAdministrator\\AppData\\Local\\Temp\\pip-install-lyc0or4x\\py2exe_b12f490fb2d949
048ed23d41020dcdaf\\setup.py'"'"'; __file__='"'"'C:\\Users\\ContainerAdministrator\\AppData\\Local\\Temp\\pip-install-lyc0or4x\\py2exe_b12f490fb2d949048ed23d41020dcdaf\\setup.py'"'"';f = getattr(token
ize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(
compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-pip-egg-info-wcv280uo'
       cwd: C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-install-lyc0or4x\py2exe_b12f490fb2d949048ed23d41020dcdaf\
  Complete output (17 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-install-lyc0or4x\py2exe_b12f490fb2d949048ed23d41020dcdaf\setup.py", line 17, in <module>
      from py2exe.py2exe_distutils import Dist, Interpreter, BuildInterpreters
    File "C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-install-lyc0or4x\py2exe_b12f490fb2d949048ed23d41020dcdaf\py2exe\__init__.py", line 9, in <module>
      patch_distutils()
    File "C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-install-lyc0or4x\py2exe_b12f490fb2d949048ed23d41020dcdaf\py2exe\patch_distutils.py", line 68, in patch_distutils
      from . import distutils_buildexe
    File "C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-install-lyc0or4x\py2exe_b12f490fb2d949048ed23d41020dcdaf\py2exe\distutils_buildexe.py", line 91, in <module>
      from . import runtime
    File "C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-install-lyc0or4x\py2exe_b12f490fb2d949048ed23d41020dcdaf\py2exe\runtime.py", line 3, in <module>
      from .dllfinder import Scanner, pydll
    File "C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-install-lyc0or4x\py2exe_b12f490fb2d949048ed23d41020dcdaf\py2exe\dllfinder.py", line 6, in <module>
      from . import pescan
    File "C:\Users\ContainerAdministrator\AppData\Local\Temp\pip-install-lyc0or4x\py2exe_b12f490fb2d949048ed23d41020dcdaf\py2exe\pescan.py", line 12, in <module>
      import pefile
  ModuleNotFoundError: No module named 'pefile'
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/7d/2b/5bb512a67aaa465f8629f2c0d759f56b8136a973b355db32870beb5c76af/py2exe-0.10.0.2.tar.gz#sha256=40d785a92c85908c4829bcfd4ae9985d20dcdb805e5
0097c8cba78608b06a81c (from https://pypi.org/simple/py2exe/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Downloading py2exe-0.9.2.2-py33.py34-none-any.whl (270 kB)
     |████████████████████████████████| 270 kB 6.4 MB/s
Installing collected packages: py2exe
Successfully installed py2exe-0.9.2.2

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

Can you run pip install -vvv py2exe, put the output of that in a GitHub Gist, and share the link to that Gist here?

[albertosottile]

Link to the gist: https://gist.github.com/albertosottile/242434120f73d7e73ed2063ef954bc16

[pfmoore]

Python 3.10 is compatible with py34 wheels, see py -m pip debug -v for the full list of supported tags. So there's nothing to tell pip that it shouldn't install this version. Later versions of py2exe appear to have C extensions,so they use cpXY tags - and CPython 3.10 is not compatible with cp37 wheels, so pip skips those.

If the user is saying "install py2exe", arguably 0.9.2.2 is better than saying "I can't install py2exe". So pip's behaviour is right (although I agree it's surprising, and likely not what the user actually wanted). The user could say py -m pip install "py2exe>=0.10" if they actually do want something from the 0.10 series or later - and then they will get told "sorry, cannot find anything I can install" because building from source doesn't work and there's no compatible wheel.

[albertosottile]

Thanks for your explanation, I now understand the behavior, I did not expect that the old py tags were compatible with newer Python versions. However, py2exe always shipped C extensions so, it is quite clear to me the old wheels were mistagged and should have been, instead cp33.cp34. A quick inspection of the wheel archive confirmed that.

is there a way to stop this behavior without actually deleting the releases and the wheels from PyPI? Is it possible to mark some old releases as obsolete, so that pip install py2exe does not pick up those but an experienced user can still run pip install [OPT] py2exe==0.9.2.2 if they really want (something like --pre, but in the opposite direction)?

[notatallshaw]

Yanking should work right? Pip still skip over them normally but if you specify the exact version it will still use it?

[albertosottile]

I learned a lot today, I did not know yanked releases were accessible via pip. I will give it a try on TestPyPI now. Thanks for your support.

[albertosottile]

Looks like it does not work, I deleted the rogue source archive of 0.10.0.2 and yanked every release before 0.10, but now this happens:

C:\>pip install -i https://test.pypi.org/simple py2exe
Looking in indexes: https://test.pypi.org/simple
Collecting py2exe
  Downloading https://test-files.pythonhosted.org/packages/15/94/cb55e5e73067547e0fe00b38944931a181c73339e29d905ed41d90e94cc8/py2exe-0.9.2.2-py33.py34-none-any.whl (270 kB)
     |████████████████████████████████| 270 kB 6.8 MB/s
WARNING: The candidate selected for download or install is a yanked version: 'py2exe' candidate (version 0.9.2.2 at https://test-files.pythonhosted.org/packages/15/94/cb55e5e73067547e0fe00b38944931a18
1c73339e29d905ed41d90e94cc8/py2exe-0.9.2.2-py33.py34-none-any.whl#sha256=a77ec1565ee61daaef963a8c40d695adc29819e45c8dcfbe122f8ceac30a5be9 (from https://test.pypi.org/simple/py2exe/))
Reason for being yanked: Outdated
Installing collected packages: py2exe
Successfully installed py2exe-0.9.2.2

It seems that pip still prefers a yanked release over nothing.

[uranusjr]

Yeah because (quoting PEP 592)

An installer MUST ignore yanked releases, if the selection constraints can be satisfied with a non-yanked version, and MAY refuse to use a yanked release even if it means that the request cannot be satisfied at all. An implementation SHOULD choose a policy that follows the spirit of the intention above, and that prevents "new" dependencies on yanked releases/files.

What this means is left up to the specific installer, to decide how to best fit into the overall usage of their installer.

And the approach pip currently implements is "use a yanked distribution if no non-yanked distributions satisfy the user request". Which is technically allowed by the PEP, but probably not the best approach.

[notatallshaw]

Ah my mistake, I thought there was something special about selecting an exact version but I guess pip just chooses a yanked packages in that case because it's the only one available.

[uranusjr]

There's a related open issue on this as well. As described in #8262 (comment), it is likely pip should tighten up the logic to ignore yanked version entirely unless the version range specifiers leading to the distribution selection contains == (but not containing a wildcard) or ===. But no-one has cared about this enough to submit a fix yet.

[albertosottile]

Would you be interested in a PR regarding this? Technically, I think the fix might be limited to patching ffb3d1b as it follows:

Current:

pip/src/pip/_internal/resolution/resolvelib/factory.py

Lines 276 to 284 in e46888b

	# PEP 592: Yanked releases must be ignored unless only yanked
	# releases can satisfy the version range. So if this is false,
	# all yanked icans need to be skipped.
	all_yanked = all(ican.link.is_yanked for ican in icans)
	# PackageFinder returns earlier versions first, so we reverse.
	for ican in reversed(icans):
	if not all_yanked and ican.link.is_yanked:
	continue

After (EDIT):

            # PackageFinder returns earlier versions first, so we reverse.
            for ican in reversed(icans):
                if (all_yanked and not specifier) and ican.link.is_yanked:
                    continue

EDIT: the solution might not be that trivial. One might skip unless there's a specifier (see above), or unless icans contains only one candidate, or a combination of the two.

However, I guess the core of the issue is more political. Is this is a behavior change that you would like to have in pip?

[uranusjr]

I think this would be a good change since it fixes a real issue (the one we have here), and those negatively impacted have a relatively easy way out (by adding a package==version pin). This needs to go through the normal deprecation process, but we can talk about how that'd work in the PR.

Regarding the implementation, the simplest design would be to only allow a yanked distribution only if the user uses == (or ===). This can be pretty easily done by checking the content of specifier.

[albertosottile]

I just created PR #10625 with the proposed change.