[BUG] 61.0.0 breaks "setup.py install" by missing dependencies in install_requires #3198
Comments
andy-maier
added
bug
Needs Triage
andy-maier
changed the title
|
Probably related to #3196 |
|
I had the same problem this morning since I upgraded to setuptools 61.0.0 (no dependencies installed). |
|
Hi @andy-maier, thank you very much for reporting that. |
|
Similar problem -- 61.0.0 broke a test in the pipenv CI that checks that when installing this setup.py it gets the sibling package which it no longer does. |
|
@abravalheri The pipenv CI just pulled in Note: the python 3.6 build is passing because it cannot install such a new setuptools, and despite my best efforts to pin the setup.py, CI, and run-tests.sh it was still pulling in the latest setuptools, perhaps pip was doing this. Please let me know if I should open a new ticket for this issue or if you have any other suggestions. |
|
Thanks @matteius, I will have a look. |
|
Hi @matteius, please note that the problem seems to have been happening since before 61.0.0: Do you known when was the last time this exact test worked before? Which was the version of setuptools you were using? BTW, I think the issue you are facing is not the same as the one reported originally here... I am even under the impression that setuptools never did manage requirements URL in the form of |
|
@abravalheri the test was passing three days ago. I think the reason pinning setup tools doesn't fix it is because internal pip is installing the latest setup tools anyway I think. If you look at the Python 3.6 build it is still passing because it will never install the new version of setup tools since it doesn't support py3.6. I am skiing today but can look more into it tomorrow. I do believe it is caused by setup tools upgrade at this point. |
|
Hi @matteius please do enjoy your leisure time, we can have a look at this later :) |
|
I will just add some notes here so when you have the time you can have a look.
That is completely possible. It might very well be that v61.0.0...v61.1.1 introduced something that is causing the problem. However this seems to be a second issue, and not the one originally reported here.
I don't understand what you mean here. The test I did in my machine does not use If I instead run: % git clone https://github.com/techalchemy/test-project
% cd test-project/parent_folder/pep508-package
% virtualenv .venv
% .venv/bin/python -m pip install -U pip setuptools==61.1.1 wheel
% .venv/bin/python -m pip list
Package Version
---------- -------
pip 22.0.4
setuptools 61.1.1
wheel 0.37.1
% .venv/bin/python -m pip install -e .
...
Running command git clone --filter=blob:none --quiet https://github.com/techalchemy/test-project.git /tmp/pip-install-fapb6ksm/sibling-package_2bc442202d024507bd538a882a35d28a
...
Successfully installed pep508-package-1.0.0 sibling_package-1.0.0 six-1.16.0 toml-0.10.2 urllib3-1.26.9
# I also tried `pip install .` and the outcome is the sameThen both packages seem to install fine. If instead I run (replacing % rm -rf .venv
% .venv/bin/python -m pip install -U pip setuptools==60.10.0 wheel
% .venv/bin/python -m pip list
Package Version
---------- -------
pip 22.0.4
setuptools 60.10.0
wheel 0.37.1
% .venv/bin/python setup.py develop
...
Couldn't find index page for 'sibling_package' (maybe misspelled?)
...
No local packages or working download links found for sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
error: Could not find suitable distribution for Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package')Then I see the same problem with v60.10.0 If you manage to find a minimal reproducer along the lines of the examples before that works for BTW, I am testing on: Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal
Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0] |
|
@abravalheri Thanks for the helpful feedback. I am back now and trying to figure this out again. So pip uses setuptools, and following this pip documentation: https://pip.pypa.io/en/stable/topics/vcs-support/ Now I try again with setuptools-61.0.0 ... it also seems to work. :atest setuptools does seem to work with pip in isolation, and here is an example with no setuptools first that show it is in fact setuptools it is using to do the install. So now I am very perplexed how our test started failing with no code changes. Thanks for your help, I am going to research if any other related packages would have bumped around the same time. |
|
Hi @matteius
Please note that when you run
Setuptools itself is not able to handle VCS URLs as dependencies. Any chance |
Doesn't seem to be -- I put a bunch of time into this last night and the perplexing thing is the breakage happened without any changes being pushed to pipenv repository. I have summarized some additional notes here, as there seems to be a similar issue encountered in pip-tools (which we do not rely upon, but whatever upstream thing it is seemed to come about at the same time for both projects): jazzband/pip-tools#1603 (comment) The telling thing to me is that our python 3.6 continues to pass for this test case, and I can see in the CI that is using an older version of setuptools because the newer versions aren't python 3.6 compatible. Despite my best efforts to pin setuptools for the other python versions, it seems to all be for not -- I think pip is bringing in the new version anyway when it goes to resolve. https://github.com/pypa/pipenv/runs/5708004672?check_suite_focus=true#step:7:38
Yes well at least pipenv does this, it is my latest understanding too it uses that method in some places as well as I am continuing to investigate. |
|
@abravalheri I have been comparing internals of the last pipenv version that still works for this case on newer python versions This is unfortunately taking me hours to debug, and will take hours more I am sure to try and unravel what is going on. |
|
Hi @matteius, let me see if I can help you somehow. You mentioned:
I am assuming you are getting this info either from the (a) a generated wheel, (b) a generated sdist or (c) you run |
This is comment intrigued me a little, because it might be related to another change in v61.*. If you are trying to install This happens because >>> import parent_folder.sibling_package.src.sibling_packageI have the impression that the name |
Unfortunately not yet and I may need to take a break from this for a while due to other obligations, I know the pip-tools project has a similar issue reported but no reproduction steps there either. Pipenv and pip pull in the latest setuptools like its candy, all over the place, so its really hard to restrict it back but I have done some of that and see some behavioral changes, but not enough to get the end result.
The thing is I am using this url Pip is using setuptools under the hood for this, so I think that is where the problem enters, because pipenv vendors in pip, so I am deep in the internals of pip code and the resolvelib code trying to figure out how this changed without us changing anything. |
|
On a side note: if |
We aren't directly relying on a name of UNKNOWN, I think when this actually works, it uses the right name from the |
|
@abravalheri I have narrowed my test example down to a single factor, In When I have that extra logic disabled, and some extra print statements, I can see that this gets built: However, with my hack logic in place, the output instead is: This makes all the difference in the world to get the expected result of: However, I am still head scratching about what could have changed to cause it to no longer automatically get the subdirectory on the path in this step, or why it would have changed behaviors ~4 days ago. |
|
Hi @matteius this might be caused by the change we have been discussing previously... Am I correct to assume that this function is being called with Previous versions of setuptools would create an |
|
@abravalheri Thanks for being supportive and helping me asses what is going on. Quick question -- Is it possible that somehow now in the Metadata is now lacking the subdirectory where before it might have included something about it? We have this logic also in requirementslib that tries to pull the subdirectory from the requirement, but its always None ... It is hard to tell where setuptools begins and ends in some of this code, so I apologize for referencing things you may not be familiar with |
|
Let me build the package normally and see what the generated metadata looks like. |
|
These are all different ways of "building" the package and how the metadata looks like: # Assuming the project is cloned somewhere
% cd test-project/parent_folder/pep508-package
% rm -rf src/pep508_package.egg-info build dist && pipx run build
% unzip -c dist/pep508_package-1.0.0-py3-none-any.whl pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...
% tar xOf dist/pep508_package-1.0.0.tar.gz pep508_package-1.0.0/src/pep508_package.egg-info/requires.txt
six
sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...
% .venv/bin/python -m pip install -U setuptools
% rm -rf src/pep508_package.egg-info build dist && .venv/bin/python setup.py bdist_wheel
% unzip -c dist/pep508_package-1.0.0-py3-none-any.whl pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...
% .venv/bin/python -m pip uninstall pep508-package
% .venv/bin/python -m pip install 'pep508-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/pep508-package'
% cat .venv/lib/python3.8/site-packages/pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...It seems that |
|
Sorry, I forgot one line in the previous log: .venv/bin/python -m pip install 'pep508-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/pep508-package'I just updated the previous comment to include it. |
|
@abravalheri Well thanks for your help, I now have a patch now for requirementslib that is generic enough for any subdirectory fragment, but I still don't understand how this just broke. Maybe before it wasn't passing the subdirectory properly either and setuptools was searching and finding the first setup.py in a subdirectory? Check it out, this test passed: https://github.com/pypa/pipenv/runs/5712591302?check_suite_focus=true |
|
Uummm... It is a possibility but I don't think setuptools is that smart 😅 |
|
Hi, I'm popping in from jazzband/pip-tools#1603, where I'm having trouble pinning down the problem, which may or may not be the same thing discussed here. EDIT: It looks like the below-folded example was creating an invalid Bad Dockerfile stuffDoes any of the knowledge grown here explain why building this FROM ubuntu:20.04
RUN apt-get update && apt-get -y install python3-venv
RUN python3 -m venv myvenv
RUN . ./myvenv/bin/activate && pip install wheel "pip-tools<6.5" "pip>=21.2,<22"
RUN echo -e "from setuptools import setup\n\nsetup(name='apollo', install_requires=['conan==1.45.0'])" > setup.py
RUN . ./myvenv/bin/activate && pip-compile$ buildah bud Dockerfile
...
subprocess.CalledProcessError: Command '['/myvenv/bin/python3', '/myvenv/lib/python3.8/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmp5ifphqs3']' returned non-zero exit status 1.
error building at STEP "RUN . ./myvenv/bin/activate && pip-compile": error while running runtime: exit status 1While this interactive shell with $ podman run -it --rm docker://ubuntu:20.04
# apt-get update && apt-get -y install python3-venv
# mkdir ~/myproj
# cd ~/myproj
# python3 -m venv venv
# . ./venv/bin/activate
# pip install wheel "pip-tools<6.5" "pip>=21.2,<22"
# echo -e "from setuptools import setup\n\nsetup(name='apollo', install_requires=['conan==1.45.0'])" >setup.py
# pip-compile |
|
The pip-tools daily scheduled CI began failing four tests 5 days ago, the day of setuptools These are: |
Hi @AndydeCleyre, I noticed that the fixtures seem to be creating distributions with no python modules. When I compare Note that, prior to setuptools v61, these different configurations are not equivalent: a) The example with For the majority of use cases, the situation in With the improvements in v61, setuptools will try to auto-detect which files to include, but it will fail if the root directory has a confusing/equivocal layout. The example generated by the fixtures seems to have extra folders
This is the breaking change motivating the major bump from v60 to v61 and is described in the CHANGELOG. The layouts recognized by setuptools for auto-discovery are described in https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#automatic-discovery. Layouts that differ from the ones documented are asked to explicitly list These include "intentionally empty" packages and "meta-packages"1. For those scenarios the users are asked to set Footnotes
|
|
A late but big "Thank You!!" for fixing this :-) We had already removed installation via setup.py from the documentation of our project some time ago, and I'll also remove it now from our install test. |
- So we stay at 60.10.0 - See pypa/setuptools#3198
setuptools version
61.0.0
Python version
3.10
OS
ubuntu-latest in GitHub Actions
Additional environment information
No response
Description
In our pywbem package, we test the different documented methods to install the package. One of them is still "setup.py install". I do understand that it is deprecated since setuptools v58.3.0, but that still means it should be expected to work.
The issue is that the "setup.py install" command successfully installs the pywbem package, but upon import it turns out that dependent packages were not installed, e.g. "six".
The GitHub Actions test run showing that is: https://github.com/pywbem/pywbem/runs/5687824533?check_suite_focus=true
"pip install" on the package works with setuptools 61.0.0 and does install the dependent packages.
The requirements.txt file of the pywbem project does specify the dependent packages and the setup.py script loads the dependencies from requirements.txt and specifies them in the "install_requires" parameter of setup().
The "setup.py install" approach worked before setuptools 61.0.0, e.g. with 60.10.0, see this test run.
Expected behavior
61.0.0 should still support "setup.py install" as before.
How to Reproduce
pip install setuptools==61.0.0; pip uninstall sixgit clone https://github.com/pywbem/pywbem.git; cd pywbem./setup.py install# should succeedpip list# should display:pip install .# should succeedpip list# should display the correctly installed dependencies:Output
See above
The text was updated successfully, but these errors were encountered: