egg-info in local directory makes pip think package is installed #6558

pv · 2019-05-30T12:15:06Z

Environment

pip version: 19.1.1
Python version: 3.7.3
OS: Linux

Description

A PACKAGE.egg-info directory in current working directory can make pip install WHEELFILE to say "Requirement already satisfied" and not actually install the wheel file.

Expected behavior

pip install would install the package in the wheel file to the appropriate site-packages directory, so that it remains available after chdir-ing out of the current directory.

How to Reproduce

Reproducing shell script:

set -e -x
mkdir tmp; cd tmp

cat <<EOF > setup.py
from setuptools import setup
setup(name='my_test_package', version="1.0", packages=['my_test_package'],
      include_package_data=True # <- forces creation of .egg-info
      )
EOF
mkdir my_test_package
> my_test_package/__init__.py

python3 -mvenv env
./env/bin/pip install -U pip wheel
mkdir dist
./env/bin/python setup.py bdist_wheel # <- creates .egg-info
./env/bin/python -mpip install dist/my_test_package-*.whl  # <- doesn't install it!
# Adding --force-reinstall option makes the package to be installed,
# but this also reinstalls any dependencies, which is undesired.

cd ..
if ./tmp/env/bin/python3 -c 'import my_test_package'; then
    echo "OK"
else
    echo "FAILED"
fi

Output

+ mkdir tmp
+ cd tmp
+ cat
+ mkdir my_test_package
+ python3 -mvenv env
+ ./env/bin/pip install -U pip wheel
Collecting pip
  Using cached https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/bb/10/44230dd6bf3563b8f227dbf344c908d412ad2ff48066476672f3a72e174e/wheel-0.33.4-py2.py3-none-any.whl
Installing collected packages: pip, wheel
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
Successfully installed pip-19.1.1 wheel-0.33.4
+ mkdir dist
+ ./env/bin/python setup.py bdist_wheel
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/my_test_package
copying my_test_package/__init__.py -> build/lib/my_test_package
running egg_info
creating my_test_package.egg-info
writing my_test_package.egg-info/PKG-INFO
writing dependency_links to my_test_package.egg-info/dependency_links.txt
writing top-level names to my_test_package.egg-info/top_level.txt
writing manifest file 'my_test_package.egg-info/SOURCES.txt'
reading manifest file 'my_test_package.egg-info/SOURCES.txt'
writing manifest file 'my_test_package.egg-info/SOURCES.txt'
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/my_test_package
copying build/lib/my_test_package/__init__.py -> build/bdist.linux-x86_64/wheel/my_test_package
running install_egg_info
Copying my_test_package.egg-info to build/bdist.linux-x86_64/wheel/my_test_package-1.0-py3.7.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/my_test_package-1.0.dist-info/WHEEL
creating 'dist/my_test_package-1.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'my_test_package/__init__.py'
adding 'my_test_package-1.0.dist-info/METADATA'
adding 'my_test_package-1.0.dist-info/WHEEL'
adding 'my_test_package-1.0.dist-info/top_level.txt'
adding 'my_test_package-1.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
+ ./env/bin/python -mpip install dist/my_test_package-1.0-py3-none-any.whl
Requirement already satisfied: my-test-package==1.0 from file:///home/pauli/tmp/xxx/tmp/dist/my_test_package-1.0-py3-none-any.whl in /home/pauli/tmp/xxx/tmp (1.0)
+ cd ..
+ ./tmp/env/bin/python3 -c 'import my_test_package'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'my_test_package'
+ echo FAILED
FAILED

The text was updated successfully, but these errors were encountered:

uranusjr · 2019-05-31T04:50:07Z

I don’t consider this a bug, in the sense of how pip “discovers” installations (i.e. looks for metadata directories), and that the package you want to install does actually present in sys.path, making it effectively “installed” (i.e. available for import) by your runtime at that time.

FWIW, this kind of confusion is what the src/ layout tries to resolve.

pv · 2019-05-31T07:46:19Z

This is an issue for build automation, which cannot necessarily assume specific package layout. It is also surprising behavior in that pip's concept of "installed" packages depends on cwd. Maybe this concept is not a good idea and should be changed?

…

On May 31, 2019 4:50:19 AM UTC, Tzu-ping Chung ***@***.***> wrote: I don’t consider this a bug, in the sense of how pip “discovers” installations (i.e. looks for metadata directories), and that the package you want to install does actually present in `sys.path`, making it effectively “installed” (i.e. available for import) by your runtime at that time.> > FWIW, this kind of confusion is what the [src/ layout](pypa/packaging.python.org#320) tries to resolve.> > -- > You are receiving this because you authored the thread.> Reply to this email directly or view it on GitHub:> #6558 (comment)

pv · 2019-05-31T07:48:21Z

Moreover, without egg-info, the package is not detected by pip as "installed", although it is in sys.path.

…

On May 31, 2019 4:50:19 AM UTC, Tzu-ping Chung ***@***.***> wrote: I don’t consider this a bug, in the sense of how pip “discovers” installations (i.e. looks for metadata directories), and that the package you want to install does actually present in `sys.path`, making it effectively “installed” (i.e. available for import) by your runtime at that time.> > FWIW, this kind of confusion is what the [src/ layout](pypa/packaging.python.org#320) tries to resolve.> > -- > You are receiving this because you authored the thread.> Reply to this email directly or view it on GitHub:> #6558 (comment)

pfmoore · 2019-05-31T08:09:51Z

It is also surprising behavior in that pip's concept of "installed" packages depends on cwd. Maybe this concept is not a good idea and should be changed?

Well, it's python's concept (as you say, the package is on sys.path). Pip requires additional metadata (the egg-info directory) to manage the package, but that's the only difference (and just means that sometimes pip doesn't recognise a package that Python considers to be present).

Yes, it's surprising, but it's been core Python language behaviour essentially forever, so it would be a significant language change to modify it.

pv · 2019-05-31T09:02:29Z

I understand your point from a pip implementation perspective, but pip's concept of installed packages does not currently match python's, as it depends on the egg-info, and that they should match appears not so strongly argued. From the perspective of an user who is well aware of how python works, this still managed to be a surprising footgun, and the fact that it depended on what is in setup.py made it more difficult to pinpoint.

…

On May 31, 2019 8:10:03 AM UTC, Paul Moore ***@***.***> wrote: > It is also surprising behavior in that pip's concept of "installed" packages depends on cwd. Maybe this concept is not a good idea and should be changed? Well, it's *python's* concept (as you say, the package is on `sys.path`). Pip requires additional metadata (the `egg-info` directory) to manage the package, but that's the only difference (and just means that sometimes pip doesn't recognise a package that Python considers to be present). Yes, it's surprising, but it's been core Python language behaviour essentially forever, so it would be a significant language change to modify it.

chrahunt · 2019-07-21T18:35:59Z

The expectation that the default behavior of pip install <requirement> is to "ensure is available in the current environment" (where environment does not include the cwd) seems reasonable to me.

Are there any functional impacts to removing the current working directory from sys.path on pip install? Is anyone relying on this behavior?

AndydeCleyre · 2020-03-10T20:22:56Z

I don't see mentioned here these specific confusions:

any scripts provided by the package for use in the shell are not available in this state, and in that sometimes crucial way it's not effectively "installed"
behavior differs depending on whether pip install or python -m pip install is used, which is totally unexpected and without any obvious rationale
when the user explicitly pip uninstalls, what does that mean, if not actually uninstalling?
if pip freeze --all doesn't list all installed packages, what does it do?
if "uninstalling" doesn't uninstall, and then a different version of the package is additionally installed, which one will the user be using at any given time?
simply touching a fake file in the current directory can break installation of packages from PyPI

$ git clone [email protected]:jazzband/pip-tools
$ cd pip-tools
$ python -m venv venv
$ . ./venv/bin/activate
$ pip install -U pip
...
Successfully installed pip-20.0.2
$ pip install -e .
...
Successfully installed click-7.1.1 pip-tools six-1.14.0
$ pip uninstall -y pip-tools
...
  Successfully uninstalled pip-tools-4.5.2.dev18+g1d0883f
$ python -m pip install pip-tools
Requirement already satisfied: pip-tools in /home/andy/Code/piptrubbl/pip-tools (4.5.2.dev18+g1d0883f)
...
$ pip-sync --help
zsh: command not found: pip-sync
$ pip freeze
click==7.1.1
six==1.14.0
$ pip freeze --all 
click==7.1.1
pip==20.0.2
setuptools==41.2.0
six==1.14.0
$ pip install pip-tools
...
Successfully installed pip-tools-4.5.1
$ deactivate
$ rm -rf venv pip_tools.egg-info
$ python -m venv venv
$ . ./venv/bin/activate
$ touch pip_tools.egg-info
$ pip install -U pip
...
Successfully installed pip-20.0.2
$ python -m pip install pip-tools
...
ValueError: ("Missing 'Version:' header and/or PKG-INFO file at path: /home/andy/Code/piptrubbl/pip-tools/pip_tools.egg-info", pip-tools [unknown version] (/home/andy/Code/piptrubbl/pip-tools))
$ pip install pip-tools
...
Successfully installed click-7.1.1 pip-tools-4.5.1 six-1.14.0

IMO this behavior is not sane, on multiple counts.

uranusjr · 2020-03-11T01:56:05Z

C/f #2926 (of which this is essentially a duplicate).

AndydeCleyre · 2020-03-11T02:36:54Z

@uranusjr

I see that it also relates to dealing with with local egg info dirs, and with divergent behavior depending on how pip is invoked. But notably, that primary issue (pip freeze including the current directory) is not reproduced by the session in my previous comment. Unless I misunderstand that linked issue altogether.

AndydeCleyre · 2020-04-05T15:58:05Z

As far as I can see, this is now fixed by #7955 ! 🎆

sbidoul · 2020-04-05T21:43:44Z

Yes, I think we can close this one too.

triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label May 30, 2019

chrahunt added the state: needs discussion This needs some more discussion label Jul 21, 2019

triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label Jul 21, 2019

chrahunt mentioned this issue Jul 28, 2019

python -m pip install skip sdist installation #4621

Closed

akaps mentioned this issue Aug 7, 2019

Python imports akaps/nand2tetris#39

Merged

chrahunt mentioned this issue Sep 21, 2019

pip/_internal/locations.py not respecting site.ENABLE_USER_SITE #7055

Closed

chrahunt mentioned this issue Dec 13, 2019

Pip uninstall doesn't find installed editable package that pip show knows about #6895

Closed

AndydeCleyre mentioned this issue Mar 10, 2020

Requirement already satisfied by egg but package not installed #3710

Closed

sbidoul mentioned this issue Apr 5, 2020

Don't list packages in current directory #7955

Merged

sbidoul closed this as completed Apr 5, 2020

lock bot added the auto-locked Outdated issues that have been locked by automation label May 5, 2020

lock bot locked as resolved and limited conversation to collaborators May 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

egg-info in local directory makes pip think package is installed #6558

egg-info in local directory makes pip think package is installed #6558

pv commented May 30, 2019 •

edited

Loading

uranusjr commented May 31, 2019

pv commented May 31, 2019 via email

pv commented May 31, 2019 via email

pfmoore commented May 31, 2019

pv commented May 31, 2019 via email

chrahunt commented Jul 21, 2019

AndydeCleyre commented Mar 10, 2020

uranusjr commented Mar 11, 2020

AndydeCleyre commented Mar 11, 2020

AndydeCleyre commented Apr 5, 2020

sbidoul commented Apr 5, 2020

egg-info in local directory makes pip think package is installed #6558

egg-info in local directory makes pip think package is installed #6558

Comments

pv commented May 30, 2019 • edited Loading

uranusjr commented May 31, 2019

pv commented May 31, 2019 via email

pv commented May 31, 2019 via email

pfmoore commented May 31, 2019

pv commented May 31, 2019 via email

chrahunt commented Jul 21, 2019

AndydeCleyre commented Mar 10, 2020

uranusjr commented Mar 11, 2020

AndydeCleyre commented Mar 11, 2020

AndydeCleyre commented Apr 5, 2020

sbidoul commented Apr 5, 2020

pv commented May 30, 2019 •

edited

Loading