python -m pip install skip sdist installation

[ikalnytskyi]

• Pip version: 9.0.1
• Python version: 2.7.13, 3.6.1, (probably any)
• Operating system: macOS, Linux

Description:

If I try to install an sdist via python -m pip install dist/foo-1.2.0.tar.gz, it says Requirement already satisfied and skip installation. Full output:

Processing ./dist/foo-1.2.0.tar.gz
  Requirement already satisfied (use --upgrade to upgrade): foo==1.2.0 from file:///Users/ihor.kalnytskyi/sandbox/pipbug/dist/foo-1.2.0.tar.gz in /Users/ihor.kalnytskyi/sandbox/pipbug
Building wheels for collected packages: foo
  Running setup.py bdist_wheel for foo ... done
  Stored in directory: /Users/ihor.kalnytskyi/Library/Caches/pip/wheels/44/47/c1/fcde10591dd0c57f282ff43314245dda409a5219120a7583ba
Successfully built foo

If I try to install it via regular pip install dist/foo-1.2.0.tar.gz, everything's alright. Full output:

Processing ./dist/foo-1.2.0.tar.gz
Building wheels for collected packages: foo
  Running setup.py bdist_wheel for foo ... done
  Stored in directory: /Users/ihor.kalnytskyi/Library/Caches/pip/wheels/44/47/c1/fcde10591dd0c57f282ff43314245dda409a5219120a7583ba
Successfully built foo
Installing collected packages: foo
Successfully installed foo-1.2.0

The expected result would be to have the same behavior for pip and python -m pip executions.

A little bit of debug

By some reasons, if we call pip install the pkg_resources.get_distribution on line req_install.py:894 returns nothing, hence the package is needed to be installed. On the other hand, if we call python -m pip install, the very same line returns a valid distribution hence the package won't be installed.

Unfortunately, I wasn't able to debug any further.. :(

[benoit-pierre]

This is because when running pip with python -m pip the current directory is added to sys.path, and the presence of foo.egg-info make the package available to the environment (check the output of python -m pip list | grep foo versus pip list | grep foo).

[ikalnytskyi]

@benoit-pierre I tried to remove current working directory from sys.path as the first action in pip/__main__.py, but it didn't help. :(

So you are absolutely right, when I removed foo.egg-info in current working directory, everything works as expected and python -m pip install dist/foo-1.2.0.tar.gz installs foo successfully.

Any idea what we can do to align behavior between pip install and python -m pip install? Sometime the later command is a way to go as the first one didn't work (e.g. virtualenv created somewhere by Jenkins, so the path to python interpreter in pip's shebang is greater than 128 characters and running pip fails with bad interpreter: no such file or directory).

[pradyunsg]

I'm not sure why this is happening... Is the output of pip --version and python -m pip --version different?

[ikalnytskyi]

@pradyunsg nope, they are the same:

$ pip --version
pip 9.0.1 from /Users/ihor.kalnytskyi/sandbox/pipbug/venv/lib/python3.6/site-packages (python 3.6)

$ python -m pip --version
pip 9.0.1 from /Users/ihor.kalnytskyi/sandbox/pipbug/venv/lib/python3.6/site-packages (python 3.6)

I think it's somehow related to detecting foo.egg-info in current working directory (as @benoit-pierre pointed), but I don't understand why removing current working directory from sys.path in pip/__main__.py (as early as possible) doesn't help here.

[benoit-pierre]

This must be done before pkg_resources is imported:

python -c "import sys; sys.path.remove(''); from pkg_resources import load_entry_point; sys.exit(load_entry_point('pip', 'console_scripts', 'pip')())"

[chrahunt]

The current discussion on this issue is in #6558. I will close this in favor of that, since it looks like the same case.