pip fails to install remote dependency when it is a .whl and follows PEP 440

[retpolanne]

Environment

• pip version: pip 18.1
• Python version: Python 2.7.12
• OS: Ubuntu 16.04.1 - Linux 4.15.0-42-generic

I'm using virtualenv on the version 15.2.0

Description

When following PEP 508 and PEP 440's direct references and adding a dependency to a file, say requirements.txt like this:

boto @ https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl

doing a pip install -r requirements.txt gives this error.

Requirement u'boto @ git+https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl' looks like a filename, but the file does not exist
Processing ./boto @ git+https:/files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/home/vinicyusmacedo/Documents/Development/myapp/boto @ git+https:/files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl'

Expected behavior

Pip should download and install the dependency.

pip install -r tests.txt
Collecting boto@ https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl# from https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl# (from -r tests.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl
Installing collected packages: boto
Successfully installed boto-2.49.0

Note that including a # to the end of the URL works just fine.

How to Reproduce

1. Create a file (say requirements.txt) with some URLs to remote .whl files following PEP 440's direct references
2. Run pip install -r requirements.txt
3. An error occurs.

[retpolanne]

Some thoughts

Is this some expected behavior? That what the PEP 440 says:

Public index servers SHOULD NOT allow the use of direct references in uploaded distributions. Direct references are intended as a tool for software integrators rather than publishers.

So, I believe I shouldn't be referencing a .whl file from somewhere like pypi.org?
Does that apply to private pypi repositories?

If that's not the case, I noticed that is_url on download.py simply splits an URL and uses the part before a : as the scheme. On the PEP 440 examples, the URL includes the package name @ the URL. So the scheme won't fall in any of the expected schemes.

def is_url(name):
    # type: (Union[str, Text]) -> bool
    """Returns true if the name looks like a URL"""
    if ':' not in name:
        return False
    scheme = url.split(':', 1)[0].lower()
    return scheme in ['http', 'https', 'file', 'ftp'] + vcs.all_schemes

[uranusjr]

I believe it is allowed to point a requirement to a file on the index server; what PEP 440 says is that the server itself should not allow it. The problem here is more likely that pip’s name parser is not smart enough to do the right thing.

[retpolanne]

@uranusjr got it, it is not up to pip to make such restriction.

[cjerdonek]

@uranusjr Quick question: is this issue related to your comment here: #6097 (comment)? If so, how is it related?

[retpolanne]

@cjerdonek I believe that's the same issue (sorry for duplicating it).

[cjerdonek]

@vinicyusmacedo No problem, that's okay. Just to clarify, does this mean your PR will completely resolve issue #6097? We can just make a note on that issue.

[retpolanne]

@cjerdonek it will, since it splits the @ from the requirement line and treats it as any other URL requirement.

I forgot to mention, but I had issues with setuptools too because of this bug, pretty much as in #6097 and #6306

[javabrett]

What is the status here - is it an accepted bug and looking-for-help, or work-underway/under-review, or issue-likely-rejected/won't-fix? This is biting me so just wondering whether it is time to pitch-in.

[retpolanne]

@javabrett it is currently under review. #6203
Ping @cjerdonek @uranusjr