extra-index-url doesn't work as suggested in "Using TestPyPI" guide

[akkana]

Environment

• pip version: 18.1
• Python version: 3.7.5
• OS: Linux, Ubuntu 19.10

Description
I'm trying to test dependencies for my package in TestPyPI before uploading to "real" PyPI. The guide at:
https://packaging.python.org/guides/using-testpypi/
says I can do this with:
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple your-package

So after uploading to TestPyPI with twine, I create a new virtualenv, activate it, pip install wheel, then:
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pytopo==1.6.1
This fails with:
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://test.pypi.org/simple/pycairo/

Which dependency it fails on varies; it's not always pycairo, but the rest of the message is always the same. Note that it's looking for dependencies in test.pypi.org, not pypi.org. If I just pip install pycairo (or whatever other dependency it fails on), that always works, so the packages are in pypi though not in test.pypi.

Expected behavior
Based on the "Using TestPyPI" guide and the pip man page, I expected that this would look for pytopo version 1.6.1 in TestPyPI, find it, try to satisfy its dependencies, and when the dependencies weren't found in TestPyPI it would look for them in PyPI, install them all then proceed to install pytopo from TestPyPI.

How to Reproduce

python3 -m venv /tmp/test3env
source /tmp/test3env/bin/activate
pip install wheel
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pytopo==1.6.1

Output

Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://test.pypi.org/simple/pycairo/

[maxpearl]

I've seen the same issue. Ubuntu 19.04, python 3.7.3, pip 18.1

[uranusjr]

Would you mind posting the output of -v? (Or maybe -vv if that’s viable.) It would very likely provide much detail why things don’t work.

[akkana]

Here's -vv output. It says " 2 location(s) to search for versions of pycairo:" and lists test.pypi and pypi; but then it looks in test.pypi and errors out without looking in the second place.

(test3env) <charon>- pip install -vv --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pytopo==1.6.1 
Created temporary directory: /tmp/pip-ephem-wheel-cache-yzn7r5io
Created temporary directory: /tmp/pip-req-tracker-1z1yli6y
Created requirements tracker '/tmp/pip-req-tracker-1z1yli6y'
Created temporary directory: /tmp/pip-install-o9zrgduf
Looking in indexes: https://test.pypi.org/simple/, https://pypi.org/simple
Collecting pytopo==1.6.1
  2 location(s) to search for versions of pytopo:
  * https://test.pypi.org/simple/pytopo/
  * https://pypi.org/simple/pytopo/
  Getting page https://test.pypi.org/simple/pytopo/
  Looking up "https://test.pypi.org/simple/pytopo/" in the cache
  Current age based on date: 16
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 0
  Starting new HTTPS connection (1): test.pypi.org:443
  https://test.pypi.org:443 "GET /simple/pytopo/ HTTP/1.1" 304 0
  Analyzing links from page https://test.pypi.org/simple/pytopo/
    Found link https://test-files.pythonhosted.org/packages/36/6f/2b8040689a9004c14a3ba025b754dc584e8e48fa2eec50c9ccc20e942f13/pytopo-1.4.tar.gz#sha256=24341c4cb5f2747309a16d3f940a27c63b8a82bd4ac3aab68182d4071ffa956c (from https://test.pypi.org/simple/pytopo/), version: 1.4
    Found link https://test-files.pythonhosted.org/packages/01/df/bdbbc0eb8381997e3dc91449ebd6456378b11791a31c54892be040ab093e/pytopo-1.6-py3-none-any.whl#sha256=f071da480e82f17c66d184b08bfbae92d1e451bd90a8eb81e8d74f400804900c (from https://test.pypi.org/simple/pytopo/), version: 1.6
    Found link https://test-files.pythonhosted.org/packages/5f/5c/c5a065a2d558e8a64c11c8dc6fa71f6bfeda5e0656e71511d174842bd47c/pytopo-1.6.tar.gz#sha256=d92c9f6417b6cdfa57f343345d044cdbe9f3a2bd6186fb71d93c45678f41d863 (from https://test.pypi.org/simple/pytopo/), version: 1.6
    Found link https://test-files.pythonhosted.org/packages/1d/dd/f46722bed2296676053b333a97babf05cb31b20e4a621161d44399a3ed18/pytopo-1.6.1-py3-none-any.whl#sha256=38b1a925ff9eb60a53d8d22fcac410b7a053aceab6f85af0c4b423891e70a7b4 (from https://test.pypi.org/simple/pytopo/), version: 1.6.1
    Found link https://test-files.pythonhosted.org/packages/b4/c9/71b68f9b4e305d33dda395091d70b0c57ba782fc6f7d7e4c281bb8138338/pytopo-1.6.1.tar.gz#sha256=b11aec14d289f0527ea986044b675db2a3aafd78e62044c53fd8a426434393be (from https://test.pypi.org/simple/pytopo/), version: 1.6.1
  Getting page https://pypi.org/simple/pytopo/
  Looking up "https://pypi.org/simple/pytopo/" in the cache
  Current age based on date: 16
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 0
  Starting new HTTPS connection (1): pypi.org:443
  https://pypi.org:443 "GET /simple/pytopo/ HTTP/1.1" 304 0
  Analyzing links from page https://pypi.org/simple/pytopo/
    Found link https://files.pythonhosted.org/packages/34/5e/0333e79fc3f9365ab3fdb54ab9c9cca8788d59fc14698053706c690a797f/pytopo-1.4.tar.gz#sha256=25480cd9d40dd8716664436e7f44272ebdc767063ea2fa528bc10af2d8b9e5ee (from https://pypi.org/simple/pytopo/), version: 1.4
    Found link https://files.pythonhosted.org/packages/01/df/bdbbc0eb8381997e3dc91449ebd6456378b11791a31c54892be040ab093e/pytopo-1.6-py3-none-any.whl#sha256=f071da480e82f17c66d184b08bfbae92d1e451bd90a8eb81e8d74f400804900c (from https://pypi.org/simple/pytopo/), version: 1.6
    Found link https://files.pythonhosted.org/packages/5f/5c/c5a065a2d558e8a64c11c8dc6fa71f6bfeda5e0656e71511d174842bd47c/pytopo-1.6.tar.gz#sha256=d92c9f6417b6cdfa57f343345d044cdbe9f3a2bd6186fb71d93c45678f41d863 (from https://pypi.org/simple/pytopo/), version: 1.6
    Found link https://files.pythonhosted.org/packages/3b/c1/039ee2fb76b6d75d87bef378de530dd93f0f809c8ccf6e3a5cc22398cdaa/pytopo-1.6.1-py3-none-any.whl#sha256=d046e06c8c3e0aa6685caea0b59140aaa412e2f190d46b35a1e95b92169474c2 (from https://pypi.org/simple/pytopo/), version: 1.6.1
    Found link https://files.pythonhosted.org/packages/a0/41/303a9baa8d143ee09eeb4b02fb2b740eb5c06656d4a78a4c9d16dc208f26/pytopo-1.6.1.tar.gz#sha256=bd6c2bf87bf962c142096e99c23a890f0526602c47eed9e3fd7e6777812456cd (from https://pypi.org/simple/pytopo/), version: 1.6.1
  Using version 1.6.1 (newest of versions: 1.6.1)
  Created temporary directory: /tmp/pip-unpack-ew8u3_co
  Looking up "https://test-files.pythonhosted.org/packages/1d/dd/f46722bed2296676053b333a97babf05cb31b20e4a621161d44399a3ed18/pytopo-1.6.1-py3-none-any.whl" in the cache
  Current age based on date: 16
  Starting new HTTPS connection (1): test-files.pythonhosted.org:443
  https://test-files.pythonhosted.org:443 "GET /packages/1d/dd/f46722bed2296676053b333a97babf05cb31b20e4a621161d44399a3ed18/pytopo-1.6.1-py3-none-any.whl HTTP/1.1" 304 0
  Using cached https://test-files.pythonhosted.org/packages/1d/dd/f46722bed2296676053b333a97babf05cb31b20e4a621161d44399a3ed18/pytopo-1.6.1-py3-none-any.whl
  Downloading from URL https://test-files.pythonhosted.org/packages/1d/dd/f46722bed2296676053b333a97babf05cb31b20e4a621161d44399a3ed18/pytopo-1.6.1-py3-none-any.whl#sha256=38b1a925ff9eb60a53d8d22fcac410b7a053aceab6f85af0c4b423891e70a7b4 (from https://test.pypi.org/simple/pytopo/)
  Added pytopo==1.6.1 from https://test-files.pythonhosted.org/packages/1d/dd/f46722bed2296676053b333a97babf05cb31b20e4a621161d44399a3ed18/pytopo-1.6.1-py3-none-any.whl#sha256=38b1a925ff9eb60a53d8d22fcac410b7a053aceab6f85af0c4b423891e70a7b4 to build tracker '/tmp/pip-req-tracker-1z1yli6y'
  Removed pytopo==1.6.1 from https://test-files.pythonhosted.org/packages/1d/dd/f46722bed2296676053b333a97babf05cb31b20e4a621161d44399a3ed18/pytopo-1.6.1-py3-none-any.whl#sha256=38b1a925ff9eb60a53d8d22fcac410b7a053aceab6f85af0c4b423891e70a7b4 from build tracker '/tmp/pip-req-tracker-1z1yli6y'
Collecting pycairo (from pytopo==1.6.1)
  2 location(s) to search for versions of pycairo:
  * https://test.pypi.org/simple/pycairo/
  * https://pypi.org/simple/pycairo/
  Getting page https://test.pypi.org/simple/pycairo/
  Looking up "https://test.pypi.org/simple/pycairo/" in the cache
  No cache entry available
  https://test.pypi.org:443 "GET /simple/pycairo/ HTTP/1.1" 404 13
  Status code 404 not in [200, 203, 300, 301]
Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
  File "/tmp/test3env/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 338, in run
    resolver.resolve(requirement_set)
  File "/tmp/test3env/lib/python3.7/site-packages/pip/_internal/resolve.py", line 102, in resolve
    self._resolve_one(requirement_set, req)
  File "/tmp/test3env/lib/python3.7/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/tmp/test3env/lib/python3.7/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
    self.require_hashes
  File "/tmp/test3env/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 218, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/tmp/test3env/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 164, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/tmp/test3env/lib/python3.7/site-packages/pip/_internal/index.py", line 572, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/tmp/test3env/lib/python3.7/site-packages/pip/_internal/index.py", line 530, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/tmp/test3env/lib/python3.7/site-packages/pip/_internal/index.py", line 675, in _get_pages
    page = self._get_page(location)
  File "/tmp/test3env/lib/python3.7/site-packages/pip/_internal/index.py", line 793, in _get_page
    return _get_html_page(link, session=self.session)
  File "/tmp/test3env/lib/python3.7/site-packages/pip/_internal/index.py", line 147, in _get_html_page
    resp.raise_for_status()
  File "/tmp/test3env/share/python-wheels/requests-2.21.0-py2.py3-none-any.whl/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://test.pypi.org/simple/pycairo/
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-1z1yli6y'
zsh: exit 1     pip install -vv --index-url https://test.pypi.org/simple/ --extra-index-url  

[uranusjr]

This looks very much like #6373. This HTTPError case should be caught if it originated from pip’s vendored requests copy (pip._vendor.requests), but your log shows pip is using a non-vendored copy instead (/tmp/test3env/share/python-wheels/requests-2.21.0-py2.py3-none-any.whl).

The problem here is Ubuntu messed up when de-bundling pip’s vendored dependencies. I don’t think there’s much pip can do at this time. Relevant Ubuntu bug report: https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1833229

[uranusjr]

One thing we probably can do is to raise/catch our own HTTP exceptions instead of requests.HTTPError if we can. This would mean to replace raise_for_exception() usages, and implement our own variant. It’s not particularly difficult, but I’m not sure whether it’d be worthwhile.

[kenahoo]

I'm hitting this today in AWS EC2 Ubuntu images. Do we know whether there are certain versions of Ubuntu this affects or doesn't affect? Trying to figure out a workaround.

[uranusjr]

@kenahoo You would likely have better luck asking on an Ubuntu bug tracker or forum instead, where people have much better understanding to Ubuntu internals.

[richard534]

I've a workaround for ubuntu18.04, might work for someone else:

apt-get install --yes software-properties-common
add-apt-repository ppa:ci-train-ppa-service/3690
apt-get install --yes python-pip=9.0.1-2.3~ubuntu1.18.04.2~ubuntu18.04.1~ppa202002141134

If you follow the discussion on the ubuntu bug report ref'd above you'll find that repo I added.

[tl24]

From my tests this seems to be broken on pip versions 20.3 and higher. If I install version of pip from 20.2.4 or lower it works correctly. I've tried 20.2.4, 20.2.3 and those work. I'm able to replicate in a python:3.8-buster docker container. However it appears to only fail when you have a requirements file with the primary index specified in the file and the extra index url specified on the command line.

# requirements.txt
-i https://mycompany.pkgs.visualstudio.com/abc/_packaging/feedname/pypi/simple/
boto3

# setup.sh
EXTRA_INDEX=https://user:[email protected]/abc/_packaging/feedname/pypi/simple/
python -m venv ~/venvs/test
source ~/venvs/test/bin/activate
pip install pip==21.0.1
pip install -r requirements.txt --extra-index-url $EXTRA_INDEX -vv --no-input

[uranusjr]

@tl24 The situation is explained very well here: #9484 (comment)

To put the situation another way, neither behaviour is broken. Which index is chosen is literally a toss-up, and pip <20.3 gives the behaviour you want because you’re lucky, not because you’re doing the right thing.

[ekamperi]

Any update on this one folks? I just hit it. Thanks!