New resolver changes the order of index urls used with --extra-index-url

[Lordshinjo]

Environment

• pip version: 20.3.3
• Python version: 3.9.1
• OS: macOS 10.15.7

Description
When installing/downloading packages with the new resolver, and if using --extra-index-url, packages are always downloaded from the url given by the extra index if possible, regardless of their availability in the base index (pypi.org by default).
(Note that I am only testing with 1 extra index)

I did not see anything mentioning this in the "Changes to the pip dependency resolver in 20.3 (2020)" section of the user guide or in the changelog, so I think this is a regression.

Expected behavior
If a package is available in the base index, it is downloaded from there.
The extra index is only used if the package is not available in the base index.
I believe this is the behavior of the legacy resolver.

How to Reproduce

1. Have a PyPI mirror available.
2. Download a package present in both indexes with the new resolver.
3. The package is downloaded from the url given by the extra index.
4. Download the same package with the legacy resolver.
5. The package is downloaded from the url given by the pypi.org index.

Output

I'll take numpy==1.19.5 as a random example of package to download.

$ pip download numpy==1.19.5 --extra-index-url=http://[snipped-host]/repository/pypi/simple --trusted-host=[snipped-host] --no-cache-dir
Looking in indexes: https://pypi.org/simple, http://[snipped-host]/repository/pypi/simple
Collecting numpy==1.19.5
  Downloading http://[snipped-host]/repository/pypi/files.pythonhosted.org/https/packages/e1/c5/65b2f257a154c7fabc1895b435e7863a1f0bb1769d3c28f1500976e090ee/numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl (15.6 MB)
     |████████████████████████████████| 15.6 MB 3.3 kB/s
Saved ./numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl
Successfully downloaded numpy

$ rm numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl

$ pip download numpy==1.19.5 --extra-index-url=http://[snipped-host]/repository/pypi/simple --trusted-host=[snipped-host] --no-cache-dir --use-deprecated=legacy-resolver
Looking in indexes: https://pypi.org/simple, http://[snipped-host]/repository/pypi/simple
Collecting numpy==1.19.5
  Downloading numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl (15.6 MB)
     |████████████████████████████████| 15.6 MB 8.3 MB/s
Saved ./numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl
Successfully downloaded numpy

[uranusjr]

You beleive wrong. The behaviour has always been like this. See #6045.

[Lordshinjo]

That's not true, considering the behavior of the legacy resolver does behave as I described, per my output.
#3454 describes an issue where the extra index is used first (with the legacy resolver) because the it uses the highest version of either index. When there is no version mismatch, my description seems correct.

I do understand that the behavior I want is described in #6045 though.

[pfmoore]

Pip treats all indexes equally, and which index is used to retrieve a package is arbitrary. So the behaviour you are seeing is as expected. Note that every copy of numpy 1.19.5 should be equivalent, so it should be irrelevant which index is used. If the copies are not equivalent, they should have different version numbers.

If you want indexes to be ordered, or otherwise don't want the "all indexes are equivalent" behaviour pip provides, you can use @uranusjr's tool simpleindex as noted in the linked issue.