requirements-txt-fixer sorting breaks pip when using --extra-index-url

[yourcelf]

When using an additional non-pypi package index along with pypi, the requirements.txt file will have declarations at the top indicating those sources:

-i https://pypi.org/simple
--extra-index-url https://example.com/myindex

or with the main index url spelled out:

--index-url https://pypi.org/simple
--extra-index-url: https://example.com/myindex

However, the naive sorting of entries that requirements-txt-fixer does will put the "extra" above "index" for both of these forms, since --e sorts alphabetically before --i or -i.

--extra-index-url: https://example.com/myindex
--index-url https://pypi.org/simple

... and pip doesn't like that. It will ignore the --extra-index-url declaration that comes before --index-url and fail to find any packages found in the extra index.

To resolve this, the sorter will need to do something smarter than just sorted(...) for these initial flags.

[asottile]

this feels like a bug in pip, not in the requirements-txt-sorter

also --index-url https://pypi.org/simple is unnecessary, just drop that and it'll work

[yourcelf]

OK. :) That makes this hook unusable with pipenv lock -r > requirements.txt, as pipenv adds it. I'll file a bug with pip, and in the mean time we'll have to live without requirements-txt-fixer.

[asottile]

note also that -i pypi.org --extra-index-url is inherently insecure. someone could upload a package to public pypi masking your package and suddenly your builds are trashed (or worse, you install malware)

[yourcelf]

Thanks for that note. Some more on that security concern: pypa/pip#8606

In short: this security issue exists regardless of whether pypi.org or the extra index is listed first -- pip offers no guarantee of priority for which index gets used to resolve a package. The only way to ensure you get a non-shadowed package is to use only a single trusted index, duplicate and control names of packages across all indexes, or switch to local indexes like https://www.devpi.net/.