Support for custom indices

[woodruffw]

For the time being, our MVP is scoped to just support for PyPI. However, it's worth considering what we'd require in order to support custom package indices (whether PyPI mirrors or entirely separate private indexes).

Some potential points of issue:

• Our Add a RequirementSource to our dependency source API. #23 adaptor will need to be sufficiently generic, adapting either PyPI or another index under the hood as configured.
• We might need custom index support from pip-api? This is unclear, since our only use of pip-api in these contexts would be for requirements parsing, and the requirements file itself shouldn't make any references to the index.

[dhuckins]

would this issue include something like the pip extra index url?
example: my team uses both the public pypi and a private index for internal packages

[woodruffw]

Yep, something like that (in general, we're trying to make pip-audit's CLI as close to pip's as possible, so whatever CLI flags pip uses for custom indices we should use as well.)

[dhuckins]

if I wanted to contribute that would be (to the best of your knowledge) this repo or pip-api to support? (probably both since would need to pass in the arg here but if you have knowledge of what would need to be changed it would be appreciated)

[di]

@dhuckins Note that pip-audit uses vulnerability reports for packages on PyPI, it won't have any vulnerability data for your private internal packages. What are you hoping to achieve by including your private index?

[dhuckins]

Collaborator

good point. let me ask the question in a different way in a new issue

[vajirag]

I am using local PyPI mirror (behind a firewall) with pip tools.

Until custom index support is sorted (what is described above), is there any recommendation on how to use pip-audit behind a firewall? (apart from setting https_proxy)

[woodruffw]

@vajirag Unfortunately not. pip-audit itself won't have any way to handle this case until #193; until then, you should handle the proxying behavior at a higher level.

[tetsuo-cpp]

@woodruffw @di

Ok, I've been doing a bit of reading about --index-url and --extra-index-url today. It's a tricky one so I'll record what I've learnt here.

Setting --index-url to a private index is straightforward but where it gets tricky is when we add multiple indexes by using the --extra-index-url flag. This is a pretty normal use case since users often want to have PyPI in addition to their own private index.

When running pip install with multiple indexes, pip treats all indexes equally (that is, there is no order of priority). So pip will check all indexes for a package that best satisfies the requirement. If there are multiple packages across multiple indexes with the same name and version, the result is undefined and pip could install any one of them. There's an interesting discussion about this over at pypa/pip#8606.

I think in order to replicate this behaviour, we'll need our resolvelib provider to actually generate candidates for every single index. I was thinking that perhaps we could simplify this behaviour and just generate the candidates from the first index that has the package that we're looking for. However, the same package can exist across multiple indexes and one of the later indexes could have the optimal candidate. It's important that we reproduce identical behaviour to pip in this instance because if we don't, our -r mode will audit a different set of packages than what pip install -r would install onto an environment.

Does this seem reasonable?

[di]

@tetsuo-cpp Seems reasonable to me!