pip download --no-deps tries to use PEP517 so badly it is not usable to download stuff

[hroncok]

pip version

21.0.1

Python version

3.9.2

OS

Fedora Linux 33, x86_64

Additional information

No response

Description

Let's say I want to download an sdist from PyPI with pip download:

$ pip download --no-deps --no-binary :all: --no-build-isolation dnspython==2.1.0
...
ModuleNotFoundError: No module named 'poetry'

Or even:

$ pip download --no-deps --no-binary :all: --no-build-isolation --no-use-pep517 dnspython==2.1.0
Collecting dnspython==2.1.0
  Using cached dnspython-2.1.0.zip (389 kB)
ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of poetry.masonry.api in pyproject.toml

I can use:

$ pip download --no-deps --no-binary :all: dnspython==2.1.0
Collecting dnspython==2.1.0
  Using cached dnspython-2.1.0.zip (389 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Saved ./dnspython-2.1.0.zip
Successfully downloaded dnspython

However, that takes 3 minutes. I assume because it installs poetry and all the transitive deps from sdists.

Expected behavior

I expect pip download --no-deps --no-binary :all: to just download the sdist without trying to build it.

How to Reproduce

See Description.

Output

See Description.

Code of Conduct

• I agree to follow the PSF Code of Conduct

[pfmoore]

The problem is that the only way to be sure that a sdist actually provides the version you specify is to build it. Yes, we could rely on the sdist filename, but it's not technically reliable, and we'd need to special-case stuff to make it work.

If and when build backends start including PEP 643 style metadata in sdists, to a level where it's worth the effort to check it before trying a build, we could use that to avoid the build step where the data is available statically. But I'm not even sure if any tools have implemented PEP 643 yet...

To be honest, though, if you want to just download a sdist from PyPI, pip probably isn't the tool you want. It's not that hard to query the PyPI JSON interface for the sdist url, and wget it. If I were doing this often enough that manually downloading via the web interface was insufficient, that's what I'd do.

The following does the basic job, although I'm sure you could easily write something as good or better yourself:

import json
from urllib.request import urlopen
import sys

package, version = sys.argv[1:]

with urlopen(f"https://pypi.org/pypi/{package}/json") as f:
    data = json.load(f)

for file in data["releases"][version]:
    if file["packagetype"] == "sdist":
        print(file["url"])

[hroncok]

The problem is that the only way to be sure that a sdist actually provides the version you specify is to build it.

I assumed PyPI can be trusted to serve the right sdist version.

For example https://github.com/pradyunsg/vendoring/blob/master/src/vendoring/tasks/license.py downloads sdists form PyPI this way.

Thanks for the API tip, that's what we'll most likely end up doing. We can even "simply" try to download https://files.pythonhosted.org/packages/source/d/dnspython/dnspython-2.1.0.tar.gz and if that fails, try again with zip.

However, the behavior here was really surprising for me, hence I've reported it.

[pfmoore]

Yeah, agreed it's non-obvious. If nothing else, a clarification in the docs of why we do this might be good.

[uranusjr]

I assumed PyPI can be trusted to serve the right sdist version.

Well actually… (OK sorry let’s not do that.)

There are some previous discussions on this, see #7995 (and the multiple references in it). I think it’s a valid argument that distribution validation is probably counter-intuitive for pip download (and potentially pip wheel), or at least when --no-deps is specified.

The biggest roadblock (aside from coming up with a rule that makes sense) is implementation; validation should only be skipped on very specific subcommand-option combinations, and it’s not trivial to pass all the needed context all the way down to where the validation is done. Definitely technically feasible, but falls very much in the “those who can don’t care; who care can’t” territory.

[pradyunsg]

FWIW, the grand #6607 plan was to make that easier to do, by moving the generation of metadata/whatnot into cleaner transition steps, that would occur when attempting certain types of access (eg: relying on Verdon of wheels, relying on versions from sdist name more cleanly, relying on versions from metadata more easily etc).

[pradyunsg]

Closing as a duplicate of #7995.