sdist not installable / Cargo workspace with non-default members with glob

[stinodego]

Bug Description

Recently I have updated Polars by moving the PyO3 crate into the workspace: pola-rs/polars#12256

Maturin still builds the wheels fine, but running maturin sdist now generates a broken sdist.

This is partly fixed by #1841 and using the latest maturin version. But there is another issue related to non-default workspace members.

Stack trace:

$ pip install --force-reinstall --verbose dist/*.tar.gz
Using pip 23.3.1 from /home/stijn/code/polars-fork/.venv/lib/python3.11/site-packages/pip (python 3.11)
Processing ./dist/polars-0.19.13rc1.tar.gz
  Running command pip subprocess to install build dependencies
  Collecting maturin@ git+https://github.com/PyO3/maturin.git@main
    Cloning https://github.com/PyO3/maturin.git (to revision main) to /tmp/pip-install-pawd50jg/maturin_0730556581e149eaa4a2439e51161708
    Running command git clone --filter=blob:none --quiet https://github.com/PyO3/maturin.git /tmp/pip-install-pawd50jg/maturin_0730556581e149eaa4a2439e51161708
    Resolved https://github.com/PyO3/maturin.git to commit e57405b3b35ffdf79d9b5052f64464752c434b28
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Building wheels for collected packages: maturin
    Building wheel for maturin (pyproject.toml): started
    Building wheel for maturin (pyproject.toml): finished with status 'done'
    Created wheel for maturin: filename=maturin-1.3.1-cp311-cp311-linux_x86_64.whl size=6330955 sha256=a98bdc4c23946ad991b74f1df3e5d4ed1c342a6febdb7b1e7d93b7557522c4f8
    Stored in directory: /tmp/pip-ephem-wheel-cache-0dhfhqal/wheels/5f/7d/94/65a37bf2e4cb1ffc20aaa0d28c6f4a1b99ce36e5db5fb5f3ef
  Successfully built maturin
  Installing collected packages: maturin
  Successfully installed maturin-1.3.1
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  error: failed to load manifest for workspace member `/tmp/pip-req-build-me64z0oz/examples/*`

  Caused by:
    failed to read `/tmp/pip-req-build-me64z0oz/examples/*/Cargo.toml`

  Caused by:
    No such file or directory (os error 2)
  💥 maturin failed
    Caused by: Cargo metadata failed. Does your crate compile with `cargo build`?
    Caused by: `cargo metadata` exited with an error:
  Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/pip-modern-metadata-wbnwxhcd', '--interpreter', '/home/stijn/code/polars-fork/.venv/bin/python']' returned non-zero exit status 1.
  Checking for Rust toolchain....
  Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-wbnwxhcd --interpreter /home/stijn/code/polars-fork/.venv/bin/python`
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/stijn/code/polars-fork/.venv/bin/python /home/stijn/code/polars-fork/.venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpv18opbcy
  cwd: /tmp/pip-req-build-me64z0oz
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Your maturin version (maturin --version)

main branch

Your Python version (python -V)

3.11.2

Your pip version (pip -V)

23.3.1

What bindings you're using

pyo3

Does cargo build work?

• Yes, it works

If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash /)?

• Yes

Steps to Reproduce

See pola-rs/polars#12370 for the workaround I implemented.

To reproduce the issue, check out the Polars repository BEFORE the workaround commit and run:

1. maturin sdist --manifest-path py-polars/Cargo.toml --out dist
2. pip install --force-reinstall --verbose dist/*.tar.gz

Cargo build works fine, maturin also builds the wheels just fine. It's just the sdist that is broken.

[stinodego]

Seems like this issue was actually fixed in #1841, but I am running into another issue... updating soon.

[stinodego]

I believe the issue is due to non-default workspace members with a path defined as a glob - in our case, examples/*.

Looking at the traceback, I can see:

failed to read `/tmp/pip-req-build-me64z0oz/examples/*/Cargo.toml`

Obviously this path does not exist - it did not expand the glob properly.