[FR] Implement PEP 660 -- Editable Installations

[pradyunsg]

What's the problem this feature will solve?

PEP 660 standardises the mechanism that setup.py develop uses to make "editable" packages.

Currently, pip needs to implement fairly horrible hacks to make legacy setup.py develop builds work. It'd be great if setuptools could implement PEP 660, which would be the last item from setuptools' end before pip can start the process of removing the legacy directly-call-setup.py build system.

Describe the solution you'd like

PEP 660's build_editable hook gets implemented in aetuptools.

Alternative Solutions

Not doing this.

I imagine this will mean that either (a) pip continues to maintain editable installs for setuptools or (b) pip drops the legacy build system entirely and users of setuptools projects will no longer be able to make editable installations.

I don't think (b) is happening any time soon, and it'd likely be best to implement the hook here.

Additional context

IIUC, this has been implemented in setuptools-pep660 already. It might not be a lot of work to migrate the work done there into seriptools directly.

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jaraco]

See the discussion and dholth/setuptools_pep660#1.

[ichard26]

I took a stab at this but I can't seem to find a reliable way to query the importable base of the packages / modules (i.e. the parent directory of {name}.dist-info generated from an in-tree build). For what it's worth this is my first time working in the setuptools codebase so there's probably something I'm missing :)

[ichard26]

Status update: after adding some more runtime patching of setuptools internals to query all of the information I need I got an implementation that can do an editable install for at least a simple single pure-Python module distribution and a src-style pure-Python multi-package distribution (psf/black, duh 🙂) 🎉

main...ichard26:pep-660

I need to test other situations and packaging configurations, but I've ran out of time for today :)

[tacaswell]

I think that preserving editable installs for packages with c / c++ / cython installations via some mechanism is cirtical (I say with my Matplotlib and h5py hats on).

I would be perfectly happy to absorb changes to our setup.py / pyproject.toml / built incantation, but there needs to be some path forward.

[ichard26]

I think that preserving editable installs for packages with c / c++ / cython installations via some mechanism is cirtical (I say with my Matplotlib and h5py hats on).

Right, unfortunately I don't really have much experience in this domain¹. It is my understanding that PEP 660's way of handling extensions is by mandating a in-tree and in-place extension build (effectively python setup.py build_ext -i) before the editable wheel is produced. This obviously means the editable install / the in-place extension build will have to be redone to see changes in the extensions. Other than that I don't see how using a root .pth file (pointing to the importable base which by now should have the built extensions in-tree)² changes things as it's basically how python setup.py develop worked anyway, just codified ^^

I'd love input and insight from people like yourself regarding extensions for both my own understanding and implementation testing!

Footnotes

1. I can only speak with my psf/black (and maybe pypa/bandersnatch) hats on 🙂 ↩

2. This is the implementation strategy I chose as it seems the simplest haha ↩

[pradyunsg]

For completeness, making an update here with how I understand the state of affairs to be.

With #1688 closed, this is now a blocker for getting editable installs working for setuptools' new pyproject.toml only projects.

In #2872, there has been extensive discussion about the implementation approach that setuptools can take for this. There seems to be no clear concensus about how to implement the PEP 660 hooks -- there's significant pushback from the maintainers for implementing the PEP 660 hooks (that enable editable installations, without a setup.py file) using a mechanism similar to setup.py develop today.

[EwoutH]

Thanks a lot for the effort so far! Is there any update it’s progress?

[abravalheri]

Hi @EwoutH I left a PR with a possible approach in #3265 (so the other setuptools maintainers can see if it matches their expectation). Please feel free to have a look and even submit a review.

[pradyunsg]

This is now one of the blockers for pip (and ensurepip) excluding setuptools in new virtual environments. The latest version of pip is capable of using the build isolation when setuptools is not available within the environment, even when there’s no pyproject.toml file.

[pradyunsg]

Closing this out, since #3265 implemented this. It'll be in a release soon-enough.

It still has an experimental flag on it, but it works and that's the part that I care about. I've now seen multiple people point to this issue to say that editable installs don't work in Python -- the only case that doesn't is setuptools without a setup.py, which is going to be solved once that PR reaches main. :)

[samuelcolvin]

Thanks so much @pradyunsg.

[djhoese]

which is going to be solved once that PR reaches main.

Is there a milestone or pull request for when that will happen? I see the PR was merged to the feature/pep660, but don't see anything else using that branch. I'm not familiar with the setuptools workflow so let me know if I'm missing something. I'm one of those people that would like to use pyproject.toml without a setup.py and it seems I need that PR.

[astrojuanlu]

I couldn't find an open PR for the feature/pep660 branch either, however @abravalheri announced a month ago that they were conducting preliminary tests https://discuss.python.org/t/help-testing-pep-660-support-in-setuptools/16904?u=astrojuanlu

[abravalheri]

@djhoese, @astrojuanlu, you can track the progress in #3488. I am considering that the round of community feedback is reaching its end, so I plan to do a release soon (in another week or so).