Implement PEP 660 hooks in setuptools/build_meta.py

[ichard26]

Summary of changes

Implements the hooks required to support PEP 660 -- Editable installs for pyproject.toml based builds (wheel based). Resolves #2816.

dholth's work which I referenced extensively: https://github.com/dholth/setuptools_pep660

I chose the root .pth editable strategy as it "just works" for implicit namespace packages and also looked easier haha. While most of the code is (hopefully) rather straightforward, it is quite verbose as unforunately setuptools doesn't have much in terms of wheel utilties.

There's also a somewhat sketchy hack used to acquire the distribution location so I can put the right path to ${dist-name}.pth in the wheel. Since the PEP 517 is isolated from the command framework, it's pretty hard querying information from the various setup.py commands ran. The shim added captures the egg_info instance when setup.py dist_info runs so the build_editable hook can query the location (i.e. egg_base).

One limitation of this commit is that the build_editable hook completely ignores the metadata_directory parameter and violates the guarantee the hook shall produce a (editable) wheel with identical metadata. Given that it seems no major backends is currently following through with this promise this is hopefully fine for now?

Pull Request Checklist

• Changes have tests (nope, still gotta figure this out)
• Test out more non-standard packaging configurations
• Test out packages with extensions
• Create wheel_directory if doesn't exist already when build_editable is called
• News fragment added in changelog.d/.

---

FWIW this is my first contribution to setuptools so I'm a little nervous I've messed everything up. Hopefully this isn't too bad :)

[ichard26]

https://github.com/pypa/setuptools/runs/4180806139?check_suite_focus=true#step:5:204

WARNING: Built editable for setuptools is invalid: Wheel has unexpected file name: expected '58.5.3.post20211111.post-20211111', got '58.5.3.post20211111.post'

Well apparently dist.version is not a valid version?

>>> import setuptools.dist
>>> setuptools.dist.Distribution._normalize_version("58.5.3.post20211111.post-20211111")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ichard26/programming/oss/setuptools/setuptools/dist.py", line 484, in _normalize_version
    normalized = str(packaging.version.Version(version))
  File "/home/ichard26/programming/oss/setuptools/setuptools/_vendor/packaging/version.py", line 277, in __init__
    raise InvalidVersion("Invalid version: '{0}'".format(version))
setuptools.extern.packaging.version.InvalidVersion: Invalid version: '58.5.3.post20211111.post-20211111'

[ichard26]

Alright, implementing this patch:

diff --git a/setuptools/build_meta.py b/setuptools/build_meta.py
index a24eb0f7..4e377916 100644
--- a/setuptools/build_meta.py
+++ b/setuptools/build_meta.py
@@ -49,7 +49,7 @@ import setuptools.command.egg_info
 import distutils
 
 import pkg_resources
-from pkg_resources import parse_requirements
+from pkg_resources import parse_requirements, safe_name, safe_version
 
 __all__ = ['get_requires_for_build_sdist',
            'get_requires_for_build_wheel',
@@ -313,7 +313,11 @@ class _BuildMetaBackend(object):
         dist = pkg_resources.DistInfoDistribution.from_location(
             location, dist_info, metadata=metadata
         )
-        wheel_name = f'{dist.project_name}-{dist.version}-ed.py3-none-any.whl'
+        # Seems like the distribution name and version attributes aren't always
+        # 100% normalized ...
+        dist_name = safe_name(dist.project_name).replace("-", "_")
+        version = safe_version(dist.version).replace("-", "_")
+        wheel_name = f'{dist_name}-{version}-ed.py3-none-any.whl'
         wheel_path = os.path.join(wheel_directory, wheel_name)
         with zipfile.ZipFile(wheel_path, 'w') as archive:
             archive.writestr(f'{dist.project_name}.pth', location)

Avoids the weird error message from pip (due to misreading the not normalized distribution name), but then predictably uncovers this issue:

  Building editable for setuptools (pyproject.toml) ... done
  Created wheel for setuptools: filename=setuptools-58.5.3.post20211111.post_20211111-ed.py3-none-any.whl size=9822 sha256=050f33119d1f68404c8eed8007f9870d5cc49e40fcd4dd3edb5c4d7e1adfc50f
  Stored in directory: /tmp/pip-ephem-wheel-cache-tb3eq2fm/wheels/29/a6/99/6cd5ddea9b773be0e848527e54dd03b1092fe671cb9b561ab5
  WARNING: Built editable for setuptools is invalid: Metadata 1.2 mandates PEP 440 version, but '58.5.3.post20211111.post-20211111' is not
Failed to build setuptools
ERROR: Could not build wheels for setuptools, which is required to install pyproject.toml-based projects

Is this is a short-coming in my implementation or can I safety assume non-PEP-440 versions are not my problem 🙂 .. although in that case I need to figure out to have dist_info return a PEP 440 version for setuptools itself ...

[ichard26]

TIL that egg_info and dist_info can return different versions.

dist-info	egg-info
Metadata-Version: 2.1 Name: setuptools Version: 58.5.3.post20211111.post-20211111	Metadata-Version: 2.1 Name: setuptools Version: 58.5.3.post20211111

bdist_wheel calls egg_info and uses that as its metadata source (which is how it avoids this error). I guess I might have to use egg_info instead then, but AFAICT PEP 660 only specifies .dist-info? I also kinda don't want to rewrite the code to use .egg-info instead :p

[pganssle]

Considering that PEP 660 requires backends to choose how to implement editable installs and the fact that pip will fall back to setup.py develop, I'm inclined to say that when we implement PEP 660 we should try and clean up some of the major failings of the current editable install mode. Specifically, I think we should use a "strict" approach, where we only install what the config files says to install, rather than the current approach where we YOLO it by dumping all the relevant paths onto the PYTHONPATH.

I don't see any rush to implement this, so it'd be better to do it right than to do it quickly.

[jaraco]

TIL that egg_info and dist_info can return different versions.

That's definitely unintentional. dist_info is preferable. Let's figure out why the metadata is incorrect and work on correcting that and supporting that format.

[pradyunsg]

One thing I'd like to suggest is that __legacy__ should basically mimic python setup.py develop as it stands today (i.e. use the pth file). That would minimize distruption to existing projects that pip uses setup.py for directly, when pip switches over to using the pyproject.toml based isolated build environments.

[ichard26]

Yeah yeah I know this implementation needs a bunch of work, my apologies for misguiding y'all expectations. I'm quite busy right now but I can look into revisiting this implementation soon.

[layday]

Specifically, I think we should use a "strict" approach, where we only install what the config files says to install, rather than the current approach where we YOLO it by dumping all the relevant paths onto the PYTHONPATH.

Can you provide some direction on how this would be accomplished?

[pfmoore]

I feel that the initial pass of PEP 660 support should replicate the current behaviour, as that's what users expect and are used to. And I agree with @pradyunsg that the __legacy__ backend should definitely use the .pth mechanism, as the whole point of the __legacy__ backend is to replicate current behaviour.

Further iterations could then investigate other implementation mechanisms that make different trade-offs than the current .pth file mechanism does.

[pganssle]

Can you provide some direction on how this would be accomplished?

I think it's pretty obvious how this could be accomplished. I seem to remember there was a proof of concept implementation kicking around when PEP 662 was being discussed. I seem to remember thinking it was easier than I thought it was going to be, but I can't find a copy of it. Maybe @gaborbernat was the one who did it?

I feel that the initial pass of PEP 660 support should replicate the current behaviour, as that's what users expect and are used to. And I agree with @pradyunsg that the __legacy__ backend should definitely use the .pth mechanism, as the whole point of the __legacy__ backend is to replicate current behaviou

I disagree on both counts here. The legacy behavior is fundamentally broken, and since PEP 660 forces us to make choices for our users rather than letting them make their own trade-offs, we should choose correctness. setuptools already has a confusing morass of behaviors that are very hard to intuit. People are constantly shipping tests and examples packages, and even I have trouble explaining the various interactions between package_data and include_package_data and MANIFEST.in.

I think people expect pip install -e . to install the package as it would be installed normally, but with live updating of the code. They also expect new files to show up automatically. Unfortunately, these two options are in conflict, and we need to make a choice. Considering that editable installs not adding new files automatically fails loudly while editable installs doing something subtly incorrect fails quietly and most likely after a wheel has been released, it seems like the obvious choice is to fix this long-standing bug as part of the PEP 660 implementation.

It doesn't seem likely that we need to worry about backwards compatibility here because this will primarily affect interactive workflows rather than automated ones — some people may be relying on the accidental feature that more stuff is included in editable installs than in normal installs for their production systems, but this is simply a bug in their setup.py that they are accidentally working around by installing via a mechanism that doesn't respect the metadata they actually specified.

[pfmoore]

The legacy behavior is fundamentally broken, and since PEP 660 forces us to make choices for our users rather than letting them make their own trade-offs, we should choose correctness.

You can use the config_settings mechanism to allow users to select between options. (I'd be happy to support improving pip's UI for passing config settings to the backend, if that turns out to be needed to make this work). So having an option to use the legacy behaviour certainly doesn't preclude adding other behaviours. And if, as you say, a better option would be an unequivocal improvement, I doubt anyone would object if you changed the default once that approach was added (particularly if they had a config option to request the older behaviour, should they for some reason want it).

[pganssle]

You can use the config_settings mechanism to allow users to select between options. (I'd be happy to support improving pip's UI for passing config settings to the backend, if that turns out to be needed to make this work). So having an option to use the legacy behaviour certainly doesn't preclude adding other behaviours. And if, as you say, a better option would be an unequivocal improvement, I doubt anyone would object if you changed the default once that approach was added (particularly if they had a config option to request the older behaviour, should they for some reason want it).

Yeah, I definitely agree that PEP 660 provides a clunky backend-specific non-standard way to allow users to choose their editable install mode. I don't think we can really rely on that or expect people to discover or use it, particularly when, as I mentioned, the problems with the current editable install mode are subtle and quiet, whereas the users will discover the problems with their assumptions quickly and loudly with the strict mode.

I seem to recall during the PEP 660 discussions that everyone in favor of PEP 660 seemed fine with the idea that backends should choose what their install mode would be, and since "strict" is really the only mode that makes sense for setuptools (as I mentioned many times), it is a natural consequence of PEP 660 that setuptools would use strict installation mode automatically.

Depending on the implementation, it might make sense for there to be a dedicated command class that can be used to customize the behavior of editable installs.

[pradyunsg]

If that's the route you want to take, I'll flag that it could be disruptive to users in situations where they can't easily control the setuptools version (like pip's isolated build environments).

To that end, I encourage doing things to reduce user pain due to it: documenting the failure modes with some amount of clarity in the documentation, clear communication about the change with known regressions clearly stated, and so on. If you can do some sort of opt-in/opt-out via build_config or an environment variable, that would help users mitigate their immediate breakage as well, reducing the amount of toxicity you'd need to deal with when the release goes out.

[pganssle]

If that's the route you want to take, I'll flag that it could be disruptive to users in situations where they can't easily control the setuptools version (like pip's isolated build environments).

To some extent I agree that PEP 660 deliberately chooses to move the choices about trade-offs into the hands of backend implementers, something I was very concerned with, and one of the main reasons I thought PEP 660 was a bad design.

On the other hand, the PEP 660 proponents had at least some point when they said that this is much less of a problem for editable installs than for normal installs, because this sort of thing would only ever be a problem if you are installing from source code, which you can easily change. I can't think of any examples where someone would be doing an editable install using pip's isolated build environment in such a way that non-strict installations would matter and they wouldn't be able to easily modify pyproject.toml to specify a setuptools version. In an isolated build environment you are doing a fresh install of the package every time, so there's no reason to do it in editable mode in the first place.

To that end, I encourage doing things to reduce user pain due to it: documenting the failure modes with some amount of clarity in the documentation, clear communication about the change with known regressions clearly stated, and so on. If you can do some sort of opt-in/opt-out via build_config or an environment variable, that would help users mitigate their immediate breakage as well, reducing the amount of toxicity you'd need to deal with when the release goes out.

I agree that we should document the failure modes. I think opt-in wouldn't achieve our goals because, as I said, the current behavior is broken in such a way that most users do not know it is broken, and so they will never opt-in. Opt-out might make sense, but I don't think it's critical, and I don't think we need to bend over backwards to accommodate the fact that PEP 660 is poorly designed.

[pradyunsg]

In an isolated build environment you are doing a fresh install of the package every time, so there's no reason to do it in editable mode in the first place.

I believe you're misunderstanding -- the moment you release PEP 660 support in setuptools, pip's isolated build environments will start picking up the latest setuptools version and use that for performing editable installs for users. If you change behaviours there vs setup.py develop, the end user has no way to make pip use a different version of setuptools (other than modifying the package's pyproject.toml). This could be exceedingly disruptive to folks who depend on the "bad" parts of the existing behaviour, who will be not-that-great to interact with when they show up on the issue tracker (and will leave with a sour taste for Python as a whole).

I think opt-in wouldn't achieve our goals because, as I said, the current behavior is broken in such a way that most users do not know it is broken, and so they will never opt-in. Opt-out might make sense, but I don't think it's critical, and I don't think we need to bend over backwards to accommodate the fact that PEP 660 is poorly designed.

No, the opt-in -> switch default with opt-out -> drop opt-out is not to deal with "PEP 660 is bad actually". It's to deal with the fact that you want to make a backwards incompatible change as part of implementing the setuptools' PEP 660 support.

Even if setuptools implemented PEP 660 via a .pth file first and then moved to adopt the strict mode, the users would still only make noise when the adoption of the strict mode happens -- because that is the problematic bit (i.e. the breaking change). Even in that situation, I would suggest adopting the gradual transitional approach to give users time to provide early feedback/adapt after the change.

[pganssle]

I believe you're misunderstanding -- the moment you release PEP 660 support in setuptools, pip's isolated build environments will start picking up the latest setuptools version and use that. If you change behaviours there vs setup.py develop, the end user has no way to make pip use a different version of setuptools (other than modifying the package's pyproject.toml). This could be exceedingly disruptive to folks who depend on the "bad" parts of the existing behaviour, who will be not-that-great to interact with when they show up on the issue tracker (and will leave with a sour taste for Python as a whole).

I understand what you mean but I'm saying that I'm struggling to imagine a situation where someone would be using an editable install in an isolated build environment. Editable installs are for interactive workflows where you are making changes on the fly. pip does a fresh install of every package every time it is invoked, so editable installs should be no different from regular installs. The "strict" part that's in contention is basically because the way setup.py develop works, you can add new files after installation and have them picked up automatically, but that will never happen in an isolated build environment, which does the install and then immediately uses the package, then immediately deletes the enviornment.

AFAICT, the only reason someone would want to use an editable install during their build is if they messed up their package configuration such that normal installs install something totally different from editable installs, in which case their package is broken and it should raise an error! It's a bug, not a feature, that using an editable install can gloss over problems with your package configuration.

No, the opt-in -> switch default with opt-out -> drop opt-out is not to deal with "PEP 660 is bad actually". It's to deal with the fact that you want to make a backwards incompatible change as part of implementing the setuptools' PEP 660 support.

The reason this has to do with PEP 660 being bad is that PEP 660 opted to standardize a method where there is no standard way for end-users to configure their installation mode. If there were, setuptools wouldn't even have had to make this choice at all, and front-ends could choose to fix the bug or present different installation options. They didn't, and so setuptools is forced to make a choice of doing it one way or the other.

Now that we know that there's no option for us to leave it up to end-users, we have to make a choice as to whether or not we fix the bug, and I think we should fix it. People already hate setuptools and packaging in general (with good reason!), so we may as well lean into the pain and at least give them as good a product as we can.

[layday]

I think it's pretty obvious how this could be accomplished. I seem to remember there was a proof of concept implementation kicking around when PEP 662 was being discussed. I seem to remember thinking it was easier than I thought it was going to be, but I can't find a copy of it. Maybe @gaborbernat was the one who did it?

I don't know which PoC you are remembering, but I'd created frontend-editables at the time which relied on symlinking to enforce 'strictness'. You can't have symlinks in a wheel, so how would you do this within the framework of PEP 660?

[layday]

I understand what you mean but I'm saying that I'm struggling to imagine a situation where someone would be using an editable install in an isolated build environment.

pip creates an isolated build environment to call PEP 660 hooks inside of. It will install the latest available version of setuptools in the isolated build environment.

the end user has no way to make pip use a different version of setuptools (other than modifying the package's pyproject.toml).

Surely passing --no-isolation and pre-installing setuptools would work?

[pganssle]

I don't know which PoC you are remembering, but I'd created frontend-editables at the time which relied on symlinking to enforce 'strictness'. You can't have symlinks in a wheel, so how would you do this within the framework of PEP 660?

I don't think PEP 660 or PEP 662 differed in how things could be implemented, because you can use a PEP 662 approach and then implement the code the front-end was expecting to implement on the backend. The problem I had always had in providing setuptools proofs of concept in the (admittedly small) time I had to work on it was that setuptools mixes installation and marshalling tasks, so there was no obvious point in the workflow where you had a list of the location of everything to be copied / zipped up and the locations it had to be copied to. PEP 662 was saying that backends should construct that list and hand it to front-ends to do the installation. In PEP 660 the backend has to do all the installation work as well as all the marshalling work, but it's fundamentally the same task.

Once you have the list of files to install, you can use one of the strategies outlined in the PEP for installation. Probably the most cross-platform of which is a proxy platform using import hooks in a .pth file, but there's also an option involving the backend building a directory full of symlinks and I think adding that directory to sys.path in a .pth file.

[pfmoore]

The reason this has to do with PEP 660 being bad is that PEP 660 opted to standardize a method where there is no standard way for end-users to configure their installation mode.

I'm confused by this statement. PEP 660 has the standard config_options mechanism for this¹. Backends may choose not to offer multiple strategies via that route, but how is that different from the PEP 662 case where the frontend offers only a single strategy? If PEP 662 had been accepted, I'd be extremely surprised if pip implemented multiple strategies - we'd likely implement a .pth file approach and stop there. (And before you claim that's sour grapes on the part of people who support PEP 660, remember that the pip maintainers have basically all said that we consider .pth files to be sufficient, so why wouldn't we implement that strategy?)

Footnotes

1. I'll happily concede that the UI for config_options sucks at the moment, but that is "just" a tool issue. I've said on a number of occasions that I'd be happy with improving how pip exposes config_options, it's just that we've never had a reason to prioritise doing so. ↩

[layday]

Once you have the list of files to install, you can use one of the strategies outlined in the PEP for installation. Probably the most cross-platform of which is a proxy platform using import hooks in a .pth file, but there's also an option involving the backend building a directory full of symlinks and I think adding that directory to sys.path in a .pth file.

The proxy module trick doesn't work with namespace packages. It might not be an option for setuptools. Creating a tree of symlinks out of band and adding them to the Python path would work, but we'll have gone from drop this one file in site packages to multi-step installation and uninstallation which is unspecified and outside the control of pip. In my opinion, PEP 660 isn't equipped to handle anything this intricate; trying to force an (oversized ;) wheel into a square hole is only gonna end in frustration.

[pganssle]

pip creates an isolated build environment to call PEP 660 hooks inside of. It will install the latest available version of setuptools in the isolated build environment.

Surely any users who are installing a package can edit their own pyproject.toml to add version caps to setuptools, no? The people in favor of PEP 660 seemed to think that "edit config files in the package" was an acceptable UI for end users, and if you are actually relying on the old behavior then of course you should modify your build dependencies to reflect that. Pradyun seemed to be indicating that there were situations where end users don't actually control what setuptools version they get, but would somehow also rely on this.

The proxy module trick doesn't work with namespace packages. It might not be an option for setuptools. Creating a tree of symlinks out of band and adding them to the Python path would work, but we'll have gone from drop this one file in site packages to multi-step installation and uninstallation which is unspecified and outside the control of pip. In my opinion, PEP 660 isn't equipped to handle anything this intricate; trying to force an (oversized ;) wheel into a square hole is only gonna end in frustration.

I haven't had much time to look into it, to be honest. How do other backends deal with this?

If it's true that PEP 660 backends can't even implement this correctly, then it's a much worse spec than I thought.

I'm confused by this statement. PEP 660 has the standard config_options mechanism for this1. Backends may choose not to offer multiple strategies via that route, but how is that different from the PEP 662 case where the frontend offers only a single strategy?

There is a standard way to pass options to backends, but the options themselves are specific to the backend, and I'm not even sure that all front-ends have a way to pass options via that mechanism. The point is that we have to go outside of the standard to allow end-users to configure this if it's possible at all.

I'm not saying we can't do it, just that it's a messy hack and we shouldn't expect users to discover it. The default should be strict installations.

[layday]

Surely any users who are installing a package can edit their own pyproject.toml to add version caps to setuptools, no?

I don't think upper-capping setuptools in a project's build metadata solely for editable installs in development is something we'd want to encourage or even suggest.

The people in favor of PEP 660 seemed to think that "edit config files in the package" was an acceptable UI for end users, and if you are actually relying on the old behavior then of course you should modify your build dependencies to reflect that.

I don't know who thought that was an acceptable UI but even if they did, editing editable-specific metadata is fundamentally different from editing project-wide metadata to satisfy a specific purpose.

I haven't had much time to look into it, to be honest. How do other backends deal with this?

They use .pth files and a few of them use Paul's editables library for non-namespace packages. Nobody has attempted to do strict editable installs that I know.

[pfmoore]

In my opinion, PEP 660 isn't equipped to handle anything this intricate

It's not clear to me that any strategy is perfect here. Symlinks don't work on all platforms, .pth files expose too much, import hooks don't work with namespace packages. If someone can develop a strategy that actually implements whatever setuptools is trying to achieve with "correct behaviour"¹, we'd be having a very different conversation. So far, though, no-one has produced anything that people can agree on as "working".

If it's true that PEP 660 backends can't even implement this correctly, then it's a much worse spec than I thought.

Define correct. This is not a rhetorical question, neither PEP 660 nor PEP 662 precisely defined an editable install, both using wording something along the lines of

some changes to the project code in the local source tree will become effective without the need for a new installation step. At a minimum, changes to the text of non-generated files that existed at the installation time should be reflected upon the subsequent import of the package
[...]
some differences may be visible, such as the presence of additional files (compared to a typical installation), either in the source tree or the interpreter's installation path

That's from PEP 662, PEP 660 was similar, more restrictive in some ways, less in others. Again, if we had a precise definition of what it meant to do an editable install, we'd be having a much different conversation. As it stands, a .pth file implements something that fits either PEP's definition. So I can only assume that you have a much more restrictive definition in mind.

But at this point we're just arguing all of the points that were covered in the PEP discussion. And we're not going to end up at any different conclusion. Even if we declared PEP 660 a failure, and abandoned it, we'd still only have symlinks, import hooks and .pth files as implementation strategies, we'd still not have a formal definition of an "editable install", and setuptools wouldn't be any more able to implement their ideal "not broken" approach to editable installs than they currently are.

Footnotes

1. Whatever that means, as there's no actual definition of what an editable install is apart from "what setup.py develop did", which is a .pth file with all its warts and limitations. ↩

[layday]

In my opinion, PEP 660 isn't equipped to handle anything this intricate

It's not clear to me that any strategy is perfect here.

I don't think that any of the strategies mentioned so far is perfect either.

When I said that symlinking would work, I meant that it would have the intended outcome. That it can't be relied on on Windows makes it a lot less than "perfect".

When I said PEP 660 isn't equipped to handle anything this intricate, I meant that the delivery medium for PEP 660 is the wheel and pre-installing packages at some unspecified location on disk, and presumably not recording them in the distribution's metadata, is very much not something that the PEP invites.

[pradyunsg]

Somehow I haven't stated this already: As much as I agree that the strict mode will be better than the status quo in terms of a UX, I don't think that coupling the changeover with PEP 660 is a good idea.

If there were, setuptools wouldn't even have had to make this choice at all, and front-ends could choose to fix the bug or present different installation options. They didn't, and so setuptools is forced to make a choice of doing it one way or the other.

Sure. Then frontends would need to make the choice, or expose the users to a mechanism to pick on their end.

And they'd need to be mindful of backwards compatibility concerns if they made the change that you want to force setuptools to make, in the same manner that setuptools needs to right now. I can tell you that pip would not do this work, and just have the pth based approach as the default for quite a while; if we'd gone with a design like that.

---

It isn't an "issue" with the PEP, it's a consequence of the fact that you absolutely want to change behaviours in the initial implementation. Making drastic, backwards incompatible changes without putting effort to reduce the transition costs/churn is how you burn goodwill with users. None of the proposals magically avoid the backwards compatibility concerns in any way or remove the change management/communication work needed in anyway; they shift who has to do that.

[pradyunsg]

Surely passing --no-isolation and pre-installing setuptools would work?

Yes, if you also manage the rest of the build-time dependencies of the project. It would be a viable workaround, but certainly not something that I'd consider to be "smooth".

Surely any users who are installing a package can edit their own pyproject.toml to add version caps to setuptools, no?

Use an older version if the newer version breaks things for you, is certainly an option. I'm not sure I'm quite on board with the idea of having users pin their setuptools versions in pyproject.toml, but... sure. It's strictly worse than providing a knob to only change the editable behaviour.

The people in favor of PEP 660 seemed to think that "edit config files in the package" was an acceptable UI for end users, and if you are actually relying on the old behavior then of course you should modify your build dependencies to reflect that.

I don't know who thought that was an acceptable UI

+1

[pradyunsg]

the options themselves are specific to the backend

Right now, setuptools is the only backend that wants to change how editable installs work.

I'm not even sure that all front-ends have a way to pass options via that mechanism.

Only one popular option does not: pip. As @pfmoore said:

I'd be happy to support improving pip's UI for passing config settings to the backend, if that turns out to be needed to make this work

The point is that we have to go outside of the standard to allow end-users to configure this if it's possible at all.

No? Unless you want to standardise the name of the config setting that only setuptools is going to use, I don't see what else can be done TBH. It's not outside of the standard, but it is backend specific configuration.

Or maybe you want to have a separate dedicated option to consider it "in standard", I still don't think that has much value.

As far as I can tell, exactly 1 backend is considering implementing something other than the existing pth file mechanism. Heck, exactly one backend maintainer wants to implement something other than the existing pth file mechanism: @pganssle.

I don't see how standardisation would have had much value here.

[layday]

As far as I can tell, exactly 1 backend is considering implementing something other than the existing pth file mechanism. Heck, exactly one backend maintainer wants to implement something other than the existing pth file mechanism: @pganssle.

PEP 660 did not standardise the nature of "editableness". If backends are only expected to use .pth files then the PEP should have standardised (path-only) .pth files and discussions like this one would have been avoided. pip doesn't care about anything else, other backends don't care about anything else, so why bother with anything else?

[pfmoore]

PEP 660 did not standardise the nature of "editableness".

Nowhere in any of the discussions on editable installs, did we achieve any sort of consensus on what an "editable install" should be. People seemed comfortable with leaving it implementation defined. Was that wrong? Maybe as a community we should be more strict over how much implementation defined behaviour we're willing to accept in PEPs?

If backends are only expected to use .pth files then the PEP should have standardised (path-only) .pth files and discussions like this one would have been avoided. pip doesn't care about anything else, other backends don't care about anything else, so why bother with anything else?

Maybe. But hindsight is easy. Why did no-one suggest that at the time? Both PEP 660 and PEP 662 left the definition implementation-defined, and a huge amount of the discussion was around ways of implementing "better" editable installs. Would anyone have got consensus if they'd said "let's just standardise a way of injecting a .pth file, like setup.py develop currently does"? I don't know.

Given that PEP 660 is currently still provisional¹, we could propose a change to define editable installs precisely, requiring a .pth based approach. So if you genuinely believe that's the right thing to do, I suggest you do that. If nothing else, it might help pin down a consensus on exactly what the packaging community does want.

Footnotes

1. But be quick, the vote to make it final is in progress right now 🙂 ↩

[gaborbernat]

PEP 660 did not standardise the nature of "editableness". If backends are only expected to use .pth files then the PEP should have standardised (path-only) .pth files and discussions like this one would have been avoided.

Would anyone have got consensus if they'd said "let's just standardise a way of injecting a .pth file, like setup.py develop currently does"? I don't know.

I remember this coming up in between the discussions but got quickly discarded as too restrictive. I for one would be against such a narrow scoped definition. Note I'm not a backend maintainer though. pth files are cheap to implement IMHO but wrong on many levels, so if we're going to standardize something we should aim (or at least allow) for something better. I also don't entirely agree that no backend wants to do anything else, I think they'll do it eventually, but the cheapest solution is the pth, so naturally, that will get the quickest implementation.

People seemed comfortable with leaving it implementation defined.

I believe this was the only way we could reach a consensus because everyone had a different implementation in mind at the end of the day or a different definition of what editable installation nature qualifies.

[layday]

Was that wrong?

I don't know if that was wrong. But maybe we should show some lenience if that's a door we intentionally left open instead of treating setuptools like an anomaly because they are thinking about trying something different.

So if you genuinely believe that's the right thing to do [...]

Please re-read my response in context.

[pfmoore]

Ah, apologies, I did indeed misread your comment.

[ichard26]

I no longer have the motivation to work on this given the .pth approach isn't acceptable. Figuring out how to basically rewrite the entire editable install mechanism as my first contribution to setuptools is just not feasible or a good idea.