Unable to upgrade setuptools with adopted distutils #8761
Comments
|
cc @pganssle , @ncoghlan , @pfmoore I see the last implicated code in pip is in locations.distutils_scheme. I'm pretty sure what's happening here is pip is getting distutils from the installed version of setuptools, and when that version of setuptools is removed, the command discovery no longer works. More importantly, the stdlib version of distutils is intended to be deprecated, so pip will need to find an alternative. Here are some ideas:
|
This feels like it could be a long-term solution, it's hardly a band-aid if
I think our short-term solution should be setuptools-based if possible, because we don't have any good way to declare an incompatibility with older versions of I don't totally understand the potential impact here, but I'm assuming it's just performance at the time you do |
|
Another possible temporary mitigation:
|
|
We've done a lot of work to decouple pip from setuptools (as a build backend). It seems like a step backward to just re-couple ourselves to get distutils functionality. Particularly as we'd end up vendoring setuptools, which vendors other stuff that we already vendor (like packaging) increasing our already-large footprint further. So -1 on vendoring setuptools from me. What's the "official" migration strategy for removing distutils from the stdlib? Are any tools that use distutils expected to add a new dependency on setuptools? I've not seen much discussion of this. I note that packaging depends on My feeling is that pip should continue to use distutils on the basis that it's in the stdlib, until it's officially removed from there. I'd be fine with moving to a non-distutils solution, and I would strongly prefer that as the solution over vendoring setuptools. But I don't feel we need to do anything urgently - until distutils is formally removed from the stdlib, I'd expect to be able to use it. If there's an issue with upgrading setuptools, that's really a matter for how the migration out of the stdlib is handled, and one for setuptools to handle smoothly in the first instance. That's not to say I'm not willing to support the process, but we shouldn't take on significant maintenance burden to do so. |
I mostly disagree with this idea — people should be attempting to abandon I also expect that the idea that Given that there's a lot of uncertainty about the way forward for dropping |
I agree 100% with this conceptually, but the reality is there are systems everywhere depending on distutils, and pip gets blamed when it tries to “fix” that (#5193 comes to mind). |
I'm not sure I see how this is relevant. That bug seems to be "debian's weird patching and removal of This migration is going to be complicated and we should try to disentangle as much of the deep relationships between |
To clarify, what I meant was that we should handle moving off distutils based on the idea that "distutils is going to be dropped from the stdlib". So we have to deal with that, and while it may be a long way off we don't want to leave it till the last minute, conversely we shouldn't treat it as an emergency that needs an immediate band-aid solution. We should focus on getting the solution right, because the timescale for pip is about distutils no longer being in the stdlib, not about the intermediate stage where setuptools is trying to expose distutils while it is also still in the stdlib (sorry if I mischaracterise what's going on here, I haven't really been following the migration). And my view is that the right long-term solution is to replace the functionality with something that doesn't need yet another large vendored library just to provide one function. So we should be looking for either a stdlib-based solution, or something that we add to a tool support library like packaging, and definitely not pull in a full build backend just to get distribution layout information. And with all that in mind, we have to prioritise the resource we have and dealing with the consequences of distutils disappearing is way down the list right now. |
|
Would it be a good idea to make migrating off |
Quite likely, in particular as we get closer to any sort of hard deadline. While there's still a relatively large amount of time, I still have some hope that someone might feel inclined to do it as volunteer work. (That's one reason I want to avoid putting undue time pressure on the work - nothing drains enthusiasm more than cries of "we need this yesterday" 🙁) |
…on 50.0.0
Aren Cambre (3):
pypa/setuptools#2302
pypa/setuptools#2302
Add changelog text for pypa/setuptools#2302
Jason R. Coombs (12):
Re-enable distutils patch by default.
Update tests to reflect new expectation.
Bypass .pth loader when distutils is loaded from pip. Workaround for pypa/pip#8761.
When pip is imported, unload any existing distutils and disable the DistutilsMetaFinder.
Replace hard-coded tox working dir with the environment variable, honoring non-standard working directories.
Refactor to use lookups and consolidate behaviors.
Remove Python 2 compatibility
Remove dependency on six from packaging.
Remove six as a vendored dependency.
Remove six from 'extern' as vendored.
Update changelog.
Bump version: 49.6.0 → 50.0.0
…0 to version 50.0.0
Aren Cambre (3):
pypa/setuptools#2302
pypa/setuptools#2302
Add changelog text for pypa/setuptools#2302
Jason R. Coombs (12):
Re-enable distutils patch by default.
Update tests to reflect new expectation.
Bypass .pth loader when distutils is loaded from pip. Workaround for pypa/pip#8761.
When pip is imported, unload any existing distutils and disable the DistutilsMetaFinder.
Replace hard-coded tox working dir with the environment variable, honoring non-standard working directories.
Refactor to use lookups and consolidate behaviors.
Remove Python 2 compatibility
Remove dependency on six from packaging.
Remove six as a vendored dependency.
Remove six from 'extern' as vendored.
Update changelog.
Bump version: 49.6.0 → 50.0.0
|
Gentle reminder that in less than a year, Python 3.12 alpha will be released without distutils. |
|
Hmm, I think this is done, actually. Since 21.3, pip running on Python 3.10 or later will use sysconfig by default; the distutils code path is still present to support 3.9 and lower (and as a stopgap measure for Python redistributors not yet able to implement a working sysconfig). I don’t think there are any more things left for pip to do. |
|
The reported issue applies to all supported versions of Python. Since the plan is for pip to remain reliant on distutils from the stdlib for older Pythons, then I think pip should also be responsible for ensuring that distutils is used, so that Setuptools doesn't need to try to detect if pip is the caller and keep track of which Python versions pip need special treatment. In particular, I'd like to get rid of this special exemption. |
|
I don't think it's done either. pip unconditionally imports distutils. I started drafting a PR to have pip set |
Ah right, I forgot to add a conditional import. Thanks for spotting it. I’ll add some mechanism so the module is only imported if the distutils backend is used.
Does setting the environment variable all that’s needed? I can incorporate this to the said conditional import mechanism to make this work. How would you recommend to reliably test that the stdlib copy is used in CI? |
pradyunsg
added
type: maintenance
I don't think setting an environment will work. Because Setuptools uses a .pth file, the environment variable will already have been evaluated before any runtime has a chance to set it. It's only honored during I'm now wondering if pip should invoke something like: (undoing the effect of the .pth file) I'm not loving that approach, but it is in some ways the most direct. Do you have any other ideas?
Probably |
I'm happy to just do this on pip's end. |
|
FWIW, all of pip's imports of distutils now live within a single module -- https://github.com/pypa/pip/blob/9351efd78b0c0f7814b01188f6e877c5411cf104/src/pip/_internal/locations/_distutils.py We could just add this snippet on top of that module, which should make things work as expected? |
|
And, this is now only affecting Python <3.10, since starting with 3.10, pip's using importlib.metadata as the source of truth for loading metadata and sysconfig for the various paths of the packages. |
|
In order to test the proposed fix, I created the https://github.com/pypa/setuptools/tree/feature/remove-distutils-hack-pip-workaround branch of Setuptools that removes its workaround and was able to repro the issue: I also confirmed that the issue does indeed no longer affect pip on Python 3.10. I then confirmed that the workaround implemented in pip also addresses the issue: Now the crucial question, what happens if a Python 3.9 user imports pip during their setup.py invocation (per pypa/setuptools#2355)? It does appear that simply importing pip does not trigger the import of distutils, so the secondary concern may be addressed as well: Setuptools will need to keep its workaround for some time until this new version of pip can be relied upon. |
|
Simply importing pip would not directly trigger importing distutils, since the import would emit DeprecationWarning on 3.10. So importing pip from |
Environment
Description
In pypa/setuptools#417, Setuptools is seeking to adopt distutils. With the current release (49.6), Setuptools has a vendored copy of distutils, but that copy is not activated except when
SETUPTOOLS_USE_DISTUTILS=localin the environment.In pypa/setuptools#2259, the team identified the need to make the distutils replacement more aggressive so that packages that
import distutilsbeforeimport setuptoolswould get the version from Setuptools (and not some unfortunate mix). This behavior was released in Setuptools 49.3, but still dormant unlessSETUPTOOLS_USE_DISTUTILS=local.In pypa/setuptools#2255, we're seeking to make that behavior the default (opt-out with
SETUPTOOLS_USE_DISTUTILS=stdlib), but the test suite caught an undesirable behavior when attempting to upgrade setuptools--it seems that there's a race condition in pip when the distutils that setuptools is supplying disappears during an upgrade.Expected behavior
pipshould be able to upgrade setuptools even when setuptools supplies distutils.How to Reproduce
SETUPTOOLS_USE_DISTUTILS=localset.Output
The text was updated successfully, but these errors were encountered: