Merge PEP 660 implementation into `main` #3488

abravalheri · 2022-08-05T10:06:59Z

The idea of this PR is just to mark the implementation as "complete enough" / "tested enough" that it can be merged into main.

The main concepts of the implementation were presented in #3265.

The development of this feature was split into several PRs previously submitted.
Anyone interested in checking these smaller PRs can have a look on the following searches:

Several discussions and feedback were collected in:

Hopefully the development was slow enough and the community engagement was broad enough that there were enough chances of involvement for feedback, design changes, etc... 🤞

Instructions for testing the feature before the PR is merged can be found in https://discuss.python.org/t/help-testing-pep-660-support-in-setuptools/16904.

Summary of changes

Added editable hooks according with PEP 660
Added a "strict" editable installation mode
Added a "compat" editable installation mode (this mode is transitional and will be removed in future versions of setuptools - my personal suggestion is to remove this mode after the end of the year).
Added mechanisms for custom build sub-commands to tap into the editable install mode and do the necessary adaptions.
Updated userguide/development_mode docs.
Changed build_py and build_ext to add a mechanism of getting a list of files being packages and to which original files they correspond to.
Improved handling of --build-option and --global-option in the config_settings argument of the build_meta hooks (to match the historic pip behaviour)
- this is, at least in theory, backwards compatible and only warn the user when a given --global-option should actually be a --build-option, but accept it anyway.

Closes #2816

Pull Request Checklist

Changes have tests
News fragment added in changelog.d/.
(See documentation for details)

…info

Changes: - Deprecate the --egg-base parameter for dist_info and add --output-dir as replacement (Since the egg format is mostly deprecated, it is nice to move away from this nomenclature...)

Avoid using the editables dependency - The current implementation is using it to simply place .pth file pointing to the project directory anyway... - Adding a dependency for creating a file with a single line is a bit overkill. Avoid importing pkg_resources directly - Setuptools wants to move away from pkg_resources Replace custom wheel build with re-use of bdist_wheel - pro: avoid re-implementing the archiving logic and make sure it is compatible with the same archive format used by the final wheels. - con: the API of wheel is not exactly stable or exported as public.

This change is heavily inspired by the existing develop command - Use namespaces.Installer to ensure legacy namespaces are handled.

- Improve the skeleton that would allow the separation between different editable strategies - Extract strategy for static .pth files - Recognise the static .pth file as the primary strategy for src-layout packages in a "non"-strict scenario

- Move test_editable_prefix from test_develop to test_editable_install - Add more tests for editable install focusing on packages using namespaces (legacy or PEP 420)

- see pypa/setuptools#3488

As with PEP-517 support, this is simply a shim over the setuptools implementation added in setuptools v64.0.0 [1]. [1] pypa/setuptools#3488 Change-Id: I32f974db37e364cf634b050b40bf0820dce0a3a3 Signed-off-by: Stephen Finucane <[email protected]>

* Update pbr from branch 'master' to 457a89ffda32cb7a49fa680a71d8c043bd23b352 - Add support for PEP-660 As with PEP-517 support, this is simply a shim over the setuptools implementation added in setuptools v64.0.0 [1]. [1] pypa/setuptools#3488 Change-Id: I32f974db37e364cf634b050b40bf0820dce0a3a3 Signed-off-by: Stephen Finucane <[email protected]>

pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488

* Update python-novaclient from branch 'master' to d9c1ba9a45d7e5265346f0f889ae7c12ff0ad466 - Merge "add pyproject.toml to support pip 23.1" - add pyproject.toml to support pip 23.1 pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: Ieea0ac142e79a9de4d2fbf45fdad70d0ff079304

pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: Ieea0ac142e79a9de4d2fbf45fdad70d0ff079304

pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: Ie31840df3c78013f49def3c5b95c8f79589ad830

* Update python-ironic-inspector-client from branch 'master' to 798d123db55248dc394ee92fc967ef3e04844d93 - Merge "add pyproject.toml to support pip 23.1" - add pyproject.toml to support pip 23.1 pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: Ie31840df3c78013f49def3c5b95c8f79589ad830

* Update python-ironicclient from branch 'master' to 5f9ae76a018e2ce1e395e2bd5142d24dbf1116ea - Merge "add pyproject.toml to support pip 23.1" - add pyproject.toml to support pip 23.1 pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: Ib5fd98d58d282df24c86defd8cce50ed861f912c

pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: Ib5fd98d58d282df24c86defd8cce50ed861f912c

* Update os-traits from branch 'master' to c23f37c0e88ddec409f52c10ee404095340478d8 - Merge "add pyproject.toml to support pip 23.1" - add pyproject.toml to support pip 23.1 pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: I50be314d6d2578fa5b7f3a46fb72502d9f38d5e8

* Update os-resource-classes from branch 'master' to 18e26529fbf7882b17256e7ce6c0cb708afd4b52 - Merge "add pyproject.toml to support pip 23.1" - add pyproject.toml to support pip 23.1 pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: I04e254db3510f61b52b2ed753fb07e00150e288d

pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: I04e254db3510f61b52b2ed753fb07e00150e288d

pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: I50be314d6d2578fa5b7f3a46fb72502d9f38d5e8

* Update osc-placement from branch 'master' to b9f711f44f0d2b3c79a4ce97cda7f84310f47f57 - Merge "add pyproject.toml to support pip 23.1" - add pyproject.toml to support pip 23.1 pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: Iba16dde8ddb5b0da2bb191c38ab6a72c0b2db007

pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Change-Id: Iba16dde8ddb5b0da2bb191c38ab6a72c0b2db007

* Update nova from branch 'master' to 7e082705de486c36cb221edce3e53463b514c617 - Merge "add pyproject.toml to support pip 23.1" - add pyproject.toml to support pip 23.1 pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Depends-On: https://review.opendev.org/c/openstack/devstack/+/902758 Change-Id: I3255edb67fd7fe7a25b5871ec071c4b4c9d98ac2

pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 Depends-On: https://review.opendev.org/c/openstack/devstack/+/902758 Change-Id: I3255edb67fd7fe7a25b5871ec071c4b4c9d98ac2

* Update ironic from branch 'master' to 488bc76cabdbd8b3d84709fc121b4ee0e2697a6a - Merge "add pyproject.toml to support pip 23.1" - add pyproject.toml to support pip 23.1 pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 This patch was taken nearly verbatim from the equivalent nova change. Co-Authored-By: Sean Mooney <[email protected]> Change-Id: I34888e8f87b4a3ab09546ba58ef5f2cf495bc7e3

pip 23.1 removed the "setup.py install" fallback for projects that do not have pyproject.toml and now uses a pyproject.toml which is vendored in pip. To address that, this change adds the minimal pyproject.toml to enable pbr to be properly used to build editable wheels. This is required to support installing devstack on centos stream 9 and related distros with GLOBAL_VENV=True Without this change the wsgi scripts are not generated in editable mode. i.e. pip install -e /opt/stack/keystone See https://pip.pypa.io/en/stable/news/#v23-1 and pypa/pip#8368 for more details on the removal of the fallback support. setuptools v64.0.0 is used to support editable installs via its PEP-660 implmentation pypa/setuptools#3488 This patch was taken nearly verbatim from the equivalent nova change. Co-Authored-By: Sean Mooney <[email protected]> Change-Id: I34888e8f87b4a3ab09546ba58ef5f2cf495bc7e3

dholth and others added 30 commits February 4, 2022 15:19

include pep660 proof of concept

be3778e

include pep660 proof of concept (#3082)

2bddfdf

Add dist_info_dir param to dist_info command

c6106b7

Change dist_info to better control output directory

ad803ae

Merge branch 'upstream/main' into build-editable-aggregate

e36951e

Change build_meta to use --output-dir instead of --egg-base for dist-…

e55e51d

…info

Make sure output_dir exists with dist_info

1afb6c0

Avoid creating dist_info_dir before the command runs

0e6870b

Fix signature for build_editable according to PEP 660

da6dd05

Fix flake8 errors with build_meta

e7123c6

Add prepare_metadata_for_build_editable hook

821d664

Editable rework dist_info to use --output-dir instead of --egg-base

c642529

Changes: - Deprecate the --egg-base parameter for dist_info and add --output-dir as replacement (Since the egg format is mostly deprecated, it is nice to move away from this nomenclature...)

Remove unecessary editable dependency

5bb97d0

Make sure setuptools itself can be installed with editable_wheel

3e9f441

Rely on wheel and bdist_wheel for editable_wheel

5866b85

Re-use dist-info dir for build_editable

aee3245

Always run editable install test

33f2431

Start adding APIs for handling multiple editable modes

06fbc36

Temporarily support deprecated namespace packages

7b402eb

Remove unused function

a31df2e

Reorganise editable_wheel to allow different strategies

7ae5a69

Handle namespace_packages in editable_wheel

a13adc8

This change is heavily inspired by the existing develop command - Use namespaces.Installer to ensure legacy namespaces are handled.

Rename variable for clarity

14eb855

Add namespace test

8cb1344

Move test from test_develop to test_editable_install

73c65b2

Add test for PEP 420 packages

59a39d9

Add tests for editable install focusing on namespaces

a27b96b

- Move test_editable_prefix from test_develop to test_editable_install - Add more tests for editable install focusing on packages using namespaces (legacy or PEP 420)

Add template for MetaPathFinder that can be used in editable mode

1a531db

This was referenced Aug 21, 2022

Fix broken editable install AMYPAD/CuVec#25

Merged

[BUG] Realtive import of nonexistent package with same name throws RecursionError #3550

Closed

casperdcl added a commit to AMYPAD/NumCu that referenced this pull request Aug 21, 2022

fix setuptools editable installs

cc4da20

- see pypa/setuptools#3488

casperdcl mentioned this pull request Aug 21, 2022

scikit-build does not work with setuptools>=64 in edit mode scikit-build/scikit-build#740

Open

This was referenced Aug 22, 2022

Repair CI (Temporary solution) cda-tum/mqt-syrec#60

Closed

Repair CI (Temporary solution) cda-tum/mqt-syrec#61

Closed

Repair CI (Temporary solution) cda-tum/mqt-syrec#62

Closed

Repair CI (Temporary solution) cda-tum/mqt-syrec#64

Merged

CaselIT mentioned this pull request Aug 29, 2022

setuptools 64 broke editable build fallback when no compiler is found. sqlalchemy/sqlalchemy#8458

Closed

d1saster mentioned this pull request Sep 6, 2022

Developer install does not work helmholtz-analytics/mpi4torch#2

Closed

jack-fs mentioned this pull request Nov 16, 2022

Use pyproject.toml for package configuration gradientinstitute/causal-inspection#46

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge PEP 660 implementation into `main` #3488

Merge PEP 660 implementation into `main` #3488

abravalheri commented Aug 5, 2022 •

edited

Loading

Merge PEP 660 implementation into main #3488

Merge PEP 660 implementation into main #3488

Conversation

abravalheri commented Aug 5, 2022 • edited Loading

Summary of changes

Pull Request Checklist

Merge PEP 660 implementation into `main` #3488

Merge PEP 660 implementation into `main` #3488

abravalheri commented Aug 5, 2022 •

edited

Loading