Use PEP 518 in packages to require astropy-helpers at build time

[astrofrog]

PEP 518 specifies that Python packages can include a pyproject.toml file that specifies build-time dependencies. This is intended to replace setup_requires, which was never useful for us (and many others) since setup_requires was only taken into account when setup() was run, whereas e.g. astropy-helpers is needed to even run setup.py at all. With PEP 518 now supported by pip, we could add a pyproject.toml file in astropy and affiliated packages with:

[build-system]
requires = ["setuptools", "wheel", "astropy-helpers"]

optionally with a fixed version for astropy-helpers. This would then allow us, I think, to get rid of a lot of ah_bootstrap.py or simplify it significantly.

[bsipocz]

would pyproject.toml recognize that astropy-helpers exists as a submodule for the packages or will try to install the latest released version from pypi?

[astrofrog]

I think it would just try and get the latest version from PyPI and ignore the sub-module. We may want to re-assess whether we really want to have the sub-module in the long term though. The whole current system was in part done because of things like setup_requires not working, but if that is changing we can re-assess how we deal with the helpers (maybe a topic for the coordination meeting).

[bsipocz]

It would be though still nice to somehow have a control over the versions installed, especially if it's an interim dev version.

[Cadair]

I am going to hijack this a little and talk about what I think the way forward for astropy_helpers should be. This is mostly based off of my desire to remove the submodule completely because it's a pain.

I think helpers should be divided up into two bits, the first should contain the components needed to build the sdist this would include:

• version helpers
• Things relating to package_data and entrypoints in setup_package.py
• The non-C extension related stuff in astropy_helpers.setup_helpers

basically this should be everything needed to help out with a pure python package, and everything needed to build the sdist of the a package with C/Cython extensions. This code would then be placed in a single file and copied into tree like ah_bootstrap.py currently is (named something like setup_helpers.py.

The second component would be an installable python package distributed via PyPI, which would be installed (pinned to a specific version at build time) by pip when installing (building) the sdist. This would contain all the compile time helpers and discovery stuff needed to build extensions.

The main reason for this is that the build time dependencies as described by PEP 518 are only installed at build time, i.e. not when generating an sdist, and given that there is a bunch of things that run at sdist time, i.e. package data, C files and version hashes, we need that stuff separated out into a different (in tree) component if we are going to utilize PEP 518

[Cadair]

@astrofrog points out that the setup_package.py files sometimes import astropy_helpers which would not work at sdist time using this separation, which would need investigating and rectifying.

[pllim]

Re: version helpers -- #420

[bsipocz]

The second component would be an installable python package distributed via PyPI, which would be installed (pinned to a specific version at build time) by pip when installing (building) the sdist. This would contain all the compile time helpers and discovery stuff needed to build extensions.

I'm having deja vu that we went through all of these arguments from above during the coordination meeting last month, and the conclusion was then that we don't make it an installable package (and that probably won't move away from submoduling just right now).

[Cadair]

PEP 518 allows you to explicitly pin a version of the build dependencies, which are installed into the build environment, nobody should have astropy_helpers installed into their env, just as they don't at the moment, it would all be handled and controlled by pip during build time. This is exactly what PEP 518 is designed to handle, tools needed to build packages.

Also, I put my thoughts down here based on digging I have done since the coordination meeting, and so that there is a record of these things in a sensible place.

[Cadair]

Also, I have just tested what happens if you have astropy-helpers installed in your env and then you use PEP 518 to specify a version of astropy-helpers:

$ pip freeze                                                                                                                                                                    
astropy==3.1.1
astropy-helpers==2.0.8

then I have:

[build-system]
# Minimum requirements for the build system to execute.
requires = ["setuptools", "wheel", "astropy-helpers==3.0.1"]

in pyproject.toml

and when I pip install . the version of astropy_helpers imported in setup.py is 3.0.1. So it seems that pip correctly handles isolating the build environment.

[Cadair]

When thinking about this, we should probably be planning to use what is specified in PEP 517 as well, which is implemented in pip but not currently released I think.

[embray]

I was just about to make an issue for the same thing. pip now supports pyproject.toml so it should be possible to use that and eventually get rid of ah_bootstrap.py as well.

[Cadair]

What came out of these experiments and discussions became: astropy/astropy-APEs#52

[pllim]

Didn't pip 19.1 broke stuff that use pyproject.toml? Did they fix that already?

[Cadair]

This is closed by APE 17 and the existence of the extension-helpers repo