Publish releases to PyPI via a GitHub Action (and replace setup.py with pyproject.toml)

[ascillitoe]

This PR implements a release process via github actions. Performing a release will be as simple as pushing a new version release tag e.g.

git tag v0.11.0
git push v0.11.0

This will be particularly useful now that we are discussing doing more frequent patch releases!

A large part of this PR involves replacing the setup.py and setup.cfg files with a pyproject.toml file. This isn't strictly neccesary but provides a more deterministic build. This simplifies the github action but also allows us to better emulate releases manually for testing (e.g. to PyPI testing).

setup.py to pyproject.toml

Historically, a Python package has been defined by the existence of the setup.py (and setup.cfg) file. This file is executed to install and build the package (e.g. setup.py install and setup.py sdist). This approach suffers from a chicken and egg problem; setup.py has dependencies e.g. setuptools and wheels, but we cannot define these deps without executing setup.py.

This prevents properly deterministic builds; for example the conda-forge alibi-detect 0.9.1 build failed because 0.9.1 was built and published to pypi using a different version of setuptools (the install and build were performed inside a conda env with an old default setuptools version), which did not package LICENCE.txt within the build.

This issue is solved by replacing setup.py/.cfg with a pyproject.toml file. This file explicitly specifies the required build backend (e.g. setuptools, poetry, flit), as well as defining all deps (previously in setup.py) and other config options (previously in setup.cfg). pip install's are also then carried out in an isolated environment, giving more deterministic installs as well as builds.

Background reading

A typical pyproject.toml looks something like the following:

[build-system]
requires = [   # PEP 518 - what deps are required to build
	"setuptools>=61.0.0, <63.0.0",
	"wheel>=0.36.0, <0.38.0"
]
build-backend = "setuptools.build_meta"  # PEP 517 - what function to call to build

# PEP 621 - Project metadata
[project]
name = "alibi-detect"
description = "Algorithms for outlier, adversarial and drift detection."

The structure of this file (and how it is used) are covered by a number of recent PEP's:

• PEP 518: This specifies how minimum build system requirements should be specified in a pyproject.toml (via the build-system.requires table).

• PEP 517: This specifies how the Python object to be used for the build should be specified (via the build-system.build-backend table). This PEP also defines how the pyproject.toml should facilitate installs i.e. that we should be able to simply run pip install . when a pyproject.toml is present instead of setup.py.

• PEP 621: This specifies how the package's core metadata is defined (via the project table). Note that not all build tools are officially compliant with this, for example poetry uses a different "table" to set the [requires-python](https://peps.python.org/pep-0621/#requires-python) table.

• PEP 631: To do with syntax for deps version specifications. Merged into PEP 621.

• PEP 639: Related to improving licence specification. Not yet approved.

• PEP 660: Similar to PEP 517, but for editable installs i.e. pip install -e .. Support for this is patchy:

  • pip: Supported (regardless of whether backend is PEP 660 complient) since 21.3.1. See Publish releases to PyPI via a GitHub Action (and replace setup.py with pyproject.toml) #495 (comment).
  • setuptools: Not yet supported. Although still OK as long as as pip is recent enough (see above point).
  • flit: Supported since 3.4.
  • poetry: Supported since 1.1.0.

Considerations

• With older setuptools a setup.py stub file (literally just containing setup()) was required in order for editable installs to still work (i.e. pip install -e .). However, the presence of a setup.py might encourage use of legacy commands such as python.py sdist. After testing these commands, I've found that they don't play nicely with the new setup. For example, the [tool.setuptools.packages.find] settings are ignored, and we end up with docs etc in the tar.gz file. I've therefore removed the setup.py entirely. pip install -e still works as long as pip >=21.3 is used.

• There are some config settings such as flake8 ones which I haven't figured out how to transition to pyproject.toml. Therefore for now we still have a setup.cfg too.

• We still need to choose the actual build backend. IMO poetry adds too much complication; the pyproject.toml becomes quite "non-standard", and I wonder if its dual use as virtual enviroment tool might cause more issues than it solves for now (e.g. see Make the lock file more merge-friendly python-poetry/poetry#496). For now, I've stuck with setuptools, but as recommended by them, this does mean installing PyPA Build to do the actual build (not the pip install!). (see Makefile in PR). IMO flit is worth exploring too.

• There is one downside to the new isolated build approach; if a user doesn't have the specified setuptools and wheel in their local cache of wheels, they won't be able to perform an install (or build) offline.

Progress and TODO's

The install and build seem to work locally, and examining the sdist tar.gz archive, it looks like we have the right stuff in there. Remaining TODO's are:

• Check what happens (especially wrt to editable install) if the setup.py is removed - See Publish releases to PyPI via a GitHub Action (and replace setup.py with pyproject.toml) #495 (comment).
• Check that publishing proceeds as expected by doing a test release to https://test.pypi.org/ - Pushing a v0.10.5-alpha tag to my fork results in this: https://test.pypi.org/project/alibi-detect/0.10.5a0/, which looks OK.
• Update release process wiki. Check if any other docs, CI etc need updating.
• Add logic to publish to Test PyPI if alpha, beta or rc tag, otherwise publish to PyPI if pure semver tag.
• Decide if we are happy to remove setup.py and move to pyproject.toml. If not, we can only use pyproject.toml to set build-system (see Publish releases to PyPI via a GitHub Action (and replace setup.py with pyproject.toml) #495 (comment)).

[jklaise]

How does this actually work? Is importlib.metadata a representation of pyproject.toml for the package? How does pyproject.toml get packaged into the package during build (e.g. no need to include in MANIFEST.in? Does some magic happen during build so that pyproject.toml is mapped to importlib.metadata?

[ascillitoe]

This is following the example here: https://github.com/pypa/setuptools_scm#retrieving-package-version-at-runtime

It relies on the package actually being installed (and yes pyproject.toml does seem to packaged by default during the build). At the very least I should probably add a PackageNotFoundError like in the example...

I've actually just pushed an update using setuptools_scm. This approach has two main benefits IMO:

1. It would simplify the release workflow slightly. We'd just create the new git tag, and then do the build. No need to update hardcoded version number (but still need to update readme annoyingly).
2. There would be no possibility of accidentally publishing dirty code. If you install/build with changes in the workdir, the version will be set to something like 0.9.2.dev16+g68e502c.d20220509 (g stands for git commit, d for date). PyPI doesn't accept these "dirty" version numbers. - No longer applies, since workflow is to push a tag and only build via github action. The "dirty" version number would still be useful for point 3 below.
3. Above "dirty" version numbers would also be written to generated config.toml files. This would be helpful as it would indicate whether or not a detector was generated from a non-official version of Alibi Detect.

[ascillitoe]

Note: alibi_detect._version.py is removed from github tracking in order to use setuptools_scm. However, this will require alibi_detect to be installed on readthedocs #499 to generate the _version.py (see setuptools_scm discussion). This might make sense to do anyway after the optional deps project...

Update: I've merged #499 into here to check this. The RTD build now succeeds.

[jklaise]

Shouldn't the version of build be specified in the pyproject.toml too?

[ascillitoe]

Yeh good point. This is a weird one as we still need to pip install build in the Makefile, since we need to run the python -m build --sdist --wheel command. If we also bound the build version in the pyproject.toml, this does then pull in the correct build version within the isolated build virtualenv, but I'd imagine this isn't actually used since we're already in the process of running python -m build with the pre-installed build...

Will have a think about this one 🤔

[ascillitoe]

Actually think it makes sense to do the build (and publish) via a github action, in order to give is complete control over the build environment. Have added a .github/workflows/publish.yml (still requires testing/refinement).

This approach also simplifies the release process to something like:

1. git checkout master in current local repo (no need for fresh virtual env etc. setuptools_scm will check for dirty workdir).
2. Update README.md and citation.
3. Final git commit and git tag.
4. Push the commit and tag. Github will build and push.

As a safety net we can have logic to push to TestPyPI if tagged with RC version, otherwise push to PyPI.

[ascillitoe]

Note to self: Investigate why tests are not being excluded from sdist. Same for doc and examples.

[ascillitoe]

Additional note on editable installs

The current setup in this PR supports pip install -e . when using pip >= 21.1.0. For older versions the following is given:

ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/ascillitoe/Software/alibi-detect
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)
WARNING: You are using pip version 21.0; however, version 22.3.1 is available.
You should consider upgrading via the '/home/ascillitoe/Software/alibi-detect/venv_tmp/bin/python -m pip install --upgrade pip' command.

pip 21.1 was released on Apr 24, 2021, so not exactly bleeding edge anymore. We could also add pip install --upgrade pip to our installation instructions. Alternatively, another option is to keep our setup.py and only use pyproject.toml to specify our build-system, like thinc does (PR incoming).