Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish releases to PyPI via a GitHub Action (and replace setup.py with pyproject.toml) #495

Closed
wants to merge 22 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,14 @@
install: ## Install package in editable mode with all the dependencies
pip install -e .

.PHONY: clean
clean: # Clean up install (and build)
pip uninstall -y alibi-detect
rm -r alibi_detect.egg-info/ dist/ build/

.PHONY: test
test: ## Run all tests
python setup.py test
pytest alibi-detect

.PHONY: lint
lint: ## Check linting according to the flake8 configuration in setup.cfg
Expand Down Expand Up @@ -34,7 +39,8 @@ clean_docs: ## Clean the documentation build

.PHONY: build_pypi
build_pypi: ## Build the Python package
python setup.py sdist bdist_wheel
pip install build
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the version of build be specified in the pyproject.toml too?

Copy link
Contributor Author

@ascillitoe ascillitoe May 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeh good point. This is a weird one as we still need to pip install build in the Makefile, since we need to run the python -m build --sdist --wheel command. If we also bound the build version in the pyproject.toml, this does then pull in the correct build version within the isolated build virtualenv, but I'd imagine this isn't actually used since we're already in the process of running python -m build with the pre-installed build...

Will have a think about this one 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually think it makes sense to do the build (and publish) via a github action, in order to give is complete control over the build environment. Have added a .github/workflows/publish.yml (still requires testing/refinement).

This approach also simplifies the release process to something like:

  1. git checkout master in current local repo (no need for fresh virtual env etc. setuptools_scm will check for dirty workdir).
  2. Update README.md and citation.
  3. Final git commit and git tag.
  4. Push the commit and tag. Github will build and push.

As a safety net we can have logic to push to TestPyPI if tagged with RC version, otherwise push to PyPI.

python -m build --sdist --wheel

.PHONY: push_pypi_test
push_pypi_test: ## Upload the Python package to the test PyPI registry
Expand Down
4 changes: 3 additions & 1 deletion alibi_detect/version.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,6 @@
# 1) we don't load dependencies by storing it in __init__.py
# 2) we can import it in setup.py for the same reason
# 3) we can import it into your module module
__version__ = "0.9.2dev"
# The version is read from the pyproject.toml file
from importlib import metadata
__version__ = metadata.version(__package__)
Copy link
Contributor

@jklaise jklaise Apr 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this actually work? Is importlib.metadata a representation of pyproject.toml for the package? How does pyproject.toml get packaged into the package during build (e.g. no need to include in MANIFEST.in? Does some magic happen during build so that pyproject.toml is mapped to importlib.metadata?

Copy link
Contributor Author

@ascillitoe ascillitoe May 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is following the example here: https://github.com/pypa/setuptools_scm#retrieving-package-version-at-runtime

It relies on the package actually being installed (and yes pyproject.toml does seem to packaged by default during the build). At the very least I should probably add a PackageNotFoundError like in the example...

I've actually just pushed an update using setuptools_scm. This approach has two main benefits IMO:

  1. It would simplify the release workflow slightly. We'd just create the new git tag, and then do the build. No need to update hardcoded version number (but still need to update readme annoyingly).
  2. There would be no possibility of accidentally publishing dirty code. If you install/build with changes in the workdir, the version will be set to something like 0.9.2.dev16+g68e502c.d20220509 (g stands for git commit, d for date). PyPI doesn't accept these "dirty" version numbers. - No longer applies, since workflow is to push a tag and only build via github action. The "dirty" version number would still be useful for point 3 below.
  3. Above "dirty" version numbers would also be written to generated config.toml files. This would be helpful as it would indicate whether or not a detector was generated from a non-official version of Alibi Detect.

Copy link
Contributor Author

@ascillitoe ascillitoe May 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: alibi_detect._version.py is removed from github tracking in order to use setuptools_scm. However, this will require alibi_detect to be installed on readthedocs #499 to generate the _version.py (see setuptools_scm discussion). This might make sense to do anyway after the optional deps project...

Update: I've merged #499 into here to check this. The RTD build now succeeds.

111 changes: 111 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# setuptools settings (we could change to poetry or flit in the future)
[build-system]
requires = [
"setuptools>=61.0.0, <63.0.0",
"wheel>=0.36.0, <0.38.0"
]
build-backend = "setuptools.build_meta"

[tool.setuptools]
include-package-data = true

[tool.setuptools.packages.find]
where = ["."]
include = ["alibi_detect*"]
exclude = ["**/tests*"] # tests should be excluded by default (https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#flat-layout) but exclude explicitly in case of change
namespaces = false

# Project metadata
[project]
name = "alibi-detect"
description = "Algorithms for outlier, adversarial and drift detection."
authors = [
{name = "Seldon Technologies Ltd."},
{email = "[email protected]"}
]
license = {text = "Apache-2.0"}
readme = "README.md"
version = "0.9.2dev"
requires-python = ">=3.7, <3.11"

classifiers = [
"Intended Audience :: Science/Research",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"License :: OSI Approved :: Apache Software License",
"Topic :: Scientific/Engineering"
]

dependencies = [
"matplotlib>=3.0.0, <4.0.0",
"numpy>=1.16.2, <2.0.0",
"pandas>=0.23.3, <2.0.0",
"Pillow>=5.4.1, <10.0.0",
"opencv-python>=3.2.0, <5.0.0",
"scipy>=1.3.0, <2.0.0",
'scikit-image>=0.14.2, !=0.17.1, <0.20', # https://github.com/SeldonIO/alibi/issues/215
"scikit-learn>=0.20.2, <1.1.0",
"tensorflow>=2.2.0, !=2.6.0, !=2.6.1, <2.9.0", # https://github.com/SeldonIO/alibi-detect/issues/375 and 387
"tensorflow_probability>=0.8.0, <0.17.0",
"transformers>=4.0.0, <5.0.0",
"dill>=0.3.0, <0.4.0",
"tqdm>=4.28.1, <5.0.0",
"requests>=2.21.0, <3.0.0",
"numba>=0.50.0, !=0.54.0, <0.56.0" # Avoid 0.54 due to: https://github.com/SeldonIO/alibi/issues/466
]

[project.urls]
repository = "https://github.com/SeldonIO/alibi-detect"
documentation = "https://docs.seldon.io/projects/alibi-detect/en/stable/"

[project.optional-dependencies]
prophet = ["fbprophet>=0.5, <0.7", "holidays==0.9.11", "pystan<3.0"]
torch = ["torch>=1.7.0"]


# pytest
[tool.pytest.ini_options]
addopts = "--tb native -W ignore --cov=alibi_detect"


# mypy
[tool.mypy]
ignore_missing_imports = true
strict_optional = false
show_error_codes = true


# tox test environment for generating licenses (below syntax subject to change - https://tox.wiki/en/latest/example/basic.html#pyproject-toml-tox-legacy-ini)
[tool.tox]
legacy_tox_ini = """
[tox:tox]

[testenv:licenses]
basepython = python
deps =
pip-licenses
typing_extensions # not present with py38
idna-ssl # not present with py36
dataclasses # not present with py36
importlib-metadata # not present with py38
zipp # not present with py38
fsspec # not present with py36
locket # not present with py36
partd # not present with py36
extras = all
commands =
pip-licenses \
--from=mixed \
--format=csv \
--output-file=./licenses/license_info.csv
pip-licenses \
--from=mixed \
--format=plain-vertical \
--with-license-file \
--no-license-path \
--output-file=./licenses/license.txt
"""
45 changes: 1 addition & 44 deletions setup.cfg
Original file line number Diff line number Diff line change
@@ -1,55 +1,12 @@
[aliases]
test=pytest

[tool:pytest]
addopts =
--tb native
-W ignore
--cov=alibi_detect
#-n auto
#--forked

[flake8]
[flake8] # currently no pyproject.toml support for flake8 (https://github.com/PyCQA/flake8/issues/234)
max-line-length = 120
exclude =
# sphinx configuration
doc/source/conf.py
# post-test
.eggs/

[mypy]
ignore_missing_imports = True
strict_optional = False
show_error_codes = True

# sphinx configuration
[mypy-conf]
ignore_errors = True

# tox test environment for generating licenses
[tox:tox]

[testenv:licenses]
basepython = python
deps =
pip-licenses
typing_extensions # not present with py38
idna-ssl # not present with py36
dataclasses # not present with py36
importlib-metadata # not present with py38
zipp # not present with py38
fsspec # not present with py36
locket # not present with py36
partd # not present with py36
extras = all
commands =
pip-licenses \
--from=mixed \
--format=csv \
--output-file=./licenses/license_info.csv
pip-licenses \
--from=mixed \
--format=plain-vertical \
--with-license-file \
--no-license-path \
--output-file=./licenses/license.txt
62 changes: 0 additions & 62 deletions setup.py

This file was deleted.