Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support pyproject.toml-style configuration (PEP 621) - Round 2 #2970

Merged
merged 63 commits into from
Mar 24, 2022
Merged
Changes from 1 commit
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
49b7a60
Rename `config` to `config.setupcfg`
abravalheri Dec 1, 2021
f866876
Extract post-processing functions from config
abravalheri Dec 2, 2021
7d9ecc0
Allow root_dir to be explicit in config.expand functions
abravalheri Dec 2, 2021
8330040
Allow single strings in config.expand.read_files
abravalheri Dec 3, 2021
a148c33
Adequate test_setupcfg to latest changes in setupcfg
abravalheri Dec 22, 2021
d96e8bf
Add news fragment
abravalheri Feb 1, 2022
61a416b
Make __all__ immutable in setuptools.config
abravalheri Feb 9, 2022
ec2071a
Split complex generator expression in setuptools.config.expand
abravalheri Feb 9, 2022
25612c5
Adopt review suggestions
abravalheri Feb 9, 2022
81c3faa
Replace pushd with monkeypatch.chdir in test_expand
abravalheri Feb 9, 2022
82779f9
Ensure proper exception matching in test_expand
abravalheri Feb 9, 2022
e5d2bc8
Parametrize test_expand.test_find_packages
abravalheri Feb 9, 2022
099ac60
Add `tomli` as vendorised dependency
abravalheri Dec 2, 2021
771488d
Add `validate-pyproject` as a vendored dependency
abravalheri Dec 3, 2021
78dc278
Add news fragment
abravalheri Feb 1, 2022
7371508
Make comment in setuptools/_vendor/vendored.txt more clear
abravalheri Feb 1, 2022
ccd2f07
Ensure relative imports for vendorised tomli
abravalheri Feb 4, 2022
e0d61d4
Update vendored tomli to 2.0.1
abravalheri Feb 8, 2022
74c7341
Improve custom vendoring logic for validate-pyproject
abravalheri Feb 9, 2022
e2f07dc
Update vendored validate-pyproject to 0.4
abravalheri Feb 10, 2022
7f68bb4
Update vendored validate-pyproject to 0.5.2
abravalheri Mar 5, 2022
af187e8
Implement read_configuration from pyproject.toml
abravalheri Dec 3, 2021
8826dc1
Expand dynamic entry_points from pyproject.toml
abravalheri Dec 9, 2021
a8112d9
Make include_package_data=True for `pyproject.toml` configs
abravalheri Dec 22, 2021
9672a48
Add means of applying config read from pyproject.toml to dist
abravalheri Dec 23, 2021
d7363d5
Add the apply_configuration API to setuptools.config.setupcfg
abravalheri Dec 23, 2021
26a9264
Test pyproject.toml config has the same effect as setup.cfg
abravalheri Dec 23, 2021
051b825
Fix pyproject config when tool table is not present
abravalheri Feb 4, 2022
c927227
Remove no longer needed tomli import workaround
abravalheri Feb 4, 2022
905eed7
Update version of test dependency 'ini2toml'
abravalheri Feb 7, 2022
b426b2b
Prevent resource warnings in test_apply_pyprojecttoml
abravalheri Feb 9, 2022
e91969a
Add a 'uses_network' marker to tests that require connectivity
abravalheri Feb 9, 2022
9ee2697
Update test dependency ini2toml to 0.8
abravalheri Feb 10, 2022
e5c5519
Avoid failing due to 3rd party config in pyproject.toml
abravalheri Feb 18, 2022
5d4457e
Add tests against "empty" pyproject.toml
abravalheri Feb 18, 2022
cf32acb
Avoid using pkg_resources for entry points
abravalheri Feb 18, 2022
a4b474e
Back-fill Description-Content-Type according to readme suffix
abravalheri Feb 23, 2022
0497954
Update test dependency ini2toml to 0.9
abravalheri Mar 5, 2022
d385330
Add pyproject.toml to dist.parse_config_files
abravalheri Dec 24, 2021
dea4be5
Add deprecation notice for config.{read,parse}_configuration
abravalheri Dec 25, 2021
2b333e9
Add backend test with pyproject.toml-based configs
abravalheri Dec 23, 2021
09f784f
Test editable installs with pyproject.toml metadata
abravalheri Dec 24, 2021
9e8e3d3
Replace skip in editable install test with xfail
abravalheri Jan 13, 2022
aab5899
Add news fragment
abravalheri Feb 1, 2022
c9cf0da
Ensure build_meta don't have problems with instructions after setup()
abravalheri Feb 11, 2022
98c8edb
Test if not-zip-safe file is being generated with project metadata
abravalheri Feb 11, 2022
86e6a10
Test static metadata in pyproject.toml is not overwritten by setup.py
abravalheri Feb 12, 2022
854969d
Explicitly inform users that pyproject.toml config is experimental
abravalheri Feb 18, 2022
e9c1a32
Rely on validate-pyproject default errors
abravalheri Mar 5, 2022
0cc7478
Show significant error messages to user and avoid traceback pollution
abravalheri Mar 5, 2022
298e745
Removed unused import
abravalheri Mar 5, 2022
b44c648
Separate setup.cfg parsing and extract common post-processing functio…
abravalheri Mar 5, 2022
088d467
Add vendored deps: tomli and validate-pyproject (#3066)
abravalheri Mar 5, 2022
2cfcf0e
Add means to apply configuration from pyproject.toml (#3067)
abravalheri Mar 5, 2022
96adc4f
Fix variable name error
abravalheri Mar 5, 2022
1bb0021
Add some type hints to config.setupcfg
abravalheri Dec 25, 2021
441a1fa
Add some type hints to config.pyprojecttoml
abravalheri Dec 25, 2021
d3e62b1
Add some type hints to config.expand
abravalheri Dec 25, 2021
54f6180
Find namespaces by default when using config in 'pyproject.toml'
abravalheri Feb 19, 2022
5c334b3
Add news fragment
abravalheri Feb 19, 2022
f54e2d9
Integrate pyproject.toml configuration into existing classes (#3068)
abravalheri Mar 5, 2022
dd75299
Add some type hints to the setuptools.config subpackage (#3069)
abravalheri Mar 5, 2022
64386ba
Adopt namespaces by default when discovering packages (#3125)
abravalheri Mar 5, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Implement read_configuration from pyproject.toml
This is the first step towards making setuptools understand
`pyproject.toml` as a configuration file.

The implementation deliberately allows splitting the act of loading the
configuration from a file in 2 stages: the reading of the file itself
and the expansion of directives (and other derived information).
  • Loading branch information
abravalheri committed Mar 5, 2022
commit af187e8fc56617a5b97deeaff6173aaee3355016
195 changes: 195 additions & 0 deletions setuptools/config/pyprojecttoml.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
"""Load setuptools configuration from ``pyproject.toml`` files"""
import os
import sys
from contextlib import contextmanager
from functools import partial
from typing import Union
import json

from setuptools.errors import OptionError, FileError
from distutils import log

from . import expand as _expand

_Path = Union[str, os.PathLike]


def load_file(filepath: _Path):
try:
from setuptools.extern import tomli
except ImportError: # Bootstrap problem (?) diagnosed by test_distutils_adoption
sys_path = sys.path.copy()
try:
from setuptools import _vendor
sys.path.append(_vendor.__path__[0])
import tomli
finally:
sys.path = sys_path

with open(filepath, "rb") as file:
return tomli.load(file)


def validate(config: dict, filepath: _Path):
from setuptools.extern import _validate_pyproject
from setuptools.extern._validate_pyproject import fastjsonschema_exceptions

try:
return _validate_pyproject.validate(config)
except fastjsonschema_exceptions.JsonSchemaValueException as ex:
msg = [f"Schema: {ex}"]
if ex.value:
msg.append(f"Given value:\n{json.dumps(ex.value, indent=2)}")
if ex.rule:
msg.append(f"Offending rule: {json.dumps(ex.rule, indent=2)}")
if ex.definition:
msg.append(f"Definition:\n{json.dumps(ex.definition, indent=2)}")

log.error("\n\n".join(msg) + "\n")
raise


def read_configuration(filepath, expand=True, ignore_option_errors=False):
"""Read given configuration file and returns options from it as a dict.

:param str|unicode filepath: Path to configuration file in the ``pyproject.toml``
format.

:param bool expand: Whether to expand directives and other computed values
(i.e. post-process the given configuration)

:param bool ignore_option_errors: Whether to silently ignore
options, values of which could not be resolved (e.g. due to exceptions
in directives such as file:, attr:, etc.).
If False exceptions are propagated as expected.

:rtype: dict
"""
filepath = os.path.abspath(filepath)

if not os.path.isfile(filepath):
raise FileError(f"Configuration file {filepath!r} does not exist.")

asdict = load_file(filepath) or {}
project_table = asdict.get("project")
tool_table = asdict.get("tool", {}).get("setuptools")
abravalheri marked this conversation as resolved.
Show resolved Hide resolved
if not asdict or not(project_table or tool_table):
return {} # User is not using pyproject to configure setuptools

with _ignore_errors(ignore_option_errors):
validate(asdict, filepath)

if expand:
root_dir = os.path.dirname(filepath)
return expand_configuration(asdict, root_dir, ignore_option_errors)

return asdict


def expand_configuration(config, root_dir=None, ignore_option_errors=False):
"""Given a configuration with unresolved fields (e.g. dynamic, cmdclass, ...)
find their final values.

:param dict config: Dict containing the configuration for the distribution
:param str root_dir: Top-level directory for the distribution/project
(the same directory where ``pyproject.toml`` is place)
:param bool ignore_option_errors: see :func:`read_configuration`

:rtype: dict
"""
root_dir = root_dir or os.getcwd()
project_cfg = config.get("project", {})
setuptools_cfg = config.get("tool", {}).get("setuptools", {})
package_dir = setuptools_cfg.get("package-dir")

_expand_all_dynamic(project_cfg, setuptools_cfg, root_dir, ignore_option_errors)
_expand_packages(setuptools_cfg, root_dir, ignore_option_errors)
_canonic_package_data(setuptools_cfg)
_canonic_package_data(setuptools_cfg, "exclude-package-data")

process = partial(_process_field, ignore_option_errors=ignore_option_errors)
cmdclass = partial(_expand.cmdclass, package_dir=package_dir, root_dir=root_dir)
data_files = partial(_expand.canonic_data_files, root_dir=root_dir)
process(setuptools_cfg, "data-files", data_files)
process(setuptools_cfg, "cmdclass", cmdclass)

return config


def _expand_all_dynamic(project_cfg, setuptools_cfg, root_dir, ignore_option_errors):
silent = ignore_option_errors
dynamic_cfg = setuptools_cfg.get("dynamic", {})
package_dir = setuptools_cfg.get("package-dir", None)
special = ("license", "readme", "version", "entry-points", "scripts", "gui-scripts")
# license-files are handled directly in the metadata, so no expansion
# readme, version and entry-points need special handling
dynamic = project_cfg.get("dynamic", [])
regular_dynamic = (x for x in dynamic if x not in special)

for field in regular_dynamic:
value = _expand_dynamic(dynamic_cfg, field, package_dir, root_dir, silent)
project_cfg[field] = value

if "version" in dynamic and "version" in dynamic_cfg:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This value can be given by extensions such as setuptools-scm, so we don't have to fail if no configuration for version is given in [tool.setuptools.dynamic].

version = _expand_dynamic(dynamic_cfg, "version", package_dir, root_dir, silent)
project_cfg["version"] = _expand.version(version)

if "readme" in dynamic:
project_cfg["readme"] = _expand_readme(dynamic_cfg, root_dir, silent)


def _expand_dynamic(dynamic_cfg, field, package_dir, root_dir, ignore_option_errors):
if field in dynamic_cfg:
directive = dynamic_cfg[field]
if "file" in directive:
return _expand.read_files(directive["file"], root_dir)
if "attr" in directive:
return _expand.read_attr(directive["attr"], package_dir, root_dir)
elif not ignore_option_errors:
msg = f"Impossible to expand dynamic value of {field!r}. "
msg += f"No configuration found for `tool.setuptools.dynamic.{field}`"
raise OptionError(msg)
return None


def _expand_readme(dynamic_cfg, root_dir, ignore_option_errors):
silent = ignore_option_errors
return {
"text": _expand_dynamic(dynamic_cfg, "readme", None, root_dir, silent),
"content-type": dynamic_cfg["readme"].get("content-type", "text/x-rst")
}


def _expand_packages(setuptools_cfg, root_dir, ignore_option_errors=False):
packages = setuptools_cfg.get("packages")
if packages is None or isinstance(packages, (list, tuple)):
return

find = packages.get("find")
if isinstance(find, dict):
find["root_dir"] = root_dir
with _ignore_errors(ignore_option_errors):
setuptools_cfg["packages"] = _expand.find_packages(**find)


def _process_field(container, field, fn, ignore_option_errors=False):
if field in container:
with _ignore_errors(ignore_option_errors):
container[field] = fn(container[field])


def _canonic_package_data(setuptools_cfg, field="package-data"):
package_data = setuptools_cfg.get(field, {})
return _expand.canonic_package_data(package_data)


@contextmanager
def _ignore_errors(ignore_option_errors):
if not ignore_option_errors:
yield
return

try:
yield
except Exception as ex:
log.debug(f"Ignored error: {ex.__class__.__name__} - {ex}")
103 changes: 103 additions & 0 deletions setuptools/tests/config/test_pyprojecttoml.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
import os

from setuptools.config.pyprojecttoml import read_configuration, expand_configuration

EXAMPLE = """
[project]
name = "myproj"
keywords = ["some", "key", "words"]
dynamic = ["version", "readme"]
requires-python = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
dependencies = [
'importlib-metadata>=0.12;python_version<"3.8"',
'importlib-resources>=1.0;python_version<"3.7"',
'pathlib2>=2.3.3,<3;python_version < "3.4" and sys.platform != "win32"',
]

[project.optional-dependencies]
docs = [
"sphinx>=3",
"sphinx-argparse>=0.2.5",
"sphinx-rtd-theme>=0.4.3",
]
testing = [
"pytest>=1",
"coverage>=3,<5",
]

[project.scripts]
exec = "pkg.__main__:exec"

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[tool.setuptools]
package-dir = {"" = "src"}
zip-safe = true
platforms = ["any"]

[tool.setuptools.packages.find]
where = ["src"]
namespaces = true

[tool.setuptools.cmdclass]
sdist = "pkg.mod.CustomSdist"

[tool.setuptools.dynamic.version]
attr = "pkg.__version__.VERSION"

[tool.setuptools.dynamic.readme]
file = ["README.md"]
content-type = "text/markdown"

[tool.setuptools.package-data]
"*" = ["*.txt"]

[tool.setuptools.data-files]
"data" = ["files/*.txt"]

[tool.distutils.sdist]
formats = "gztar"

[tool.distutils.bdist_wheel]
universal = true
"""


def test_read_configuration(tmp_path):
pyproject = tmp_path / "pyproject.toml"

files = [
"src/pkg/__init__.py",
"src/other/nested/__init__.py",
"files/file.txt"
]
for file in files:
(tmp_path / file).parent.mkdir(exist_ok=True, parents=True)
(tmp_path / file).touch()

pyproject.write_text(EXAMPLE)
(tmp_path / "README.md").write_text("hello world")
(tmp_path / "src/pkg/mod.py").write_text("class CustomSdist: pass")
(tmp_path / "src/pkg/__version__.py").write_text("VERSION = (3, 10)")
(tmp_path / "src/pkg/__main__.py").write_text("def exec(): print('hello')")

config = read_configuration(pyproject, expand=False)
assert config["project"].get("version") is None
assert config["project"].get("readme") is None

expanded = expand_configuration(config, tmp_path)
assert read_configuration(pyproject, expand=True) == expanded
assert expanded["project"]["version"] == "3.10"
assert expanded["project"]["readme"]["text"] == "hello world"
assert set(expanded["tool"]["setuptools"]["packages"]) == {
"pkg",
"other",
"other.nested",
}
assert "" in expanded["tool"]["setuptools"]["package-data"]
assert "*" not in expanded["tool"]["setuptools"]["package-data"]
assert expanded["tool"]["setuptools"]["data-files"] == [
("data", ["files/file.txt"])
]