feat: Add support for writing pyproject.toml files (#33)

This PR adds support for in-place modification of pyproject.toml files.
The modification cannot be done out of place in this case because only a
subset of the file is dependencies and there is no way for a
pyproject.toml file to point to another location for its dependencies.

On the positive side, the changeset here is quite small and easily
manageable. OTOH, there are a number of open design questions that
should be addressed before merging this PR. Note that some of these
considerations are the same as for conda meta.yaml files in #28. I would
consider pyproject.toml to be the easier of the two and require solving
a subset of the problems needed for meta.yaml. Essentially all of the
problems below exist in similar form for those files, so figuring out
how pyproject.toml support should look will get us a large part of the
way towards enabling meta.yaml.

1. The interpretation of the `files` key of dependencies.yaml is
somewhat suspect when used for pyproject.toml because there is no
sensible reason to want to rename the file (this is also true for
meta.yaml)
2. For the optional dependencies, we need a way to specify the key of
the optional dependencies section. I am currently using the file name
for this purpose, but this may be problematic because the file name may
not be unique i.e. we might need optional test dependencies in
pyproject.toml as well as a test_*.yaml env file (This same problem
exists for meta.yaml due to the different host/build/run/test sections).
Furthermore, I've currently hacked this in by passing the filename key
into the `make_dependency_file` function even though that function
doesn't need it for anything else because the output filename has
already been generated and is one of the inputs here (this is also why
one test is currently failing). If we move forward with this approach,
that code will need a bit of refactoring.
3. I am currently splitting up the build requirements, the run
dependencies, and the optional dependencies as different OutputTypes,
which is functional but somewhat contrived as being distinct from the
different sections in (2).

**EDIT**
The above concerns have mostly been addressed. We are now using a single
output type for all pyproject sections and allowing a new `extras`
section that can be used to provide data specific to pyproject.toml, in
this case the table and key under which to store a particular dependency
list. This addresses points 2 and 3 above. Point 1 is still somewhat
unaddressed, but is mitigated by better documenting the behavior.

README.md

@@ -53,10 +53,11 @@ Here is an example of what the `files` key might look like:
 
 ```yaml
 files:
-  all: # used as the prefix for the generated dependency file names
-    output: [conda, requirements] # which dependency file types to generate. required, can be "conda", "requirements", "none" or a list of non-"none" values
+  all: # used as the prefix for the generated dependency file names for conda or requirements files (has no effect on pyproject.toml files)
+    output: [conda, requirements] # which dependency file types to generate. required, can be "conda", "requirements", "pyproject", "none" or a list of non-"none" values
     conda_dir: conda/environments # where to put conda environment.yaml files. optional, defaults to "conda/environments"
     requirements_dir: python/cudf # where to put requirements.txt files. optional, but recommended. defaults to "python"
+    pyproject_dir: python/cudf # where to put pyproject.toml files. optional, but recommended. defaults to "python"
     matrix: # (optional) contains an arbitrary set of key/value pairs to determine which dependency files that should be generated. These values are included in the output filename.
       cuda: ["11.5", "11.6"] # which CUDA version variant files to generate.
       arch: [x86_64] # which architecture version variant files to generate. This value should be the result of running the `arch` command on a given machine.
@@ -98,6 +99,28 @@ files:
 
 When `output: none` is used, the `conda_dir`, `requirements_dir` and `matrix` keys can be omitted. The use case for `output: none` is described in the [_Additional CLI Notes_](#additional-cli-notes) section below.
 
+#### `extras`
+
+A given file may include an `extras` entry that may be used to provide inputs specific to a particular file type
+
+Here is an example:
+
+```yaml
+files:
+  build:
+    output: pyproject
+    includes: # a list of keys from the `dependencies` section which should be included in the generated files
+      - build
+    extras:
+      table: table_name
+      key: key_name
+```
+
+Currently the supported extras by file type are:
+- pyproject.toml
+  - table: The table in pyproject.toml where the dependencies should be written. Acceptable values are "build-system", "project", and "project.optional-dependencies".
+  - key: The key corresponding to the dependency list in `table`. This may only be provided for the "project.optional-dependencies" table since the key name is fixed for "build-system" ("requires") and "project" ("dependencies"). Note that this implicitly prohibits including optional dependencies via an inline table under the "project" table.
+
 ### `channels` Key
 
 The top-level `channels` key specifies the channels that should be included in any generated conda `environment.yaml` files.

pyproject.toml

@@ -22,6 +22,7 @@ requires-python = ">=3.8"
 dependencies = [
     "PyYAML",
     "jsonschema",
+    "tomlkit",
 ]
 
 [project.scripts]

src/rapids_dependency_file_generator/constants.py

@@ -4,6 +4,7 @@
 class OutputTypes(Enum):
     CONDA = "conda"
     REQUIREMENTS = "requirements"
+    PYPROJECT = "pyproject"
     NONE = "none"
 
     def __str__(self):
@@ -22,4 +23,5 @@ def __str__(self):
 
 default_conda_dir = "conda/environments"
 default_requirements_dir = "python"
+default_pyproject_dir = "python"
 default_dependency_file_path = "dependencies.yaml"

src/rapids_dependency_file_generator/rapids_dependency_file_generator.py

@@ -3,13 +3,15 @@
 import textwrap
 from collections import defaultdict
 
+import tomlkit
 import yaml
 
 from .constants import (
     OutputTypes,
     cli_name,
     default_channels,
     default_conda_dir,
+    default_pyproject_dir,
     default_requirements_dir,
 )
 
@@ -87,7 +89,7 @@ def grid(gridspec):
 
 
 def make_dependency_file(
-    file_type, name, config_file, output_dir, conda_channels, dependencies
+    file_type, name, config_file, output_dir, conda_channels, dependencies, extras=None
 ):
     """Generate the contents of the dependency file.
 
@@ -106,6 +108,8 @@ def make_dependency_file(
         CONDA.
     dependencies : list
         The dependencies to include in the file.
+    extras : dict
+        Any extra information provided for generating this dependency file.
 
     Returns
     -------
@@ -127,8 +131,41 @@ def make_dependency_file(
                 "dependencies": dependencies,
             }
         )
-    if file_type == str(OutputTypes.REQUIREMENTS):
+    elif file_type == str(OutputTypes.REQUIREMENTS):
         file_contents += "\n".join(dependencies) + "\n"
+    elif file_type == str(OutputTypes.PYPROJECT):
+        # This file type needs to be modified in place instead of built from scratch.
+        with open(os.path.join(output_dir, name)) as f:
+            file_contents = tomlkit.load(f)
+
+        toml_deps = tomlkit.array()
+        for dep in dependencies:
+            toml_deps.add_line(dep)
+        toml_deps.add_line(indent="")
+        toml_deps.comment(
+            f"This list was generated by `{cli_name}`. To make changes, edit "
+            f"{relative_path_to_config_file} and run `{cli_name}`."
+        )
+
+        # Recursively descend into subtables like "[x.y.z]", creating tables as needed.
+        table = file_contents
+        for section in extras["table"].split("."):
+            try:
+                table = table[section]
+            except tomlkit.exceptions.NonExistentKey:
+                # If table is not a super-table (i.e. if it has its own contents and is
+                # not simply parted of a nested table name 'x.y.z') add a new line
+                # before adding a new sub-table.
+                if not table.is_super_table():
+                    table.add(tomlkit.nl())
+                table[section] = tomlkit.table()
+                table = table[section]
+
+        key = extras.get(
+            "key", "requires" if extras["table"] == "build-system" else "dependencies"
+        )
+        table[key] = toml_deps
+
     return file_contents
 
 
@@ -171,14 +208,15 @@ def get_requested_output_types(output):
     return output
 
 
-def get_filename(file_type, file_prefix, matrix_combo):
+def get_filename(file_type, file_key, matrix_combo):
     """Get the name of the file to which to write a generated dependency set.
 
     The file name will be composed of the following components, each determined
     by the `file_type`:
         - A file-type-based prefix e.g. "requirements" for requirements.txt files.
         - A name determined by the value of $FILENAME in the corresponding
-          [files.$FILENAME] section of dependencies.yaml.
+          [files.$FILENAME] section of dependencies.yaml. This name is used for some
+          output types (conda, requirements) and not others (pyproject).
         - A matrix description encoding the key-value pairs in `matrix_combo`.
         - A suitable extension for the file (e.g. ".yaml" for conda environment files.)
 
@@ -199,14 +237,21 @@ def get_filename(file_type, file_prefix, matrix_combo):
     """
     file_type_prefix = ""
     file_ext = ""
+    file_name_prefix = file_key
     if file_type == str(OutputTypes.CONDA):
         file_ext = ".yaml"
-    if file_type == str(OutputTypes.REQUIREMENTS):
+    elif file_type == str(OutputTypes.REQUIREMENTS):
         file_ext = ".txt"
         file_type_prefix = "requirements"
+    elif file_type == str(OutputTypes.PYPROJECT):
+        file_ext = ".toml"
+        # Unlike for files like requirements.txt or conda environment YAML files, which
+        # may be named with additional prefixes (e.g. all_cuda_*) pyproject.toml files
+        # need to have that exact name and are never prefixed.
+        file_name_prefix = "pyproject"
     suffix = "_".join([f"{k}-{v}" for k, v in matrix_combo.items()])
     filename = "_".join(
-        x for x in [file_type_prefix, file_prefix, suffix] if x
+        filter(None, (file_type_prefix, file_name_prefix, suffix))
     ).replace(".", "")
     return filename + file_ext
 
@@ -237,8 +282,10 @@ def get_output_dir(file_type, config_file_path, file_config):
     path = [os.path.dirname(config_file_path)]
     if file_type == str(OutputTypes.CONDA):
         path.append(file_config.get("conda_dir", default_conda_dir))
-    if file_type == str(OutputTypes.REQUIREMENTS):
+    elif file_type == str(OutputTypes.REQUIREMENTS):
         path.append(file_config.get("requirements_dir", default_requirements_dir))
+    elif file_type == str(OutputTypes.PYPROJECT):
+        path.append(file_config.get("pyproject_dir", default_pyproject_dir))
     return os.path.join(*path)
 
 
@@ -303,12 +350,22 @@ def make_dependency_files(parsed_config, config_file_path, to_stdout):
 
     channels = parsed_config.get("channels", default_channels) or default_channels
     files = parsed_config["files"]
-    for file_name, file_config in files.items():
+    if to_stdout and any(
+        OutputTypes.PYPROJECT in get_requested_output_types(f["output"]) for f in files
+    ):
+        raise ValueError("to_stdout is not supported for pyproject.toml files.")
+    for file_key, file_config in files.items():
         includes = file_config["includes"]
 
         file_types_to_generate = get_requested_output_types(file_config["output"])
 
+        extras = file_config.get("extras", {})
+
         for file_type in file_types_to_generate:
+            if file_type == "pyproject" and "matrix" in file_config:
+                raise ValueError(
+                    "matrix entries are not supported for pyproject.toml files."
+                )
             for matrix_combo in grid(file_config.get("matrix", {})):
                 dependencies = []
 
@@ -367,7 +424,7 @@ def make_dependency_files(parsed_config, config_file_path, to_stdout):
                                 )
 
                 # Dedupe deps and print / write to filesystem
-                full_file_name = get_filename(file_type, file_name, matrix_combo)
+                full_file_name = get_filename(file_type, file_key, matrix_combo)
                 deduped_deps = dedupe(dependencies)
 
                 output_dir = (
@@ -382,6 +439,7 @@ def make_dependency_files(parsed_config, config_file_path, to_stdout):
                     output_dir,
                     channels,
                     deduped_deps,
+                    extras,
                 )
 
                 if to_stdout:
@@ -390,4 +448,7 @@ def make_dependency_files(parsed_config, config_file_path, to_stdout):
                     os.makedirs(output_dir, exist_ok=True)
                     file_path = os.path.join(output_dir, full_file_name)
                     with open(file_path, "w") as f:
-                        f.write(contents)
+                        if file_type == str(OutputTypes.PYPROJECT):
+                            tomlkit.dump(contents, f)
+                        else:
+                            f.write(contents)

src/rapids_dependency_file_generator/schema.json

@@ -12,10 +12,12 @@
                     "type": "object",
                     "properties": {
                         "output": {"$ref": "#/$defs/outputs"},
+                        "extras": {"$ref": "#/$defs/extras"},
                         "includes": {"type": "array", "items": {"type": "string"}},
                         "matrix": {"$ref": "#/$defs/matrix"},
                         "requirements_dir": {"type": "string"},
-                        "conda_dir": {"type": "string"}
+                        "conda_dir": {"type": "string"},
+                        "pyproject_dir": {"type": "string"}
                     },
                     "additionalProperties": false,
                     "required": ["output", "includes"]
@@ -116,7 +118,7 @@
             "items": {"$ref": "#/$defs/matrix-matcher"}
         },
         "output-types": {
-            "enum": ["conda", "requirements"]
+            "enum": ["conda", "requirements", "pyproject"]
         },
         "output-types-array": {
             "type": "array",
@@ -156,6 +158,28 @@
             },
             "additionalProperties": false,
             "required": ["pip"]
+        },
+        "extras": {
+            "type": "object",
+            "properties": {
+                "table": {
+                    "type": "string",
+                    "enum": ["build-system", "project", "project.optional-dependencies"]
+                },
+                "key": {"type": "string"}
+            },
+            "if": {
+                "properties": { "table": { "const": "project.optional-dependencies" } }
+            },
+            "then": {
+                "required": ["key"]
+            },
+            "else": {
+                "not": {
+                    "required": ["key"]
+                }
+            },
+            "additionalProperties": false
         }
     }
 }

tests/examples/invalid/pyproject_bad_key/dependencies.yaml

@@ -0,0 +1,27 @@
+files:
+  py_build:
+    output: pyproject
+    pyproject_dir: output/actual
+    matrix:
+      cuda: ["11.5", "11.6"]
+    includes:
+      - build
+    extras:
+      table: build-system
+      key: dependencies
+channels:
+  - rapidsai
+  - conda-forge
+dependencies:
+  build:
+    specific:
+      - output_types: [conda, requirements]
+        matrices:
+          - matrix:
+              cuda: "11.5"
+            packages:
+              - cuda-python>=11.5,<11.7.1
+          - matrix:
+              cuda: "11.6"
+            packages:
+              - cuda-python>=11.6,<11.7.1

tests/examples/pyproject_matrix/dependencies.yaml

@@ -0,0 +1,26 @@
+files:
+  py_build:
+    output: pyproject
+    pyproject_dir: output/actual
+    matrix:
+      cuda: ["11.5", "11.6"]
+    includes:
+      - build
+    extras:
+      table: build-system
+channels:
+  - rapidsai
+  - conda-forge
+dependencies:
+  build:
+    specific:
+      - output_types: [conda, requirements]
+        matrices:
+          - matrix:
+              cuda: "11.5"
+            packages:
+              - cuda-python>=11.5,<11.7.1
+          - matrix:
+              cuda: "11.6"
+            packages:
+              - cuda-python>=11.6,<11.7.1