Add a tool to create and render changelog entries

Writing changelog entries is not super fun. Writing them after the
fact, especially when you didn't write the original feature, is even
less fun. And merge conflicts because of synchronizing a changelog
file are SUPER not-fun.

This change solves those problems. It introduces a changelog tool
similar to what many of the SDK teams use (and indeed cribs some code
from them).

> Why?

Using distinct files for changelog entries solves a major problem:
merge conflicts. These are horribly annoying to deal with during
development, especially on a file that is essentially documentation
and semantically has no reason to ever conflict. This problem is
less bad for Smithy itself than it is for SDKs that have daily
releases, but it's a problem for us too with our relatively large
team.

> What's different?

This differs from what other SDKs do in a few ways. Firstly, and
most notably, we aren't deriving a version number from these. We
could! It would not be hard! But I elected not to do that in this
PR since it's not as big a problem for us.

We also have a habit of linking the PRs for all of our changes in
the changelog. This is actually kinda annoying because it creates
a bit of a chicken-egg scenario. This is why I added the github
action to automatically add it. If the action is annoying we can
always get rid of it and just stop relying on the links. But that
would be a shame.

Mechanically there's also a change in that I used modern python to
write this, and updated any borrowed code to also be modern python.

> Why not use jmeslog?

jmeslog is a standalone tool derived from the python team's
changelog tool (which started it all). It is essentially exactly what
they do and want, but there's a few things that prevent us from using
it. Notably, we need the current date and pr links. jmeslog doesn't
have that and doesn't currently allow for easy extension in that way.

It also has things we don't necessarily want, like a category. That's
really mostly used in the sdks to indicate changes made for a
particular service. With hundreds of services, that's an issue worthy
of calling out. But smithy itself doesn't need it.

But also, it's a dependency, and it has dependencies. I'd rather not
have dependencies if I can avoid it. And what we need is ultimately
not that complex. And doing it ourselves gives us some flexibility.

> Why not derive changelogs from commits?

The audience for changelogs and the audience for commit messages are
two separate (though sometimes intersecting) audiences. Commits may
contain conext about technical junk that only mattters to people
working on the code itself.

There is also not necessarily a 1-1 relationship between a changelog
entry and a commit. Large features are often broken up into logical
commits that make reviewing and code diving easier, but aren't
important to the changelog. You might also have commits that make
changes unimportant to the changelog audience, such as formatting
changes or changes to CI configurations.

Commits are also, critically, ***immutable***. Did you make a typo
in your commit? Now it's preserved forever in your changelog. With
JSON files, you can just change them. Those changes are now part of
the commit record!

Also, on a more subjective level, have you seen repos that have
semantic commits? They're ugly. Having more than half your commits
starting with `chore` just makes it look like your project is in
maintenance mode and/or like you hate your job. And if you're
following best practices for commit messages, that's eating into
your 50 character limit for the title.

> Any unfortunate bits?

There's no line wrapping, so you better believe the changelog is
gonna have some long lines. It doesn't really matter though, because
it'll all be natural when you're seeing it fully rendered. The line
boundary thing these days mostly helps with readability when you're
editing a file, but you will never manually edit the file.

> What's next?

Provided this goes through, the next step will be to backfill all
the current changelog entries. That'll certainly be a task.

A github action to create the version bump pr would be sweet. We'd
need to add the ability to derive a version number from the feature
list, which isn't hard. Then the script could just aslo perform the
version bump by writing that one file. Then it could be just a button
press away in github. Maybe as a fully automated release?

.changes/README.md

@@ -0,0 +1,50 @@
+# Changelog Tool
+
+This directory contains the changelog tool that is used to generate the changelog
+from a set of entry files that can be added to pull requests without risking
+merge conflicts.
+
+## Usage
+
+### When Writing a Pull Request
+
+Before submitting a pull request, run the `new-change` script. If called
+without arguments, it will prompt for all the information it needs. It will
+then store that information in a file in the `next-release` directory that you
+must commit.
+
+The tool will ask for an optional pull request number. If you haven't opened a
+pull request yet, this is fine. Simply do not fill in that line, and when you
+do open a pull request a bot will automatically add a pull request comment with
+a change suggestion that adds it.
+
+To get details about optional arguments to the command, run `new-change -h`.
+
+### When Releasing Smithy
+
+Before performing a release, ensure that every pull request since the last
+release has an associated changelog entry. If any entries are missing, use
+the `new-change` tool as described above to add them in.
+
+You may optionally edit or combine the entries as is necessary. If you combine
+entries, ensure that the combined entry contains all of the relevant pr links.
+
+Once the entries have been verified, use the `render` tool to combine the
+staged entries and generate the changelog file. From the root of the Smithy
+repository, run the following with the version being released:
+
+```sh
+> ./.changes/render --release-version RELEASE_VERSION > CHANGELOG.md
+```
+
+If the `VERSION` file has already been updated, this can be simplified:
+
+```sh
+> ./.changes/render --release-version "$(cat VERSION)" > CHANGELOG.md
+```
+
+Then commit the changelog and the `.changes` directory:
+
+```sh
+> git add CHANGELOG.md .changes
+```

.changes/amend

@@ -0,0 +1,61 @@
+#! /usr/bin/env python3
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+from argparse import ArgumentParser
+
+from tool.amend import amend
+
+
+def main() -> None:
+    parser = ArgumentParser(
+        description="""\
+            Amend recently-introduced changelog entries to include a pull request \
+            link. This is intended to be run in GitHub as an action, but can be run \
+            manually by specifying parameters GitHub would otherwise provide in \
+            environment variables.
+            
+            This only checks entries that have been staged in the current branch, \
+            using git to get a list of newly introduced files. If the entry already \
+            has one or more associated pull requests, it is not amended.""",
+    )
+    parser.add_argument(
+        "-n",
+        "--pull-request-number",
+        required=True,
+        help="The numeric identifier for the pull request.",
+    )
+    parser.add_argument(
+        "-r",
+        "--repository",
+        help="""\
+            The name of the repository, defaulting to 'smithy-lang/smithy'. This is \
+            provided by GitHub in the GITHUB_REPOSITORY environment variable.""",
+    )
+    parser.add_argument(
+        "-b",
+        "--base",
+        help="""\
+            The name of the base branch to diff against, defaulting to 'main'. This \
+            is provided by GitHub in the GITHUB_BASE_REF environment variable.""",
+    )
+    parser.add_argument(
+        "-c",
+        "--review-comment",
+        action="store_true",
+        default=False,
+        help="""\
+            Instead of amending the local files on disk, post a review comment on the \
+            PR. This will also post a normal comment if no changelog entry is found.""",
+    )
+
+    args = parser.parse_args()
+    amend(
+        base=args.base,
+        repository=args.repository,
+        pr_number=args.pull_request_number,
+        review_comment=args.review_comment,
+    )
+
+
+if __name__ == "__main__":
+    main()

.changes/new-change

@@ -0,0 +1,47 @@
+#! /usr/bin/env python3
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+from argparse import ArgumentParser
+
+from tool import ChangeType
+from tool.new import new_change
+
+
+def main() -> None:
+    parser = ArgumentParser(
+        description="""\
+            Create a new changelog entry to be staged for the next release. Required \
+            values not provided on the command line will be prompted for.""",
+    )
+    parser.add_argument(
+        "-t",
+        "--type",
+        choices=[t.name.lower() for t in ChangeType],
+        help="The type of change being logged.",
+    )
+    parser.add_argument(
+        "-d", "--description", type=str, help="A concise description of the change."
+    )
+    parser.add_argument(
+        "-p",
+        "--pull-requests",
+        nargs="+",
+        help="The pull request that implements the change.",
+    )
+    parser.add_argument(
+        "-r",
+        "--repository",
+        help="The name of the repository, defaulting to 'smithy-lang/smithy'.",
+    )
+
+    args = parser.parse_args()
+    new_change(
+        change_type=ChangeType[args.type.upper()],
+        description=args.description,
+        pull_requests=args.pull_requests,
+        repository=args.repository,
+    )
+
+
+if __name__ == "__main__":
+    main()

.changes/render

@@ -0,0 +1,49 @@
+#! /usr/bin/env python3
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+from argparse import ArgumentParser
+
+from tool.release import release
+from tool.render import render
+
+
+def main() -> None:
+    parser = ArgumentParser(
+        description="""\
+            Render the changelog as markdown, optionally including pending features \
+            as a new release.""",
+    )
+    release_group = parser.add_argument_group(
+        "release",
+        description="""\
+            These arguments allow for releasing all pending features in the \
+            next-release folder as a new release. If not set, the exisiting releases \
+            will be re-rendered.""",
+    )
+    release_group.add_argument(
+        "-v",
+        "--release-version",
+        type=str,
+        help="""\
+            The version to use for the staged changelog release. If set, all pending \
+            features will be compiled into a release.""",
+    )
+    release_group.add_argument(
+        "-d",
+        "--release-date",
+        type=str,
+        help="""\
+            The date of the release in ISO format (e.g. 2024-11-13). If not set, \
+            today's date, according to your local time zone, will be used.""",
+    )
+
+    args = parser.parse_args()
+
+    if args.release_version:
+        release(args.release_version, args.release_date)
+
+    render()
+
+
+if __name__ == "__main__":
+    main()

.changes/tool/__init__.py

@@ -0,0 +1,96 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+import datetime
+import json
+from dataclasses import asdict, dataclass, field
+from enum import Enum
+from pathlib import Path
+from typing import Any, Self
+
+CHANGES_DIR = Path(__file__).absolute().parent.parent
+NEXT_RELEASE_DIR = CHANGES_DIR / "next-release"
+RELEASES_DIR = CHANGES_DIR / "releases"
+REPO_ROOT = CHANGES_DIR.parent
+
+
+class ChangeType(Enum):
+    FEATURE = "Features", 1
+    BUGFIX = "Bug Fixes", 2
+    DOCUMENTATION = "Documentation", 3
+    BREAK = "Breaking Changes", 0
+    OTHER = "Other", 4
+
+    def __init__(self, section_title: str, order: int) -> None:
+        self.section_title = section_title
+        self.order = order
+
+    def __str__(self) -> str:
+        return self.name.lower()
+
+    def __lt__(self, other: Self) -> bool:
+        return self.order < other.order
+
+
+@dataclass
+class Change:
+    type: ChangeType
+    description: str
+    pull_requests: list[str] = field(default_factory=list)
+
+    @classmethod
+    def from_json(cls, data: dict[str, Any]) -> Self:
+        return cls(
+            type=ChangeType[data["type"].upper()],
+            description=data["description"],
+            pull_requests=data.get("pull_requests") or [],
+        )
+
+    @classmethod
+    def read(cls, file: Path) -> Self:
+        return cls.from_json(json.loads(file.read_text()))
+
+    def write(self, file: Path | None = None) -> str:
+        contents = json.dumps(asdict(self), indent=2, default=str) + "\n"
+        if file is not None:
+            file.write_text(contents)
+        return contents
+
+
+def _today() -> str:
+    return datetime.date.today().isoformat()
+
+
+@dataclass
+class Release:
+    version: str
+    changes: list[Change]
+    date: str = field(default_factory=_today)
+
+    def change_map(self) -> dict[ChangeType, list[Change]]:
+        result: dict[ChangeType, list[Change]] = {}
+        for change in self.changes:
+            if change.type not in result:
+                result[change.type] = []
+            result[change.type].append(change)
+        return dict(sorted(result.items()))
+
+    @classmethod
+    def from_json(cls, data: dict[str, Any]) -> Self:
+        return cls(
+            version=data["version"],
+            changes=[Change.from_json(c) for c in data["changes"]],
+            date=data["date"],
+        )
+
+    @classmethod
+    def read(cls, file: Path) -> Self:
+        return cls.from_json(json.loads(file.read_text()))
+
+    def write(self, file: Path | None = None) -> str:
+        contents = json.dumps(asdict(self), indent=2, default=str) + "\n"
+        if file is not None:
+            file.write_text(contents)
+        return contents
+
+    def __lt__(self, other: Self) -> bool:
+        return self.date < other.date

.changes/tool/amend.py

@@ -0,0 +1,89 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+import os
+import subprocess
+from pathlib import Path
+
+from . import REPO_ROOT, Change
+from .github import post_comment, post_review_comment
+
+DEFAULT_REPO = "smithy-lang/smithy"
+GITHUB_URL = os.environ.get("GITHUB_SERVER_URL", "https://github.com")
+
+
+def amend(
+    *,
+    pr_number: str,
+    repository: str | None = None,
+    base: str | None = None,
+    review_comment: bool = False,
+) -> None:
+    repository = repository or os.environ.get("GITHUB_REPOSITORY", DEFAULT_REPO)
+    pr_ref = f"[#{pr_number}]({GITHUB_URL}/{repository}/pull/{pr_number})"
+
+    changes = get_new_changes(base)
+    if not changes and review_comment:
+        print("No changelog found, adding reminder comment.")
+        description = os.environ.get("PR_TITLE", "Example description").replace(
+            '"', '\\"'
+        )
+        comment = (
+            "This pull request does not contain a staged changelog entry. To create "
+            "one, use the `./.changes/new-change` command. For example:\n\n"
+            f'```\n./.changes/new-change --pull-requests "#{pr_number}" '
+            f'--type feature --description "{description}"\n```\n\n'
+            "Make sure that the description is appropriate for a changelog entry and "
+            "that the proper feature type is used. See [`./.changes/README`]("
+            f"{GITHUB_URL}/{repository}/tree/main/.changes/README) or run "
+            "`./.changes/new-change -h` for more information."
+        )
+        post_comment(
+            repository=repository,
+            pr_number=pr_number,
+            comment=comment,
+        )
+
+    for change_file, change in changes.items():
+        if not change.pull_requests:
+            print(f"Amending changelog entry without associated prs: {change_file}")
+            change.pull_requests = [pr_ref]
+
+            if review_comment:
+                print("Posting amended change as a review comment.")
+                comment = (
+                    "Staged changelog entries should have an associated pull request "
+                    "set. Commit this suggestion to associate this changelog entry "
+                    "with this PR.\n\n"
+                    f"```suggestion\n{change.write().strip()}\n```"
+                )
+                post_review_comment(
+                    repository=repository,
+                    pr_number=pr_number,
+                    comment=comment,
+                    file=change_file,
+                    start_line=1,
+                    end_line=len(change_file.read_text().splitlines()),
+                )
+            else:
+                print("Writing amended change to disk.")
+                change.write(change_file)
+
+
+def get_new_changes(base: str | None) -> dict[Path, Change]:
+    base = base or os.environ.get("GITHUB_BASE_REF", "main")
+    print(f"Running a diff against base branch: {base}")
+    result = subprocess.run(
+        f"git diff origin/{base} --name-only",
+        check=True,
+        shell=True,
+        capture_output=True,
+    )
+
+    new_changes: dict[Path, Change] = {}
+    for changed_file in result.stdout.decode("utf-8").splitlines():
+        stripped = changed_file.strip()
+        if stripped.startswith(".changes/next-release") and stripped.endswith(".json"):
+            file = REPO_ROOT / stripped
+            print(f"Discovered newly staged changelog entry: {file}")
+            new_changes[file] = Change.read(file)
+    return new_changes