Breaking change in cargo test --doc behavior not documented in 1.62 release notes, despite relnotes-tagged PR.

[steffahn]

Hi, I just came across this “bug” report rust-lang/rust#98735 was made because of an intentional breaking change (bugfix of rust-lang/rust#88038) which wasn’t documented in the release notes of 1.62 where the change happened. However, the PR that made the change (rust-lang/rust#96630) has a relnotes tag, so it’s supposed to be documented in the release notes AFAICT.

@Mark-Simulacrum suggested opening an issue here for investigating this case (rust-lang/rust#96630 (comment)).

[Mark-Simulacrum]

I ran our relnotes generator (https://github.com/rust-lang/relnotes) for 1.62 again and that does show this in the UNSORTED relnotes category. I don't recall looking at it when producing the 1.62 release notes, so my current best guess is that I just accidentally deleted that line when producing the PR for the release notes.

Unfortunately, I'm not sure what we can try to do to avoid that being a problem in the future. Release notes generation currently involves at least looking at ~most PRs merged in a release cycle (~730 for 1.62), by title, which as you can imagine is a pretty, uh, easy to make mistakes in process. My personal hope is that we can make release notes generation more of a community-oriented task where adding something to release notes is largely just done by tagging it relnotes (and possibly poking rustbot or some other tool with a short snippet; that's a harder part). That would at least make this case ~impossible, since anything tagged relnotes would have to make it in.

The other possibility is trying a manual audit after producing the cut-down version of the release notes, which doesn't seem terrible (10 things marked as such in 1.62) but is another manual step which just adds to our limited overhead budget.

I think I'm inclined to say that the best fix here is the automated generation based on relnotes-tags -- that should mostly work okay, and gives us a cleaner path toward getting this right in the future. I don't think it'll be that hard to make that happen, mostly just means doing some work either in triagebot (my preference, personally) or in some other ad-hoc tooling. (I am personally hoping to [continue to] invest in automating our release process as much as we can, so this goes into that time budget).