Release: changelog can be out-of-sync

[victorlin]

With the practice of adding changelog entries in individual PRs rather than collectively as the first step of a release, there is potential for changelog updates to become out-of-sync (example: #1619).

Possible solutions

1. Add a step in the release docs to check that no changes have been made to already released versions.
2. Write a script to check that changes to the changelog are only under the __NEXT__ section. Run this as part of PR checks and release checks.
3. Write a changelog-specific merge driver that would properly relocate entries during a merge and configure it to be used for CHANGES.md.
4. Use towncrier.

[tsibley]

This happens during the merge of the topic branch into the main branch. The topic branch correctly placed the entry under __NEXT__

and the merge algo preserved the entry's immediately surrounding context (the adjacent entries). This put it under the already-released heading instead of the __NEXT__ heading (because it's only a computer and not aware of the meaning of those headings).

Note the "combined diff" with two initial ± columns. For this particular merge, you'll need to increase diff context to actually see the above (e.g. -U10) given the default --diff-merges=dense-combined (--cc) of git show for merge commits. Alternatively, use --diff-merges=combined (-c).

If you do the merge on the command-line, you can inspect the merge result a) during committing when commit.verbose is enabled in your gitconfig or b) afterwards with git show. If you notice it's wrong then, you have a chance to pause or abort the merge and fix it before continuing or retrying. The output of git merge will also (subtly) alert you to this with a message, e.g. Auto-merging CHANGES.md.

As a possible (sophisticated) solution that would "get it right" without manual or automatic checks, we could write a changelog-specific merge driver that would properly relocate entries during a merge and configure it to be used for CHANGES.md.

[victorlin]

Thanks for the detailed explanation. This is one of the downsides to GitHub's PR page – on #1599 I don't see any way to preview the fact that the changes will be added under the already-released heading, and clicking the magical merge button did not show anything like Auto-merging CHANGES.md.

[tsibley]

As a possible (sophisticated) solution that would "get it right" without manual or automatic checks, we could write a changelog-specific merge driver that would properly relocate entries during a merge and configure it to be used for CHANGES.md.

(I'm going to try hard to resist doing this, but it might be too tempting.)

[victorlin]

Another possible solution is towncrier which had been previously proposed in #980 (comment). We did not pursue at the time but it does prevent this type of problem.

Not advocating for it now – just noting that if editing a single changelog file does not work well with multiple open PRs (as we are starting to see over time), it could be worth considering.