Meta: Improve our release process

[fingolfin]

Meta: Improve our release process

Based on discussion on our mailing list, I propose some steps towards
a better release process. Details can be found in the corresponding
mailing list thread; this issue is based on one of my mails in that thread.

This is a meta-issue, tracking our overall progress on achieving this.
Feel free to clarify and reorder these bullet point, and to add links to sub-issues
which discuss them in more detail.

Release schedule

We should decide on a relatively rigid release schedule, e.g.: "Major releases happen in mid-february and mid-august; minor releases typically happen every 1-2 months afterwards, as need arises."

Also decide on (and document) the various phases of schedule: For the major release: When is a new stable branch made; for major and minor releases: at which point should stable be "frozen"; when will we last pick up packages; etc.

Then document this release schedule on a webpage that is easily found and linked to from suitable places. It should both list the current releases schedule, and the general pattern.

We probably will want to "freeze" the stable branch during certain times in the
release process. We need to document that, and what a "freeze" means, exactly. E.g.: Only regression fixes are allowed, no other changes, not even bug fixes.

• Decide on a general release schedule, including its phases
• Document the release schedule (both general, and also the specific one for the next N months)
• Document the various phases of the release schedule
• Give some indications when we might delay making a new stable branch, and when not
• Decide on rules for "freezing" the stable branch
• Let's have an official release manager, or a small release team, at most 2-3 people.
• Document that also package updates won't be picked up automatically during part of the release process (will only happen manually if there is urgent need, e.g. to fix a regression).
• Improve automation of releases (if needed?)

Quality metrics

Put more and better metrics in place, in particular:

• Improve test suite, run more tests in travis
• Improve test suite code coverage, visualize the results on a webpage (see Make it super-easy to collect code coverage for test suite #545, Create a website where we display the latest code coverage results for our testsuite #546, Gamify test coverage #547)
• Measure speed (e.g. of the test suite) over time, visualize the results on a webpage; something similar to http://speed.pypy.org/ or http://perf.rust-lang.org
• Put rules in place that restrict commits / pull requests which degrade performance (they must be justified in some way, and it must be a good justification)

[olexandr-konovalov]

Update: release checklist created under #1477 and available at https://github.com/gap-system/gap-distribution/blob/master/DistributionUpdate/RELEASE_CHECKLIST.md

[fingolfin]

We've done this during GAP Days 2021