Automate pre-releases

[gojomo]

I know that prior experience suggests that a new release is only 'battle-tested' by the larger community when it becomes official, & thus installed by a default pip install gensim.

However, there are enough changes in recent work, and I suspect large enough gaps in our testing & understanding of actual user patterns, that I think we'd gain a lot by having at least one, and perhaps several, 'soft releases' (tests labeled dev or beta or release-candidate where only eager/advanced users try it out.

I wouldn't expect an instant race-to-install & voluminous feedback. But, I think if it's announced a few times over a few weeks, with both teasers of improvements-needing-testing, and an explicit request for feedback (even if it's just "all was great"), we'd get actionable feedback.

We wouldn't need to coach people about working with git checkouts: the pip/PyPI/Python versions infrastructure should support this. As I understand it, we can push a release with the version 4.0.0.dev1 or 4.0.0b1 – or any of the other dev/prerelease formats in PEP440. Then, anyone who explicitly tries pip install gensim still gets only the official release, but anyone who does pip install --pre gensim, or an installation explicitly naming the exact release, will get the pre-release version.

We would create a Wiki page about the availability of a 4.0.0 pre-release, with info about how to install it (or roll-back if things go wrong). It'd also link to the migration tips. We'd announce on project list, project twitter, gitter, etc and encourage feedback via replies or new issues.

Some big ways I think it would help include:

• We almost certainly have some little issues that will be easy to fix as soon as reported; we might have some bigger ones because use-in-the-field is different than we expect or unit-test for. (I'm especially concerned about users who've done customization, or deployment optimizations around mmap, FastText, or multi-step/retraining workflows.) We almost certainly have some outdated docs/tutorials, in project materials or 3rd party sites. If these are 1st hit by naive/beginner/unaware users, they'll generate much more frustration (including for us in incomplete reports of problems) than if they're 1st hit by users who know what it means to try a new version, who'll make better issue reports and be more patient handling issues.
• While we could fix such issues in little 4.0.1, 4.0.2, etc releases, that involves more mess & pressure. If there's a serious or embarrassing issue, I'd like to be able to think about it and fix it right over the course of a day/week, rather than rush a micro release ASAP (which might then miss something that comes up one day later).
• It lets any attentive users feel a bit special & helpful, whether they find bugs or simply report "all smooth" - and might contribute to more positive attention in a slightly-later improved release, as they chime in with any variant of "been working great for me for weeks" or "I found a big and I'm happy to say it's fixed".
• There are still a few festering things that may be best to change, due to API disruption, coincident with the 'major' revision than in a 4.0.1 or 4.1.0. Ensuring mmap is still up to snuff is one already on your list, but I'd also like to restore the distinct stages of build_vocab() that were once more amenable to saving-the-vocab, then trying multiple train-parameters, than they are now. (I have some interim work on this I can show in a PR tomorrow.) I'll highlight a few other things that might fit this category on other issues.

Overall, my sense is that a good process for polish & positive launch attention could be something like: (1) test release this week, see what feedback comes back; (2) 2nd test release in a couple weeks, see what feedback comes back; (3) official release a couple weeks later; (4) 4.0.1 a month after that.

But if we don't do that, we might find ourself in something like: (1) 4.0.0 release ASAP; (2) bugs & complaints along the "why wasn't I warned" theme; (3) scrambles to do 4.0.1, 4.0.2 soon after; (4) eventually a 4.0.3 a couple weeks after, that's the 1st release that's actually reasonable for non-experts to risk. But that's then based more on reactive feedback from whoever stepped on things than users who consciously chose to affect an upcoming release, and many less-attentive users still have the 4.0.0/4.0.1/4.0.2 for a while, until they hit some new urgent reason to update.

[piskvorky]

An RC release is a good idea, I'm +1. @mpenkov do you see any issue with that?

[mpenkov]

No objections from me.

[gojomo]

Great!

Regarding labeling: 'Release Candidate' implies "if nothing found, this could be final". And so, indicates higher expectations, such that I'd not want to push any such-labeled release until the mmap issue (for example) is fixed.

But 'dev' or 'a'(lpha) or 'b'(eta) indicates "known fixes/changes still coming", and thus needn't wait for anything. If we had a one-button or otherwise very-simple prerelease-push process - where the right labels are put in git/source-code, then the next CI auto-build is grabbed from the cloud & put on PyPI under the right label - we could do that at any time, including ASAP, without waiting for any more merges/review. We'd just want to be clear on the 'pre-release' wiki page about what's still pending/iffy.

[piskvorky]

Yes. I don't want to release until the pre-3.8.3 compatibility question is settled though. @gojomo can we have a call to finalize this?

[gojomo]

Well, my point above is that by labeling the pre-release sufficiently tentatively, neither the 3.8.3-or-earlier issue or others need be blockers. We'd just note, "that may still change".

But, I just put a long summary of my reasoning & what I think would work well as a back-compat policy, for right now & the future, in #2967. Happy to discuss further by call either utc2300+ today (Wed/30th) or utc1700-2100 (Thu/1st) if either of those times work.

[piskvorky]

-1 on releasing anything without clarity on model serialization. Serialization is functionality that I don't want to change, even from a beta/release candidate. Stability is important.

I'll try to read #2967, was hoping for a concise list of pros+cons, to be decided during a live call.

[gojomo]

The point of test-prereleases-to-willing-experts is that anything can be provisionally/half/tentatively tried. They are an official declaration that, "for this release, stability is not important - and we will resume our actual stability commitments on the next official release."

Users who choose to use such prereleases, like those that work from dev git branches, know what they're getting into. If an 'alpha' or 'beta' release breaks some loading we wanted to support, and then actually fix before an official release, no harm. If it temporarily supports some loading that (for good reasons) we then discontinue before the official release, also no harm. The 'alphaor 'beta' wasn't any promise of stability, explicitly. (A true "Release Candidate", though, would imply no major functionality expansions/reductions - which is why if we did a prerelease right this very moment, I'd label itdevoraorb`. )

No matter how rough and subject-to-change, test prereleases can start collecting the rare, valuable, uncompensated testing feedback that would help immensely, and holding such prereleases back to get things "certain" defers that.

I think of it this way: at this very moment, someone, a hobbyist or professional, may be starting or dusting off a project where they'd be willing to risk a 'new & improved' Gensim, for the sake of learning & enjoying the benefits of the latest stuff 1st. If they can get it via an easy, official pip install --pre gensim (rather than a full git sync & developer install/build), they'll do it, and we'll get valuable free labor. If they can't get it that way, they won't, and we've lost a low-risk/big-upside opportunity that might not wander by again for another few days, or weeks.

Personally, I think it'd be ideal if we set up an auto-dev-release process, where every week or few, as long as develop is passing tests, a script increments a .devN number, lets the CI builds finish, then uploads those CI builds to PyPI under a prerelease-naming. Then we'd always be helping the adventurous help us, at no risk to the stodgy who prefer officially stable releases. We'd also then never need to have this conversation about future releases again: the benefits would be constant, automatic, perpetual as long as the build works/passes, which is good project hygiene anyway.

A further step in this direction would then be to extend that script such that, when the in-project version-identifier is detected as "X.Y.Z", and a few other surface "ready-for-release" checks pass (all tests pass, new version has a section header in changelog, etc), the release happens completely automatically, including PyPI upload and promotion of the in-source-tree identifier to the presumed next-dev-prerelease value.

[piskvorky]

@gojomo is your email address still gojomo@x**y.com?

[piskvorky]

I'm putting this issue on the back burner again, since no one took it up. Not really blocking for the next release.

[gojomo]

@gojomo is your email address still gojomo@x**y.com?

Yes! (Also: sent a belated reply to your email inquiries a week ago – May 11 – in case that reply wound up mis-sorted somewhere.)

[gojomo]

@piskvorky If you don't see an email from me in the last few hours, please check spam folders!