Release tagging

[gagern]

The repository happily bumps version numbers in the version.txt file. Currently it's at 5.1.51. The web page however states 5.1.25 as the latest release. And that's also the latest tag in the repository. It's an unannotated tag, so git describe master describes the current master as 4.9.28-408-g7598fdf.

I suggest that you

1. start tagging releases after every update to version.txt
2. use annotated tags for releases
3. update the web page to refer to that latest version

unless these version bumps are not meant as releases, in which case you should make that fact clearer somehow.

[geoffmcl]

@gagern thank you for the opportunity to explain, and yes, will try to add something somewhere to make the release process clearer...

unless these version bumps are not meant as releases, ...

Exactly! The version.txt will be bumped for just about every code change. This make it easy for testing, to make sure you are using the correct version. And documenting, like make sure you are using version 5.1.51 plus, and so on... this has proved very useful in many cases...

Only when we actually make a release will a tag be created. Well much more than that, a branch will also be created. More or less accidentally, this was not done for the 5.1.25 release...

We are right now considering a release 5.2.0, when a tag and a branch, release/5.2 will be created, version.txt set to 5.2.0, the web site updated, etc, etc... Quite a number of steps. Barring any unforeseen problems this is scheduled for the 6/4/2016... ie in a few days... or soonest thereafter...

At that time the master version.txt will be bumped to 5.3.0, and again each code change will bump this - 5.3.1, 5.3.2, etc... As stated many times, we really try to keep the master branch very stable, so for some this can also be treated as a kind of release...

If an important, critical fix comes along, then this will also be pushed back into the release/5.2 branch, and the version bumped to 5.2.1... and so on, but we do not expect any/many of these...

Some of this is already described in VERSION.md, and we will certainly try to improve that wording, and suggestions, even a PR, very welcome...

And we will also try to find a suitable place in the web pages to make this clear. Again, suggestions, or a PR very welcome... Note the site page are in a separate repo - https://github.com/htacg/html-tidy.org - That is new and please excuse any hiccups...

Another of those release steps is to generate the API documentation. That is also done in their own repo - https://github.com/htacg/html-tidy.org.api - again new so...

And finally the release binaries in - https://github.com/htacg/html-tidy.org.binaries - As you can see in Issue 1 there, we do need some help with the DEB creation... As usual I can produce WIN32 32 and 64 bit releases, @balthisar can produce a MAC OS X release, but linux/unix remains a problem...

Any help you can offer in this process would be most appreciated...

[gagern]

Thanks for the detailed explanation, @geoffmcl. If I find the time, I'll suggest some changes to wording to the VERSIONS.md and website repo. In particular I'd put more emphasis on the even/odd numbering style, since that concept is rather common but not self-evident. It's not consistent with the website naming 5.1.25 as latest release, either.

If you wanted to, you could let git do the incemental version numbering on development versions. Do a git tag -a 5.3 at the beginning of the 5.3 series and then you can use git describe --always | sed 's/-\(.*\)-g.*/.\1/' to generate a version number and date -r $(git cat-file -p HEAD | awk '/^committer/{print $5; exit}') +%Y/%m/%d to generate a date for it. This would avoid one commit per merge. Not sure whether that's desirable for you. The patch level increments might be by more than 1, though, since describe counts commits, not merges.

You could generate Linux binaries using Docker. Despite the fact that it's using complicated virtualization machinery in the background, setting up and accessing Docker from Mac (and presumably Windows as well) isn't too difficult.

[geoffmcl]

@gagern thanks for the reply...

I'll suggest some changes to wording...

That would be most appreciated... Yes, the 5.1.25 was a little mistake, and when we do release 5.2.0 maybe its mentioning can be removed, at least from the web site, proudly announcing 5.2.0! Also the target release date for 5.2.0 is Apr 7, not 6 as previously mentioned...

But of course that depends on a lot of things... for instance I lost my internet connection for 3-4 hours today... that is the first time I have had such a stoppage for a long time, and I am struggling on with a weak wifi connection - hope it is all back tomorrow - but it indicates you can never exactly know...

Do a git tag -a 5.3 at the beginning of the 5.3 series...

Ok, that was one thing I only just now started reading up on, that is namely annotated tags. Todate I have always used lightweight tags, but I still do not really see any additional benefit? And I do not see the use of $ git describe ... for anything...

I already have a neat way to generate verhist.log, and that gives me commit value, if/when I want to back up to any previous version... its easy...

And I also have a way to generate a 5.2.0.html, like this 5.1.25.html, which serves as a simple changelog...

And when you add a tag, GitHub has a whole mechanism for generating releases - see releases - so this all works fine...

Do I really need annotated tags?

generate Linux binaries using Docker...

Well I did not quite understand this... First I have two linux boxes... both Ubuntu 14.04... one 64-bit, one 32-bit... so why do I need docker...

I can generate and install tidy using CMake, and unix a makefile... so I do not need docker for that...

I have been able to generate DEB and RPM using CMake/CPack, but that seems cmake version specific - see #344 - a problem...

So began searching for other ways... we have had a Dockerfile suggested before - see #255... I will try to look at it again, but wonder what if offers not in CMake/CPack...

But I am really not looking for yet another way ;=))

But thanks for the info... maybe I did not understand what you were actually suggesting...

[gagern]

I still do not really see any additional benefit?

The main benefit of an annotated tag is that you not only know that a given name refers to a given commit, but also who established that relationship, and when. All the best practice guides I read suggest using lightweight tags for personal bookmarks, and annotated tags for anything you'd want to share.

Personally, I also try to use the annotation text to summarize the changes since the last release, so that the annotation texts can serve as a change log that is easier and faster to read than a full history of commits. But that's a matter of taste, and you don't have to provide any description with your tag annotations if you don't want to. Just say “Release x.y.z” as the message and that's fine.

See also Stack Overflow: Why should I care about lightweight vs. annotated tags?

And I do not see the use of $ git describe ... for anything...

Workflow I used today: go to my node-libtidy source directory. Go to the tidy-html5 subdirectory thereof, which is a git submodule. Do a git fetch origin to get the latest commits. Do a git decribe and read the output as 4.9.28-236-gf567088. Start scratching my head: why the hell did I pick that particular commit when I started the project? Have there really been 236 commits without a release? Doing the same in my libxmljs fork would give me the libxml version, as a plain version number.

Another use case is as the version field reported by some library. I'm using it like this in a project called CindyJS. That's essentially what I suggested for automatic version numbering here as well. There I also have a flag indicating that the source tree has been modified since the commit. So I know that if the commit is from my repo and doesn't carry that flag, then it's equivalent to an official snapshot. Otherwise the user is tinkering with stuff, and I might take that into account when analysing problems.

First I have two linux boxes...

I had assumed that to be the main problem here, thanks for the clarification.

Still, Docker allows you to quickly switch between different versions and distributions of Linux, or more specifically, of the Linux user land. (Linux the kernel is always that of the host system.) That way, you can build and test your package on different versions of Ubuntu, Debian, Fedora, CentOS and whatever else you can think of, and even run distribution-provided testing tools on them to check for possible complaints.

You'd probably still run CMake/CPack, but you'd be doing so inside Docker, in an environment which resembles what the package will finally get installed into. That makes sure that it's linking against the right version of all the libraries. It will also check you're only using those libraries installed inside the docker, not anything you might have lying around on your dev machine and forget to declare as a dependency.

[gagern]

VERSION.md currently writes

The actual versioning of the library more or less follows the Semantic Versioning style.

I don't see how that would fit in with the even/odd versioning scheme. The way I see it, once you have 5.2.0 released, semver would mandate subsequent development to be published as 5.3.0-pre.1 then 5.3.0-pre.2 and so on. (You may use and other prerelease identifier instead of pre.) That way, your non-release versions would be identified as such, and your API changes would fall under the semver item 9:

A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version.

So you wouldn't have to worry about whether a change during development broke API; it would be enough to make sure to bump the correct amount once it's time for a release. So which shall it be, even/odd versioning or semantic versioning? Given the choice, I'd opt for the latter, but it's your call.

[geoffmcl]

@gagern thanks for the comments... these are very helpful!

more or less follows the Semantic Versioning

I guess here I should have stressed the less MORE ;=))

After reading, searching around a lot, looking at the many projects I have work on, I personally like the odds/even style... odds for current, even for official releases...

But let's be very clear here - this is for package maintainers ONLY! Thus the tag and branch... supported well by GitHub...

As stated, we make every effort to keep the master branch absolutely stable, so anyone downloading the master branch is in fact downloading a better release - full stop!

If there is any doubt about a fix it will be done in a branch! We have had many of those, come and gone, and still have one fix, namely issue #329. It is an important fix, but presently needs more work, so only exists within a branch...

And in those branches we often use additional identifiers, like 5.0.0RC1, 5.1.14.EXP1, 5.1.33Exp, 5.1.36issue-373, 5.1.45-Exp1, etc, etc... we have not yet settled on a style for these additional identifiers, but I guess we should! But really additional items like -pre.1, -pre.2 just do not seem very descriptive... But ideas, comments, welcome here...

Again these additions to the version are very important to make sure the testers are using the correct version for testing a change...

But I do not expect package maintainers to try to follow, what can be, daily changes in the master code, even though each change is probably a change-for-the-better. And thus we will more or less arbitrarily choose when to issue a new official release. I really hope this makes some sense!

And as stated, if an important, critical fix comes alone, then it will be pushed back into say the release/5.2 branch, making a new 5.2.1 release...

And to add to that, so far we have had no change that broke backward compatibility to version 5.0.0. There have been a number of additions, even changes, to the API, but these do not break compatibility with any existing config file! When/if we do that, then the major version would change to 6, and so on...

I certainly agree versioning is a very important topic, and very much welcome comments, ideas, suggestions... from everybody... thanks...

PS: My internet connection seems to have recovered! PHEW!

[geoffmcl]

Have commenced the 5.2 release process...

• create release/5.2 branch and tag 5.2.0
• update verhist.log - git/perl scripted
• add a change log - 5.2.0.html - gen git/perl scripted
• push the GitHub new release button
• upload some GitHub binaries, DEB included, but maybe open issue
• reset master version.txt to 5.3.0

All little steps... I have not thought this through completely, but some other things to be done...

• update tests to 5.2.0
• update web page, to announce...
• build api 5.2.0 docs
• update bin releases, adding other OSes, also to GitHub, mirrors?
• check distro/build updates, debian, Ubuntu, homebrew, etc...
• probably lots of things I missed...

Any help with any of these steps, or others, most appreciated... thanks...

[balthisar]

@geoffmcl, sorry I've been MIA lately, but I had a flurry of activity today!

• You got to the web page already.
• Binaries updated using the bins already in the github release page.
• API site updated.
• Mac installer built, signed, and deployed on the release.
• Mac installer materials pushed.

[geoffmcl]

@balthisar, thanks for completing many more steps... beautiful ;=))

As a note to self, the order of the release process should/could have been -

1. reset master version.txt to 5.2.0
2. update verhist.log - git/perl scripted
3. add a change log - README\RELEASE.html == 5.2.0.html - gen git/perl scripted
4. create release/5.2 branch and tag 5.2.0 - $ git push origin release/5.2 --tags
5. push the GitHub new release button
6. upload GitHub binaries - coord with multi-OS binary generators... maybe over 24 hours...
7. update web page to 5.2.0... maybe a link to the RELEASE.html changelog
8. Binaries updated using the GitHub bins. Copy RELEASE.html to tidy-5.2.0\5.2.0.html
9. API site update
10. Adjust Milestones - del 5.1, 5.2 show all closed, if poss., mod/add 5.3, 5.4 target dates?
11. checkout master, reset master version.txt to 5.3.0
12. Check distro/build updates, debian, Ubuntu, homebrew, etc, etc...
13. Probably lots of things I missed...

So at this point it seems 1-11 have been accomplished, in less than 48 hours... that's good!

I suppose Check distro/build updates is the most difficult...

It would be good to develop a list to monitor, and maybe add to the site - http://www.html-tidy.org/#get-tidy - we could list what version is available in what distribution channels... both the Windows and Linux distro channels need to be fill out... again with version... need help here...

I tried to look at homebrew, but could not find the repo containing tidy_html5.rb... I was going to adjust the source to 5.2.0, and maybe present a PR... but could not find it! Can someone help with this?

I wish we could do something about the dreadful situation with debian/ubunutu... Is there some way I, or another, could become the maintainer for orphaned tidy? This is a big important channel that is not open... UGH!

Now that I have a Raspberry Pi, Raspbian OS, I will try to learn about their distribution. You will note I have now added a tidy-5.2.0-arm7.deb... any help with this most appreciated...

You will note Items 2. and 3. are scripted, but I need a place to publish these... at this time it is is just a batch file, genhisthtm.bat, running 3-4 perl scripts - vershist.pl, emailobfuscate.pl, lib_utils.pl, ... maybe need some adj for linux, and sh added... I was thinking of say a tidy-meta repo... sort of "Misc. Tidy Items - shell, batch, scripts", or something along those lines...

Or is it appropriate to add these somewhere in the current repos? I would not mind using tests/tools-perl, but then tools-cmd gets muddied... but I am sure it could benefit from using _environment.bat... not urgent, just for next 5.4 release, in October maybe... advice, ideas, welcome...

And would appreciate any additions to this list... what else should we be looking at as part of the Release process?

This seems to fully cover this specific issue of Release tagging, and maybe this can be closed?

[geoffmcl]

@gagern, @balthisar, @hannob ...

Well, release 5.2 has now been long out the door, and maybe it is time to consider a 5.4 release... see #471

So closing this, as we consider release 5.4... but we should not forget all the above comments... will add a cross-link...