add release.sh script to generate changelog, release to GH and Dockerhub

[kba]

How to create PR for updated submodules

1. Create a new branch with any name, e.g. git checkout -b update-$(date +'%Y-%m-%d')
2. Selectively update submodules in a way that makes sense to you. To upgrade
   all submodules to the latest upstream versions, run ./release.sh update. To
   update only a specific submodule, such as ocrd_cis, run ./release.sh update ocrd_cis. Or update individual submodules manually: cd <submodule>; git pull origin master (replace master if you want to merge another branch instead)
3. Manually check with git status that the changes are consistent with what
   you want to update in the PR.
4. git add / git commit
5. git push to your ocrd_all fork on GitHub
6. Open a new PR for that branch (The git push request will show you the right URL)

How to merge update PR for updated submodules

1. Wait for CI to successfully finish (ensuring that the maximum image can be built)
2. Do not merge on GitHub. Do the following locally:
3. Check out the master branch locally, git pull to make sure it's up-to-date.
4. Merge the PR branch with the --no-commit flag so the merge commit doesn't hide the changes: git merge --no-commit pr-branch
5. Generate the changelog: ./release.sh changelog.
6. Inspect the CHANGELOG.md and remove superfluous information like merge commits or spurious newlines. Copy the new section to the clipboard (see step 8)
7. Release to GitHub with ./release.sh release-github. This will take care of comitting, tagging and pushing the release.
8. Create a new release on GitHub, paste the new changelog section as the release notes.

[kba]

Here's how the Changelog will look like (core, ocrd_cis, workflow-configuration updated):

Changelog

v2020-06-10

core b9977a2..dc0cc14

Release: v2.7.0

• 📦 v2.9.0
• Merge branch 'master' into validate-file-id
• OcrdMets.add_file validates file ID syntax
• generateDS PAGE API: Escape literal % signs, document recent changes
• generateDS PAGE API: allow custom filenames in _add_method
• 📦 v2.8.1
• 📝 changelog
• remove_file: return first element if only one removed
• mets/workspace.remove_file_group: support regex USE, #458
• mets/workspace.remove_file: delegate to find_files, #458
• 📦 v2.8.0
• Merge branch 'master' into get-all-regions
• Document extend_AllIndexed validate_contiunuity param
• 📝 changelog
• Merge branch 'master' into get-all-regions
• extend_AllIndexed: increment @index when adding elements
• add sort_AllIndexed to sort in-place
• get_AllIndexed: index_sort parameter to enable/disable sorting
• get_AllIndexed: allow filtering by child type
• add get_UnorderedGroupChildren, let get_AllIndexed handle UnorderedGroupIndexed properly
• reading order test sample: add unorderedgroups for testing
• get_AllRegions: Update example
• get_AllRegions: fix recursion
• 📝 get_AllRegions: document example
• get_AllRegions: catch negative depth, test depth==0
• get_AllRegions: differentiate "reading-order"/"reading-order-only"
• add_AllIndexed -> extend_AllIndexed
• 🎨 pylint
• regenerate PAGE API
• Merge branch 'get-all-regions' of https://github.com/kba/ocrd-core into get-all-regions
• get_AllRegions: _region_id method unneccessary now
• get_AllRegions: clean-up merge artifacts and reorganize
• regenerate PAGE API
• recursion (with both finite or arbitrary depth) for get_AllRegions
• Update ocrd_models/README.md
• Update ocrd_models/README.md
• Update ocrd_models/README.md
• README: explain how to add user methods to PAGE API
• get_AllRegions: adapt to signature proposed in adapt TESSDATA to resmgr default #240, test with order=reading-order
• refactoring: move generateDS methods to their own files
• Merge branch 'get-all-regions' of https://github.com/kba/ocrd-core into get-all-regions
• Merge branch 'hotfix-ocrd-page-exports' into get-all-regions
• Update ocrd_models/ocrd_page_user_methods.py
• PageType.get_AllRegions to list all kinds of regions

ocrd_cis a6a2ecd..f2b42d4

• Update to latest jar
• only override log level if it is actually set
• Merge pull request tessdata "fast" vs "best" #52 from cisocrgroup/revert-49-load-gz
• load gzipped models

workflow-configuration 403ef75..54f8a9e

• ocrd-make: override loglevel only if $(LOGLEVEL) explicitly set
• pass on LOGLEVEL to tool invocation

2020-05-31

Update:

• ocrd_olena: e000f24 (1.1.7)
• tesseract: 62eae84
• ocrd_cis: a6a2ecd
• ocrd_tesserocr: 9d2253e (0.8.3 +1)
• sbb_textline_detector: 3593506
• workflow-configuration: 403ef75

Added:

• CHANGELOG.md to track changes

[bertsky]

The workflow would be:

* Create a PR branch (e.g. `git checkout -b update-$(date +'%Y-%m-%d')`

You mean this as an example, right? I could still name the branch however I want IIUC.

* Update all submodules: `./release.sh update`

IIUC, I can also list the modules to be updated after this command. Or I do the submodule updates myself manually (esp. if I am not interested in a master version, but in feature branches or PR branches), right?

* Manually check/adapt the submodule checkouts for consistency

You mean after committing, right? I ask because our Makefile rules will immediately reset the worktree to HEAD.

* Open the PR and wait for CI

…after pushing, of course.

The following parts can only be done by collaborators, of course:

* **Do not merge on GitHub**, but locally: `git checkout master; git merge update-....`

you mean git merge --no-commit pr-branch, right? (This is crucial, because with autocommit the subsequent steps won't see any status changes in the submodules.)

* Generate the CHANGELOG.md: `./release.sh changelog`

…and afterwards validate that this looks correct by git diff CHANGELOG.md HEAD, right?

* Release to GitHub: `./release.sh release-github` (which will commit the changelog with commit message `:package: $version`, tag that commit as `$version`, push branch and tag to GitHub). This will trigger the Circle CI deploy script which will, after building/pushing all the min/med/max/git/non-git images, tag the `ocrd/all:maximum` image as `ocrd:all:$version` and push to Dockerhub).

Ok, so in summary:

• What will change for contributors?
  Nothing. (This is good!)
• What will change for collaborators who want to merge PRs?
  They will have to apply the PR locally through release.sh, using their clone's working directory a lot (so they must be careful about local.mk or shell variables deactivating components) and downloading all submodules. This is not a big change for their own PRs, but a little extra effort for foreign PRs.

However, this process is still opt-in, i.e. as long as we don't forbid merging on Github UI altogether, one can still use the same procedure as before (esp. for small PRs which only change the README or ocrd_all Makefile) – this won't trigger a git tag or GH release or CHANGELOG update. It will trigger a Docker alias for the current date, though.

Good work!

[kba]

The workflow would be:

* Create a PR branch (e.g. `git checkout -b update-$(date +'%Y-%m-%d')`

You mean this as an example, right? I could still name the branch however I want IIUC.

Yes, branch name is arbitrary and not used in the script.

* Update all submodules: `./release.sh update`

IIUC, I can also list the modules to be updated after this command. Or I do the submodule updates myself manually (esp. if I am not interested in a master version, but in feature branches or PR branches), right?

Yes, that's what I meant with

* Manually check/adapt the submodule checkouts for consistency

The release.sh update [submodules...] command is just a helper. I have in the past done git submodule foreach git pull origin master and then modified manually what should be committed. The advantage of release.sh update is that it will correctly handle ocrd_cis being developed in a branch named dev, not master. What is actually committed is up to the person doing the PR.

You mean after committing, right? I ask because our Makefile rules will immediately reset the worktree to HEAD.

Yes, after comimitting.

* Open the PR and wait for CI

…after pushing, of course.

Yes (I'll update the initial post shortly).

The following parts can only be done by collaborators, of course:

* **Do not merge on GitHub**, but locally: `git checkout master; git merge update-....`

you mean git merge --no-commit pr-branch, right? (This is crucial, because with autocommit the subsequent steps won't see any status changes in the submodules.)

You're right.

* Generate the CHANGELOG.md: `./release.sh changelog`

…and afterwards validate that this looks correct by git diff CHANGELOG.md HEAD, right?

Yes, also probably delete merge commits to keep it concise.

* Release to GitHub: `./release.sh release-github` (which will commit the changelog with commit message `:package: $version`, tag that commit as `$version`, push branch and tag to GitHub). This will trigger the Circle CI deploy script which will, after building/pushing all the min/med/max/git/non-git images, tag the `ocrd/all:maximum` image as `ocrd:all:$version` and push to Dockerhub).

Ok, so in summary:

• What will change for contributors?
  Nothing. (This is good!)
• What will change for collaborators who want to merge PRs?
  They will have to apply the PR locally through release.sh, using their clone's working directory a lot (so they must be careful about local.mk or shell variables deactivating components) and downloading all submodules. This is not a big change for their own PRs, but a little extra effort for foreign PRs.

Yes, this requires some work for maintainers and is not integrated into GitHub. I considered using GitHub actions / GitHub API for this but that would require much more work than a shell script and clear conventions. Not saying that it might not be worth the effort in the future but let's see how well the local-shell-script-approach works first.

[kba]

It will trigger a Docker alias for the current date, though.

We could adapt the Circle CI script to only run if Makefile or submodules have changed. But I doubt it's worth the possible side-effects over manually stopping unnecessary Circle CI builds

[bertsky]

one can still use the same procedure as before (esp. for small PRs which only change the README or ocrd_all Makefile) – this won't trigger a git tag or GH release or CHANGELOG update. It will trigger a Docker alias for the current date, though.

We could adapt the Circle CI script to only run if Makefile or submodules have changed. But I doubt it's worth the possible side-effects over manually stopping unnecessary Circle CI builds

I meant this as a mere observation. In fact, I think this (also) is a good thing: Docker alias and Dockerhub push can be repeated as often as necessary. For example, one might need to merge multiple PRs on one day, or amend a wrong previous version

[bertsky]

Could you copy these guidelines somewhere permanent, maybe in a PR template?

[kba]

I added contributing guidelines. GitHub will show a sidebar notifying users of them if they open PR/Issues, also linked in the README.

[bertsky]

This all has turned out very well – is there anything more to do before merging? @kba are you waiting on more reviews?

[kba]

are you waiting on more reviews?

I requested one @stweil to make sure he's on board with the PR, maybe he missed it. If I don't hear anything to the contrary, I will merge it later today. I really want to try out, if all of this actually works :)