Release flow #1019

savente93 · 2024-07-03T13:15:07Z

Issue addressed

Fixes #972

Explanation

I originally phrased this issue as "migrate release flow to pixi" because I was under the impression that pixi had a publish command. it does have a upload command for conda, which I still want to investigate. Perhaps this can be a more efficient way of dealing with the feedstock but given that this still needs to be investigated I decided to leave it out of scope for this one. Currently the release flow happens entirely within github actions (which has some limitations that I'll get to in a second) and I could move these to the pixi tasks, depending on what people prefer. The upside to doing it within github actions is that it generalises easier to projects which do now use pixi yet, but whether that trade off is worth it I'll leave up to the reviewer.

With the changes in this PR the release flow will go as follows:

Use the prepare release workflow on github. This will:
a. Create a release branch with the correct new version number
b. update the hydromt/__init__.py and docs/changelog.rst with the new version and headers as necessary
c. Create a PR with these changes
On every commit to the Release PR it will create test and upload
a. docs
b. docker image (incl one for binder integration)
c. pypi package
(d. conda package is excluded here because that has to happen externally through the feedstock)
Due to the fact that workflows cannot run other workflows (enforced by github at the platform level) a (possibly) empty commit will be necessary to start the artifact creation the first time. The github bot will post a comment explaining this and how to do it. Do not worry about the empty commit, the PR will be squash merged like any other at the end.
Each of the release artifacts created can be downloaded and inspected manually if desired. The github bot will post links to the artifacts in a comment if they are successfully created
Once everything looks okay, someone should approve the PR. Once this is done, a workflow will be started that will merge the PR, download all the previously created release artifacts and publish them (instead of regenerating them so you know the ones you inspected are the ones being released), in addition to creating a github release. Once all this is done it will create a commit on main, adding a new unreleased header in docs/chagelog.rst and setting the version in hydromt/__init__.py back to a dev version.
your release is now complete.

Hope this was a good enough explanation, I know it's a bit hairy on the back end, but it makes the process of actually creating a release much more automated and inspect able I think. It's a very "set it and forget it" setup which I like. Let me know if any additional clarification is required

General Checklist

savente93 · 2024-07-09T09:32:56Z

once this has had an initial review and people are happy with the approach I'll also update the docs with this explanation, but I wanted to get some feedback on the idea first.

savente93 · 2024-07-09T09:37:33Z

Oh, also, by it's very nature, this is pr is very hard to test, but I have developed this using this repo: https://github.com/savente93/ci-playground There things can be inspected and tested as much as I could figure out, so if reviewer wants to test/play with it first feel free to do it there.

Jaapel · 2024-07-09T13:36:44Z

Seems like you can trigger workflows from workflows? Max 3 it seems in the notes. https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_run

.github/workflows/prep-release.yml

Jaapel · 2024-07-09T13:41:30Z

I also do not understand your note about publishing to docker and pypi on every push to a release branch. Pypi at least does not support removing artifacts, I can't remember if testpypi does. But there should be some cleanup if you push a new version of the artifact.
If pixi can easily fix pushing artifacts to conda, I would happily support that. If not, then it is fine to include it in GitHub. Is there a GitHub action for it? The argument of sharing this with people is stronger if we create our own GH Action for it, as other developers will for sure want this as well.

savente93 · 2024-07-12T09:49:52Z

I think pixi might be able to also take on the release to conda which would mean we don't have to wait for the feedstock to wake up which would be nice, but this probably requires more research and development so I'm leaving this off for now. I've tested it as well as possible, but given that we can only really test it by actually releasing, it will have to be made more robust over time. I did add as much detail as I thought was appropriate to the docs, and the workflows are (hopefully) fairly well documented, so I think it's good enough for now.

Jaapel

Most of it looks good, only the branch matching comment is critical

Jaapel · 2024-07-15T07:37:11Z

.github/workflows/finish-release.yml

+on:
+  pull_request:
+    types:
+    - closed


Should this not also include some filter on the release branch naming convention? Now this will also trigger on any PR being closed if I understand correctly.

oh that's a very good catch! (that's the downside of working in a simplified test environment I guess 😅). Will fix

I've fixed this now, though thinking about it, if this ran on a random PR, there wouldn't be any artifacts for it to download so it wouldn't have succeeded anyway. I guess that's one upside to doing it the way I have

Jaapel · 2024-07-15T07:39:06Z

.github/workflows/finish-release.yml

+            git tag "v$RELEASE_VERSION"
+            # post release stuff so we don't forget
+            sed -i "s/v$RELEASE_VERSION/Unreleased\n==========\n\nNew\n---\n\nChanged\n-------\n\nFixed\n-----\n\nDeprecated\n----------\n\nv$RELEASE_VERSION/" docs/changelog.rst
+            sed -i 's/__version__.*=.*"\(.*\)".*/__version__ = "\1.dev0"/' hydromt/__init__.py


__version__ seems hardcoded here?

I get why you think that but it's not. In sed \1 is the code for "first capture group" so this will get substituted with what was captured in the (.*) earlier in the command

.github/workflows/prep-release.yml

docs/dev/contributing.rst

Jaapel · 2024-07-15T07:45:04Z

docs/dev/contributing.rst

+5. After the PR is merged, a action should start (though it will not show up under the PR itself) that will publish the latest artifacts created to their respective platform. After this, a bot will add a final commit to the `main` branch, setting the hydromt version back to a dev version, and adding new headers to the `docs/changelog.rst` for unreleased features. It will also create a tag and a github release for you automatically. The release is now done as far as this repo is concerned.
+6. The newly published PyPi package will trigger a new PR to the `HydroMT feedstock repos of conda-forge <https://github.com/conda-forge/hydromt-feedstock>`_.
+   Here you can use the comment posted to the release PR to see if the `meta.yml` needs to be updated. Merge the PR to release the new version on conda-forge.
+7. celebrate the new release!


Still looking forward to this :)

celebrating the new release?

Co-authored-by: Jaap Langemeijer <33715902+Jaapel@users.noreply.github.com>

Jaapel

👍

savente93 added 9 commits June 21, 2024 16:20

wip

023d794

wip

8894859

Update finish-release.yml

d4ed11e

Update create-release-artifacts.yml

ce106ee

Update prep-release.yml

7b1df34

Delete stray finish-release.yml

83214cf

Delete stray prep-release.yml

1a8809c

Merge branch 'main' into release-flow

df1ae4d

lint

d310751

savente93 requested a review from Jaapel July 9, 2024 09:33

squash merge release at end

f85da5b

Jaapel reviewed Jul 9, 2024

View reviewed changes

.github/workflows/prep-release.yml Outdated Show resolved Hide resolved

savente93 added 4 commits July 11, 2024 15:22

feat: import work from ci-playground

66f5bc2

chore: update docs and action versions

e36aec3

fix: check docker creds during testing

457e9ca

chore: update changelog

a6158ca

savente93 marked this pull request as ready for review July 12, 2024 09:48

Jaapel suggested changes Jul 15, 2024

View reviewed changes

savente93 and others added 2 commits July 15, 2024 09:49

Apply suggestions from code review

f41aec5

Co-authored-by: Jaap Langemeijer <33715902+Jaapel@users.noreply.github.com>

fix: only create release when release pr is merged

e536a2f

savente93 requested a review from Jaapel July 15, 2024 08:06

Jaapel approved these changes Jul 15, 2024

View reviewed changes

savente93 merged commit 4a3d6a5 into main Jul 15, 2024
9 checks passed

savente93 deleted the release-flow branch July 15, 2024 09:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release flow #1019

Release flow #1019

savente93 commented Jul 3, 2024 •

edited

Loading

savente93 commented Jul 9, 2024

savente93 commented Jul 9, 2024

Jaapel commented Jul 9, 2024

Jaapel commented Jul 9, 2024

savente93 commented Jul 12, 2024

Jaapel left a comment

Jaapel Jul 15, 2024

savente93 Jul 15, 2024

savente93 Jul 15, 2024

Jaapel Jul 15, 2024

savente93 Jul 15, 2024 •

edited

Loading

Jaapel Jul 15, 2024

savente93 Jul 15, 2024

Jaapel left a comment

Release flow #1019

Release flow #1019

Conversation

savente93 commented Jul 3, 2024 • edited Loading

Issue addressed

Explanation

General Checklist

savente93 commented Jul 9, 2024

savente93 commented Jul 9, 2024

Jaapel commented Jul 9, 2024

Jaapel commented Jul 9, 2024

savente93 commented Jul 12, 2024

Jaapel left a comment

Choose a reason for hiding this comment

Jaapel Jul 15, 2024

Choose a reason for hiding this comment

savente93 Jul 15, 2024

Choose a reason for hiding this comment

savente93 Jul 15, 2024

Choose a reason for hiding this comment

Jaapel Jul 15, 2024

Choose a reason for hiding this comment

savente93 Jul 15, 2024 • edited Loading

Choose a reason for hiding this comment

Jaapel Jul 15, 2024

Choose a reason for hiding this comment

savente93 Jul 15, 2024

Choose a reason for hiding this comment

Jaapel left a comment

Choose a reason for hiding this comment

savente93 commented Jul 3, 2024 •

edited

Loading

savente93 Jul 15, 2024 •

edited

Loading