feat: improve_release_notes

[fabergat]

Description

This PR improves the release note generation, creating a draft release following a predefined template.
With this new workflow, the draft release is prepared after the image generation and executed only when a tag is published.

Closes: #1348

Changes

• Add release draft generation
• Change the way the docker hub digest is added

Testing Information

You can check the result here: https://github.com/stacks-network/sbtc/releases/tag/untagged-bdc9ed203ee2efb66a2f

Checklist:

• I have performed a self-review of my code
• My changes generate no new warnings
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[fdefelici]

@fabergat
I would like to better understand the reason behind splitting the release build into 2 workflows (release_draft_action.ym on the tag event and release_docker_hub.yaml on the release publish event).

Apart from the need to build the docker link, are there other needs that led to this decision?

[fabergat]

@fabergat I would like to better understand the reason behind splitting the release build into 2 workflows (release_draft_action.ym on the tag event and release_docker_hub.yaml on the release publish event).

Apart from the need to build the docker link, are there other needs that led to this decision?

This is an interesting question that it is worth a discussion.
So the docker hub flow is highly dependent on two things:

1. A new image is published on docker hub (after a new tag release and the approval process)
2. A new release associated with that tag.

So technically if we run the docker hub flow as parallel it will probably expire (if not approved) or fail because data is not anyway ready. We also have to consider the fact that the drafter is publishing a draft and not the official release and more time will pass but I am open to evaluating this possibility. If it is a bit confusing we can have a chat offiline

[fdefelici]

@fabergat I would like to better understand the reason behind splitting the release build into 2 workflows (release_draft_action.ym on the tag event and release_docker_hub.yaml on the release publish event).
Apart from the need to build the docker link, are there other needs that led to this decision?

This is an interesting question that it is worth a discussion. So the docker hub flow is highly dependent on two things:

1. A new image is published on docker hub (after a new tag release and the approval process)
2. A new release associated with that tag.

So technically if we run the docker hub flow as parallel it will probably expire (if not approved) or fail because data is not anyway ready. We also have to consider the fact that the drafter is publishing a draft and not the official release and more time will pass but I am open to evaluating this possibility. If it is a bit confusing we can have a chat offiline

If it is just that, I was thinking to have just 1 workflow with 2+ jobs (to be run sequentially) to be triggered on tag or at release publish. Basically the Jobs could "prepare" the release text in an incremental manner starting from a template file (basically reorganizing the what have been done in the 2 workflows), and publishing the release text at the very end.

This avoid to hook to two different github events (tag and release published) and eventually keep one of them "free" for eventual future use case.

[fabergat]

@fabergat I would like to better understand the reason behind splitting the release build into 2 workflows (release_draft_action.ym on the tag event and release_docker_hub.yaml on the release publish event).
Apart from the need to build the docker link, are there other needs that led to this decision?

This is an interesting question that it is worth a discussion. So the docker hub flow is highly dependent on two things:

1. A new image is published on docker hub (after a new tag release and the approval process)
2. A new release associated with that tag.

So technically if we run the docker hub flow as parallel it will probably expire (if not approved) or fail because data is not anyway ready. We also have to consider the fact that the drafter is publishing a draft and not the official release and more time will pass but I am open to evaluating this possibility. If it is a bit confusing we can have a chat offiline

If it is just that, I was thinking to have just 1 workflow with 2+ jobs (to be run sequentially) to be triggered on tag or at release publish. Basically the Jobs could "prepare" the release text in an incremental manner starting from a template file (basically reorganizing the what have been done in the 2 workflows), and publishing the release text at the very end.

This avoid to hook to two different github events (tag and release published) and eventually keep one of them "free" for eventual future use case.

in the middle, there is the docker build that is triggered after a tag creation and nightly, this task also takes a bit of time, around 30 minutes... maybe we can trigger the release update after the docker build and check if a release with that name exists but in case we want to create the release after we lost the action

[fdefelici]

@fabergat I would like to better understand the reason behind splitting the release build into 2 workflows (release_draft_action.ym on the tag event and release_docker_hub.yaml on the release publish event).
Apart from the need to build the docker link, are there other needs that led to this decision?

This is an interesting question that it is worth a discussion. So the docker hub flow is highly dependent on two things:

1. A new image is published on docker hub (after a new tag release and the approval process)
2. A new release associated with that tag.

So technically if we run the docker hub flow as parallel it will probably expire (if not approved) or fail because data is not anyway ready. We also have to consider the fact that the drafter is publishing a draft and not the official release and more time will pass but I am open to evaluating this possibility. If it is a bit confusing we can have a chat offiline

If it is just that, I was thinking to have just 1 workflow with 2+ jobs (to be run sequentially) to be triggered on tag or at release publish. Basically the Jobs could "prepare" the release text in an incremental manner starting from a template file (basically reorganizing the what have been done in the 2 workflows), and publishing the release text at the very end.
This avoid to hook to two different github events (tag and release published) and eventually keep one of them "free" for eventual future use case.

in the middle, there is the docker build that is triggered after a tag creation and nightly, this task also takes a bit of time, around 30 minutes... maybe we can trigger the release update after the docker build and check if a release with that name exists but in case we want to create the release after we lost the action

Do you refer to image-build.yaml? Doesn't it just run on tags?

name: Docker Image (Binary)

on:
  workflow_dispatch:
  # schedule:
  #   - cron: '0 2 * * *'
  push:
    tags:
      - '*'

If I'm not wrong, even in the current scenario you need to wait for the image-build to finish before safetly running the release publishing (release_docker_hub.yaml).

If it is so, we could make a release workflow that does sequentially the relevant steps:

1. build image with tag
2. create the release message at once

Possibly we could even use docker inspect --format="{{index .RepoDigests 0}}" .... after image push to get the image SHA instead of quering dockerhub afterwards (in the same workflow we could easily share this info)

Even if we still want to not wait for image building before publishing the release, this process could still work adding a step 3 for updating the release message (or maybe just inverting step 1 with step 2)

Eventually, the build image job can be reused in different workflow to implement a proper nightly build for example.

I'm just wondering if we could make the whole process more linear and possibly without actively waiting for completing the release, but I'm opened if we want to stick with the current solution. In the end, it does the job.

[fabergat]

@fabergat I would like to better understand the reason behind splitting the release build into 2 workflows (release_draft_action.ym on the tag event and release_docker_hub.yaml on the release publish event).
Apart from the need to build the docker link, are there other needs that led to this decision?

This is an interesting question that it is worth a discussion. So the docker hub flow is highly dependent on two things:

1. A new image is published on docker hub (after a new tag release and the approval process)
2. A new release associated with that tag.

So technically if we run the docker hub flow as parallel it will probably expire (if not approved) or fail because data is not anyway ready. We also have to consider the fact that the drafter is publishing a draft and not the official release and more time will pass but I am open to evaluating this possibility. If it is a bit confusing we can have a chat offiline

If it is just that, I was thinking to have just 1 workflow with 2+ jobs (to be run sequentially) to be triggered on tag or at release publish. Basically the Jobs could "prepare" the release text in an incremental manner starting from a template file (basically reorganizing the what have been done in the 2 workflows), and publishing the release text at the very end.
This avoid to hook to two different github events (tag and release published) and eventually keep one of them "free" for eventual future use case.

in the middle, there is the docker build that is triggered after a tag creation and nightly, this task also takes a bit of time, around 30 minutes... maybe we can trigger the release update after the docker build and check if a release with that name exists but in case we want to create the release after we lost the action

Do you refer to image-build.yaml? Doesn't it just run on tags?

name: Docker Image (Binary)

on:
  workflow_dispatch:
  # schedule:
  #   - cron: '0 2 * * *'
  push:
    tags:
      - '*'

If I'm not wrong, even in the current scenario you need to wait for the image-build to finish before safetly running the release publishing (release_docker_hub.yaml).

If it is so, we could make a release workflow that does sequentially the relevant steps:

1. build image with tag
2. create the release message at once

Possibly we could even use docker inspect --format="{{index .RepoDigests 0}}" .... after image push to get the image SHA instead of quering dockerhub afterwards (in the same workflow we could easily share this info)

Even if we still want to not wait for image building before publishing the release, this process could still work adding a step 3 for updating the release message (or maybe just inverting step 1 with step 2)

Eventually, the build image job can be reused in different workflow to implement a proper nightly build for example.

I'm just wondering if we could make the whole process more linear and possibly without actively waiting for completing the release, but I'm opened if we want to stick with the current solution. In the end, it does the job.

The current scenario pushes a draft release that is updated when you run the docker build, for the docker inspect is the same thing if you query it from the docker hub. You still have to wait, before we didn't want to push a new release for every new build so it was attached only to the release process but now I think it makes more sense to have it after the docker build but I have to check if this a release from the tag or the nightly.

[fdefelici]

@fabergat I would like to better understand the reason behind splitting the release build into 2 workflows (release_draft_action.ym on the tag event and release_docker_hub.yaml on the release publish event).
Apart from the need to build the docker link, are there other needs that led to this decision?

This is an interesting question that it is worth a discussion. So the docker hub flow is highly dependent on two things:

1. A new image is published on docker hub (after a new tag release and the approval process)
2. A new release associated with that tag.

So technically if we run the docker hub flow as parallel it will probably expire (if not approved) or fail because data is not anyway ready. We also have to consider the fact that the drafter is publishing a draft and not the official release and more time will pass but I am open to evaluating this possibility. If it is a bit confusing we can have a chat offiline

If it is just that, I was thinking to have just 1 workflow with 2+ jobs (to be run sequentially) to be triggered on tag or at release publish. Basically the Jobs could "prepare" the release text in an incremental manner starting from a template file (basically reorganizing the what have been done in the 2 workflows), and publishing the release text at the very end.
This avoid to hook to two different github events (tag and release published) and eventually keep one of them "free" for eventual future use case.

in the middle, there is the docker build that is triggered after a tag creation and nightly, this task also takes a bit of time, around 30 minutes... maybe we can trigger the release update after the docker build and check if a release with that name exists but in case we want to create the release after we lost the action

Do you refer to image-build.yaml? Doesn't it just run on tags?

name: Docker Image (Binary)

on:
  workflow_dispatch:
  # schedule:
  #   - cron: '0 2 * * *'
  push:
    tags:
      - '*'

If I'm not wrong, even in the current scenario you need to wait for the image-build to finish before safetly running the release publishing (release_docker_hub.yaml).
If it is so, we could make a release workflow that does sequentially the relevant steps:

1. build image with tag
2. create the release message at once

Possibly we could even use docker inspect --format="{{index .RepoDigests 0}}" .... after image push to get the image SHA instead of quering dockerhub afterwards (in the same workflow we could easily share this info)
Even if we still want to not wait for image building before publishing the release, this process could still work adding a step 3 for updating the release message (or maybe just inverting step 1 with step 2)
Eventually, the build image job can be reused in different workflow to implement a proper nightly build for example.
I'm just wondering if we could make the whole process more linear and possibly without actively waiting for completing the release, but I'm opened if we want to stick with the current solution. In the end, it does the job.

The current scenario pushes a draft release that is updated when you run the docker build, for the docker inspect is the same thing if you query it from the docker hub. You still have to wait, before we didn't want to push a new release for every new build so it was attached only to the release process but now I think it makes more sense to have it after the docker build but I have to check if this a release from the tag or the nightly.

Yes, probably there will be different workflows: one more CI oriented (that can be launched on commit, nightly or whatever) and one instead dedicated to the release process (launched on tag or eventually on release published).

By the fact that anyhow for the release process have to "wait" (for the image_build), this is the reason I think we could make the release process "synchronous" as described above (this can avoid that the approvers must be aware if the image build workflow has been completed succesfully)

[fabergat]

@fabergat I would like to better understand the reason behind splitting the release build into 2 workflows (release_draft_action.ym on the tag event and release_docker_hub.yaml on the release publish event).
Apart from the need to build the docker link, are there other needs that led to this decision?

This is an interesting question that it is worth a discussion. So the docker hub flow is highly dependent on two things:

1. A new image is published on docker hub (after a new tag release and the approval process)
2. A new release associated with that tag.

So technically if we run the docker hub flow as parallel it will probably expire (if not approved) or fail because data is not anyway ready. We also have to consider the fact that the drafter is publishing a draft and not the official release and more time will pass but I am open to evaluating this possibility. If it is a bit confusing we can have a chat offiline

If it is just that, I was thinking to have just 1 workflow with 2+ jobs (to be run sequentially) to be triggered on tag or at release publish. Basically the Jobs could "prepare" the release text in an incremental manner starting from a template file (basically reorganizing the what have been done in the 2 workflows), and publishing the release text at the very end.
This avoid to hook to two different github events (tag and release published) and eventually keep one of them "free" for eventual future use case.

in the middle, there is the docker build that is triggered after a tag creation and nightly, this task also takes a bit of time, around 30 minutes... maybe we can trigger the release update after the docker build and check if a release with that name exists but in case we want to create the release after we lost the action

Do you refer to image-build.yaml? Doesn't it just run on tags?

name: Docker Image (Binary)

on:
  workflow_dispatch:
  # schedule:
  #   - cron: '0 2 * * *'
  push:
    tags:
      - '*'

If I'm not wrong, even in the current scenario you need to wait for the image-build to finish before safetly running the release publishing (release_docker_hub.yaml).
If it is so, we could make a release workflow that does sequentially the relevant steps:

1. build image with tag
2. create the release message at once

Possibly we could even use docker inspect --format="{{index .RepoDigests 0}}" .... after image push to get the image SHA instead of quering dockerhub afterwards (in the same workflow we could easily share this info)
Even if we still want to not wait for image building before publishing the release, this process could still work adding a step 3 for updating the release message (or maybe just inverting step 1 with step 2)
Eventually, the build image job can be reused in different workflow to implement a proper nightly build for example.
I'm just wondering if we could make the whole process more linear and possibly without actively waiting for completing the release, but I'm opened if we want to stick with the current solution. In the end, it does the job.

The current scenario pushes a draft release that is updated when you run the docker build, for the docker inspect is the same thing if you query it from the docker hub. You still have to wait, before we didn't want to push a new release for every new build so it was attached only to the release process but now I think it makes more sense to have it after the docker build but I have to check if this a release from the tag or the nightly.

Yes, probably there will be different workflows: one more CI oriented (that can be launched on commit, nightly or whatever) and one instead dedicated to the release process (launched on tag or eventually on release published).

By the fact that anyhow for the release process have to "wait" (for the image_build), this is the reason I think we could make the release process "synchronous" as described above (this can avoid that the approvers must be aware if the image build workflow has been completed succesfully)

yeah working on it!

[fdefelici]

Added a couple of small remarks.

Now, I think. we reached the objective to avoid manual intervention of approvers to complete the release process (more than anything prevent them for actively waiting for image build to be completed).

Strong advice for future improvements (eventually to be addressed in specific issue/PR) is to have different worklows for CI things (like nightly) and for release. Usually operations done in this kind of workflows can diverge quickly and so the risk of having just one workflow to do it all, is to fullfill it with conditional statements to skip/do specific action based on the context.

[fabergat]

Added a couple of small remarks.

Now, I think. we reached the objective to avoid manual intervention of approvers to complete the release process (more than anything prevent them for actively waiting for image build to be completed).

Strong advice for future improvements (eventually to be addressed in specific issue/PR) is to have different worklows for CI things (like nightly) and for release. Usually operations done in this kind of workflows can diverge quickly and so the risk of having just one workflow to do it all, is to fullfill it with conditional statements to skip/do specific action based on the context.

There is this issue that will move the night job to ECS, we can split the workflow there: #1305