Continuous Delivery for H9

[f1337]

Specifically: an automated process which triggers the following when it detects a new tag. Given a new tag of 1.1.1:

• update the version in package.json to 1.1.1
• publish to npm: npm publish
• update the Dockerfile with the latest npm version (keep it explicit): npm install -g haiku9@1.1.1
• rebuild the Docker latest image: docker build . -t pandastrike/haiku9
• tag the Docker image version: docker tag pandastrike/haiku9 pandastrike/haiku9:1.1.1
• publish to Docker hub: docker push pandastrike/haiku9

[f1337]

We could watch the package.json version instead of a separate VERSION file, if that's easier. But it seems to me watching a file with a single value is less prone to error than watching a file with many keys and values.

[dyoder]

Why watch a file at all? Why not just have a release branch?

[f1337]

We can have a release branch, but putting on my CD hat, master is the release branch.

Why watch a file? Because our npm and Docker Hub deployments are versioned. And a human being makes the semver decision re: "backwards-compatible change". So the simplest pattern I could think of is have a human being update the VERSION file, and if changed, deploy. This logic applies to master or release equally.

[f1337]

…more re: watching a VERSION file:

Basically we need some way to track state of "what version is deployed to npm and docker hub?" versus "what is the latest version?", in order to prevent our CD process from overwriting pandastrike/haiku9:1.0.0-beta-21 with changes that should be in pandastrike/haiku9:1.0.0-beta-22, b/c someone merged to the release/master branch without incrementing the version strings in package.json and Dockerfile. It's a minor annoyance when dealing with patch-level changes, but with breaking changes it could get quite irritating.

I foresee this happening often enough to be frustrating, given that the panda-sky version currently deployed is one patch version ahead of master bc someone forgot to check in the updated package.json. And I've run into similar issues with fairmont and npm.

[f1337]

Stealing a pattern from Disney: replaced the notion of watching a VERSION file with using a tag as the trigger, in the description above. Should be cleaner.

[automatthew]

What is the workflow when using a Git tag as a trigger? Won't that cause us to have tags that don't match the version in package.json?

[automatthew]

• developer gets all code ready for a release
• dev commits and pushes
• dev does git tag -am 1.1.1 1.1.1
• git push --tags
• CD notices a new tag (any new tag, or just those conforming to semver?)
• CD updates package.json with the new version

Now we have a tagged commit in the repo where the version in package.json is behind.

[dyoder]

Yup. All these still have the same basic problem where everything gets out of sync. As @f1337 it's terribly easy to do and it happens in practice all the time.

I tried to work around this by having an npm post-publish hook that updates the version and tags it at the same time. In fact, this is already implemented in H9:

https://github.com/pandastrike/haiku9/blob/master/package.json#L46

My hope was to build on that for CI. That is, we'd add the Docker build and publish step as part of the post-publish hook. But this approach has an annoying drawback in that it isn't tied to a commit.

Which brings me back to my original question, which I dashed off thinking that everyone already knew about the post-publish hook in NPM. I really like the idea of using git hooks because it ties to the commit to all the CI/CD activity directly.

Instead of a post-publish hook, we use a git hook. The git hook runs the tests and, presuming they pass, tags the release, pushes the tags, does the NPM publish, and builds and pushes the Docker image. Possibly not in that order, since if the NPM publish or Docker build/publish fail, we might want to avoid tagging the release. OTOH, there's no way to make them atomic operations (I don't think, although both NPM and Docker are getting more sophisticated by the day…) so in those cases, maybe we just log the errors.

The reason I said release branch wasn't because we need a literal release branch (although I've preferred to use an explicit release branch so that every merge into mainline isn't a release, since that introduces a blocking step into the workflow, but that's a different conversation) but that we can just use git hooks on whatever we consider to be the release branch, without watching a file.

[dyoder]

Yup. All these still have the same basic problem where everything gets out of sync. As @f1337 it's terribly easy to do and it happens in practice all the time.

I tried to work around this by having an npm post-publish hook that updates the version and tags it at the same time. In fact, this is already implemented in H9:

https://github.com/pandastrike/haiku9/blob/master/package.json#L46

My hope was to build on that for CI. That is, we'd add the Docker build and publish step as part of the post-publish hook. But this approach has an annoying drawback in that it isn't tied to a commit.

Which brings me back to my original question, which I dashed off thinking that everyone already knew about the post-publish hook in NPM. I really like the idea of using git hooks because it ties to the commit to all the CI/CD activity directly.

Instead of a post-publish hook, we use a git hook. The git hook runs the tests and, presuming they pass, tags the release, pushes the tags, does the NPM publish, and builds and pushes the Docker image. Possibly not in that order, since if the NPM publish or Docker build/publish fail, we might want to avoid tagging the release. OTOH, there's no way to make them atomic operations (I don't think, although both NPM and Docker are getting more sophisticated by the day…) so in those cases, maybe we just log the errors.

The reason I said release branch wasn't because we need a literal release branch (although I've preferred to use an explicit release branch so that every merge into mainline isn't a release, since that introduces a blocking step into the workflow, but that's a different conversation) but that we can just use git hooks on whatever we consider to be the release branch, without watching a file.

[dyoder]

thinking that everyone already knew about the post-publish hook in NPM

Meaning the specific hooks in h9 and in a few other repos.