Define naming of docker images for packages + registry

[ruflin]

The Elastic Package Registry (epr.elastic.co) run a docker container that contains the packages + the registry service itself. Today the packages are in the same repository as the registry but this is changing. Packages will be moving to https://github.com/elastic/package-storage Also there is work ongoing to have a Dockerfile available that is independent of both repository.

This means the Docker image will contain data from two different sources. For debugging and support purpose I think it is important that we can identify what exactly went into this Docker image to potentially reproduce issues.

My proposal related to naming each Docker image is as following:

{date}-{package-storage-hash}-{package-registry-hash}

hash in the above can be a commit hash but also a branch or a tag. The date part is for human and sorting purpose to identify when exactly an image was built. Two examples of the above would look as following:

• 202004231234-c890763-c1d2c2c
• 202004231234-c890763-v0.3.0

The package hash will change much more often than the registry one but it also the more important one for debugging purpose. That is why I think it should come first.

To build this image, both the package-storage hash and package-registry hash are passed to the Dockerfile and then used to check out the correct version (https://github.com/elastic/package-registry/blob/master/dev/release/Dockerfile#L19).

Questions

• Is the above possible to build / automate? Does it make sense?
• What is the date/time format we should use?

[ruflin]

@kuisathaverat Would be great to get your feedback / input on the above.

[kuisathaverat]

• Is the above possible to build / automate? Does it make sense?

It depends, 😄 What are the use cases for that tag? I think on the following

• Someone wants to get the latest version
• Someone wants to get a specific version of the package-registry
• Someone wants to get a specific version of the package-registry and specific version of the package-storage

There are some questions about what we will release, or how we will release:

• Would we release several versions of the same package-registry on the same day? probably yes, but would change the package-storage?
• Would we release old versions of the package-registry with new versions of the package-storage?
• What is more important the package-registry version or the package-storage version?

we can have several tags for the same version depending on the use, a date based tag is good for human usage, but again it depends on how we will release the versions it would be the latest version or something else. A tag {package-storage-hash}-{package-registry-hash} or {package-registry-hash}-{package-storag-hash} is really good for developers we should release one of those too.

• What is the date/time format we should use?

no doubt on that ISO-8601, it would be something like 20200423T121930Z

[ruflin]

• Several releases of the package registry in the same day? Probably not. I expect the package registry to change very rarely and if there is a change, it is not urgent.
• Old versions of the package registry with new version of the package storage: No
• Package storage version is more important from my perspective as it changes much more often.

Use cases for the tags are mainly around debugging: A user reports a problem and we want to reproduce the exact same setup locally. For this we need to be able to look at the current running contains and see what exact version is running.

In the future there will also be a part around someone wants to deploy his own registry. But for this we could introduce a tag aligned with the Elastic Stack version. In the background, it would also have a tag like discussed before. Here probably also the "latest" comes into play.

All the above also reminds me it would be nice to have some logs on "when" we rolled out a new version and which one. I assume we get this from the kubernetes logs?

[kuisathaverat]

Old versions of the package registry with new version of the package storage: No
Package storage version is more important from my perspective as it changes much more often.

This make the {package-registry-hash} redundant, a {package-storage-hash} would be only available for a specific {package-registry-hash}.

Several releases of the package registry in the same day? Probably not. I expect the package registry to change very rarely and if there is a change, it is not urgent.

This makes the timestamp a unique tag for each version.

All the above also reminds me it would be nice to have some logs on "when" we rolled out a new version and which one. I assume we get this from the kubernetes logs?

When you create/restart a pod it stores the timestamp in the metadata.

[ruflin]

I think I need to clarify my comment related to the registry and storage. If a new registry is released an no new packages come it, we could still do a release with the same storage of packages as a previous version of the registry. This only applies to the current one, but not any older.

Agree that the timestamp is unique. I think the part I miss is how will a user tell based on the timestamp which version of the registry and package is packaged inside without having to either go into the container or check git logs on what exactly was there at the moment it was built?

[kuisathaverat]

I understand that we will not release more than a version per day, the time stamp has the date and time of the release, so identified a release minute. We do not release older package-registry versions with new package-storage versions. Let's put an example.

• The package registry is in the version 1.0.0 (sha1 AAAAAAA)

• The storage registry is in the version 1.0.0 (sha1 BBBBBBB)

• Today is 202004280000

• we release a version of the release-package+release-storage at 2020-04-20 17:10

• Proposed tags will be 202004281710-BBBBBBB-AAAAAAA and 202004281710-BBBBBBB-v1.0.0

• We detect some mistake in the storage-registry and we have to release a new version 1.0.1 (sha1 CCCCCCC) in the same day at 17:35 (shit happens), it will have the tags 202004281735-CCCCCCC-AAAAAAA and 202004281735-CCCCCCC-v1.0.0

• The next day we release a new version of the package registry 1.0.1 (sha1 DDDDDDD) at 10:12, the tags will be 202004291012-CCCCCCC-DDDDDDD and 202004291012-CCCCCCC-v1.0.1

as you can see the timestamp identifies a tuple of package-storage + package-registry

[ruflin]

I synced up with @kuisathaverat over zoom. Following conclusions.

• Releases are triggered by tags on both Github repositories
• The docker image name is 202004281710-BBBBBBB-AAAAAAA
• The hash versions also as labels to the docker image for debugging purpose
• The hash versions should also be to the k8s pods as labels for debugging purpose
• A release must be approved through Jenkins

A release will look as following:

• A CI job is looking for release-production tag changes in package-registry and package-storage
• If one of the tags is updated, the script will figure out the commit hash of each tag and trigger a build
• This build then needs to be approved in Jenkins. E-Mail is sent out to team to be approved.
• As soon as the process is approved, the image 202004281710-BBBBBBB-AAAAAAA is built. The timestamp is the timestamp of when the build was approved.

@kuisathaverat Any opinion on the tag names?

• release:production
• release/production
• release-production

[ruflin]

@kuisathaverat Just realised the above will work any tag / branch we have. To try out the above, could we already build it based on the master branches for the two repository and use this to deploy our staging environment? I already adjusted the Dockerfile: https://github.com/elastic/package-registry/blob/master/dev/release/Dockerfile It will need some cleanup and optimisations but should work for now. I plan to move this Dockerfile to the package-storage as I think it is more fitting there.

[kuisathaverat]

I was thinking in something like this:

• A CI job is looking for tag creation in package-registry and package-storage, the tag are regular tag versions (v1.0.0)
• This build then needs to be approved in Jenkins. E-Mail is sent out to team to be approved.
• As soon as the process is approved, the image 202004281710-BBBBBBB-AAAAAAA is built. The timestamp is the timestamp of when the build was approved.
• This new release is deployed in the staging environment
• when you are confident enough that this versions is OK you promote that version (Docker image) to the production environment.

About to deploy master in some environment, this environment is an "edge" enviroment with latest changes, not sure if the staging is the correct one.

[ruflin]

The package-storage will have so many releases, the regular versioning would not work here. The nice part about using the tag as I proposed above, it can also be used to roll back. Having just 1 tag for each release type also removes logic on the CI to figure out what is newest. It only must bring the two equal tags together.

I like the idea about promoting to prod. How would this work? Also in Jenkins?

For staging: Yes, at the moment it is our testing environment and we will need to rename / a new one in the future.

[kuisathaverat]

The package-storage will have so many releases, the regular versioning would not work here. The nice part about using the tag as I proposed above, it can also be used to roll back. Having just 1 tag for each release type also removes logic on the CI to figure out what is newest. It only must bring the two equal tags together.

My only concert about reusing the same tag it is that you lose the pointer to the commit in the Git repo, even do you have it in the Docker image.

I like the idea about promoting to prod. How would this work? Also in Jenkins?

Promote a docker image to production can be as easy as creating a PR with the changes to the repo where the deploy live, we can make it in a job to grab all the metadata and standardized the PR format. Then you only have to review the PR and merge.

[ruflin]

When would it be a problem that we loose the pointer? My assumption is that debugging will always start with the container running and there we have only the commit ids anyway?

The nice part about the "staging" release above is that we don't need the PR workflow and just an approval. Would be nice if we have something similar for prod. I'm getting back here to my previousl proposal with the tags. If we have a tag release:staging and want to promote it to be prod, we just have to set release:production to the same commit and will get the approval process as we already have for staging.

[kuisathaverat]

When would it be a problem that we lose the pointer? My assumption is that debugging will always start with the container running and there we have only the commit ids anyway?

e.g you release the first version of the package registry 2 weeks ago, at that time you create the tag T that points to the commit C1, since this release you release two more times T -> C2 and T-> C3, so right now, the tag T point to the commit C3. In the case, we need to rebuild the first version, we have to go to the Docker image released on that time and grab the real commit, the same if we want to make a fix-branch for that versions.

The nice part about the "staging" release above is that we don't need the PR workflow and just an approval. Would be nice if we have something similar for prod. I'm getting back here to my previousl proposal with the tags.

Instead of a PR, we can make directly a commit to the repo with the changes (labels, tags, ...)

[ruflin]

I think we can simplify things if we decide there is never an option to roll back. On the package-storage side that is already the case. You can never change a package that was already added, only add a new version. On the registry I assume we would rather release again a newer version which reverts the old commits and is because of this also a new version.

What the above means, going back in history or seeing what is actually running, is only for debugging purpose. Independent of the above, on the registry side we should have normal releases but these are for the collaboration with Kibana which uses the API not for releases.

How would a commit be different from tagging directly a commit?

Will ping you so we can chat live again, will make it easier.

[ruflin]

I had again a conversation with @kuisathaverat . Few conclusions:

• Docker image name is {timestamp-of-approval}-{package-storage-git-hash}-{package-registry-git-hash} as discussed
• Release is triggered by release:production etc. in the two repos
• Approval process in Jenkins is needed

We need to follow up with the Infra team on how this works with their environment.

[ruflin]

I just filed elastic/package-storage#86 which I think has an effect on this discussion. What we still need is all the above discussed values around the commit hashes and time as part of the Dockerfile that is built as a label. But as the deployment of all environments happens fully automated, I think we can get rid of the complex naming. As we have staging and snapshot before the production release, I think we can also get rid of the approval flow.

Now getting into the naming of the Docker images. There is also a need for having a package-registry without any packages. These containers look as following:

docker.elastic.co/package-registry/package-registry:master
docker.elastic.co/package-registry/package-registry:v0.4.0

The ones that contain the registry + package-storage (basically a certain distribution) are called as following:

docker.elastic.co/package-registry/distribution:snapshot
docker.elastic.co/package-registry/distribution:staging
docker.elastic.co/package-registry/distribution:prod

To also have some images which do not constantly change, we could have some distributions tied to a specific release:

docker.elastic.co/package-registry/distrbution:v7.9.0

This would be a tag for the day where v7.9.0 was released so someone could pull this version locally.

If we still need Docker images for each commit hash, we now have a naming scheme with distribution prefix to define if it contains packages or not.

[ruflin]

Closing as with elastic/package-storage#86 we landed on #375 (comment)

@kuisathaverat What we call the internal hash containers I would consider an implementation detail. As the package-storage as a fix registry version in each commit, we only have one hash left + a timestamp which might be informative.