Docker Labels for Latest of Major (or Minor) Versions

[dlvenable]

Is your feature request related to a problem? Please describe

Currently, to pull a Docker image, administrators can either pull the latest or a specific version. In this Issue, I will Data Prepper as an example, but this could be applied for OpenSearch as well.

docker pull opensearchproject/data-prepper:latest

or

docker pull opensearchproject/data-prepper:1.2.1

However, pulling the latest will begin to pull breaking changes after the next major version (2.0) is released. There is no shorthand way to specify pulling the latest version for either a major version or a minor version. So administrators must either specify specific versions or pull latest and risk pulling breaking changes.

Describe the solution you'd like

Provide the following Docker tags on images:

• {major} - Pulls the latest version for a SemVer major version sequence. This tag will change to be on the latest image for any build within the specified {major} version.
• {major}.{minor} - Pull the latest patch release for a SemVer major/minor version. This tag will change to be on the latest image for any build within the specified {major}.{minor} version.

Example Scenario

Data Prepper already has the following tags: 1.0.0, 1.1.0, 1.1.1, 1.2.0, 1.2.1.

In this new scheme, releasing Data Prepper 1.3.0 will create a new image. That image will have the following tags: 1.3.0, 1.3, 1, and latest.

docker pull opensearchproject/data-prepper:1

The command above would pull data-prepper 1.3.0.

After releasing Data Prepper 1.3.1, Data Prepper would have a new image with the 1.3.1 tag. Additionally, we would move the 1.3, 1, and latest tags to the 1.3.1 image.

After releasing Data Prepper 2.0.0, Data Prepper would have a new image with the following new tags: 2.0.0, 2.0, and 2. Additionally, we would move the latest tag to the 2.0.0 image. However, the 1 and 1.3 tags would remain on Data Prepper 1.3.1.

If a security fix requires that we create a Data Prepper 1.3.2 image, we would create a new image with the 1.3.2 tag. Additionally, the 1 and 1.3 tags would move to the 1.3.2 image. The latest remains on the 2.0.0 image.

Releasing Data Prepper 2.1.0 would create a new image with the following new tags: 2.1.0 and 2.1. Additionally, we would move the latest and 2 tags to point to the 2.1.0 image.

Handling Build Numbers

There is a different proposal to include build numbers in image tags in #1461 when updating the base image. My proposal and #1461 are compatible.

Say we release Data Prepper 1.3.0 with the label: 1.3.0-1 where -1 represents the build. (Docker tags do not support +). This image would have the following labels: 1.3.0-1, 1.3.0, 1.3, 1, and latest. If we release a new build with a Docker image, we would tag it 1.3.0-2, 1.3.0, 1.3, 1, and latest.

Describe alternatives you've considered

OpenSearch projects could use a different syntax for the tags. One option is to include a ".x" suffix. It is shown below:

• {major}.x
• {major.minor}.x

This approach can be advantageous since there is no possibility for conflicts with a fully resolved version. However, OpenSearch projects already use branch names of {major}.{minor} and tags of {major}.{minor}.{patch} so this probably doesn't matter.

The proposed solution will keep in line with Git branch names.

One other alternative, is that OpenSearch projects could supply only the major versions. Since OpenSearch projects use semver, the risk when pulling a new minor version is not as great. The major/minor version could be a future add-on if necessary.

Additional context

OpenJDK provides a similar pattern where you can specify specific Java versions. Examples: docker pull openjdk:11, docker pull openjdk:17.
https://hub.docker.com/_/openjdk?tab=description

This proposal has some similarities to #1492. That issue is specifically for archives, but the conventions used should be similar between both proposals.

[bbarani]

We need to think beyond version numbers (especially for Dockers) considering we are updating the base OS images frequently to remediate CVE's without changing the OpenSearch product version number. Also we need to see if we can also solve the build number issues as well #1461

[dlvenable]

we are updating the base OS images frequently to remediate CVE's without changing the OpenSearch product version number.

I take this mean that pulling opensearch:1.2.4 may have a different base image depending on when you actually pull it. That should remain fully compatible with my proposal.

Since major versions will introduce breaking changes, I'm proposing a way to allow users to pull the latest for any given major version. This way, users get CVE fixes for their major version without having to pull in breaking changes from OpenSearch of Data Prepper.

[bbarani]

@dlvenable We might also end up adding an additional identifier to a specific release version number to explicitly let the users know of the updated Docker image (With updated Base OS) Example: opensearch:1.2.4+1 ,opensearch:1.2.4+2 confirming to SemVer versioning guidelines.

[dlvenable]

Thanks for clarifying that. I do believe this could work with my proposal. The opensearch:1.2.4+1 build would be labeled with latest, 1.2.4, and 1. Then if you build opensearch:1.2.4+2, the same labels are applied to the new +2 build: latest, 1.2.4, and 1.

[zelinh]

[Triage] Could you come up with a formal proposal covering all the scenarios? @dlvenable

[dlvenable]

@zelinh , For the most part, I do have the proposal in the issue description. But, I will expand it with additional description on the different labels to apply.

Also, Docker does not permit + in labels. So we will probably need to use another - instead when tagging specific builds.

[dlvenable]

@zelinh and @bbarani , I updated the proposal. I added a more complete example and included a discussion on the build numbers.

[dlvenable]

@bbarani , @zelinh ,

With the upcoming OpenSearch 2.0 release, I believe this proposal will be helpful for users who cannot immediately update to OpenSearch 2.0.

Allowing users to specify the latest OpenSearch 1.x version can be helpful for these users.

docker pull opensearchproject/opensearch:1

[gaiksaya]

[Triage] @stockholmux Any thoughts you would like add on this?

[stockholmux]

Thanks for tagging me in on this @gaiksaya!

As always, the answer is semver.

https://medium.com/@mccode/using-semantic-versioning-for-docker-image-tags-dfde8be06699

Which is remarkably close to what @dlvenable is proposing.

[dlvenable]

Yes, I propose this largely because OpenSearch uses semver. Under semver, we know that a major version change may introduce breaking changes, but should feel confident updating within a major version, and certainly within a major/minor.

Right now, users of OpenSearch and Data Prepper must rely on either latest (which can introduce breaking changes) or an exact version.

[dlvenable]

It could be nice to have this for OpenSearch 1.3.2.

docker pull opensearchproject/opensearch:1

This could pull 1.3.2. If a 1.3.3 comes up later, it would then pull that.

[gaiksaya]

Yes, the proposal looks good and makes a lot of sense. We will look into this soon. Adding it to sprint backlog.
Thanks!

[andrross]

Somewhat related to this issue, but we had some confusion when the latest tag moved from 2.0 release candidate back to 1.3.2: opensearch-project/OpenSearch#3222 I think it would be good to ensure in the future that the latest tag will never move backwards across major versions, since that will almost certainly cause compatibility issues.

[bbarani]

@peterzhuamazon @zelinh @prudhvigodithi Can you please update the current status of this issue? I assume we have added the ability to tag the versions as per this proposal. Please confirm and update this issue with the details.

[bbarani]

@peterzhuamazon @zelinh @prudhvigodithi Can you please update the current status of this issue?

[prudhvigodithi]

Hey we already have support to tag images based on an specific input
for data-prepper using the shared library copyContainer, should be able to tag the data-prepper images as required.

[dlvenable]

Our plan for Data Prepper 1.5.0 - which will be the first with the 1 tag applied - is to use the Docker copy job rather than perform the copy in the Data Prepper promotion job. This should yield the same result of having a data-prepper:1 tag.

[zelinh]

Closing this issue as this has been accomplished in my new PR linked above.