Split, simplify and move usage of CI runners between OpenSearch and OpenSearch Dashboards

[dblock]

Is your feature request related to a problem? Please describe

CI runners are the kitchen sink of everything to support OpenSearch and OpenSearch Dashboards. We will run into conflicts sooner than later. We will also want to build multiple versions at the same time which will make it difficult to track what's what. For example, in centos7-x64-arm64-jdk14-node10.24.1-cypress6.9.1-20211005, OpenSearch doesn't need node or cypress, while OpenSearch Dashboards doesn't need Java. Then, we're going to lower the Java version for OpenSearch, so we'll need to use multiple images.

Describe the solution you'd like

• Split images, e.g. centos7-x64-arm64-jdk14 and centos7-x64-arm64-node10.24.1-cypress6.9.1.
• Move which build image to use into the build manifest so that it sticks to the version being built and not to the Jenkinsfile.

[reta]

Additionally, it would be great to have images for:

• centos7-x64-arm64-jdk11 with JDK-11
• centos7-x64-arm64-jdk17 with JDK-17

In case there is a question which JDK distributions to use, https://adoptium.net/ would make it. @peterzhuamazon @peternied could you please help, thank you!

[dblock]

Should we be a) installing JDK part of the build instead to minimize all these images, or b) making an image that contains all the JDKs pre-installed and having scripts select one? We can use sdk to install for example.

[reta]

We certainly could do that, pre-baking the images has the benefits that we don't need to download and install JDKs every time (+ gradle as well), so in general - builds are faster and more stable.

[gaiksaya]

The starting point to add dockerfiles to create those docker images would be https://github.com/opensearch-project/opensearch-build/tree/main/docker/ci/dockerfiles
@peterzhuamazon Need your input on how we build these images today?

[peterzhuamazon]

1. My concern is too fine-grain of a design if we split these images up. It would be even more complex to manage at that point. We can also include multiple JDK within the same docker and use alternative to switch based on demand.
2. All the build instructions are in the readme file of root.
3. I dont think we should associate any dependencies with Jenkins as @reta has pointed out consistency and time.

Thanks.

[reta]

Thanks @peterzhuamazon , just to summarize: it is easier to bake 1-2 all-in-one images (with multiple JDKs) than have the dedicated ones. It believe it should work. Do you have time to update the Dockerfiles or we could help you there and ask for the review? Thank you.

[peterzhuamazon]

Thanks @peterzhuamazon , just to summarize: it is easier to bake 1-2 all-in-one images (with multiple JDKs) than have the dedicated ones. It believe it should work. Do you have time to update the Dockerfiles or we could help you there and ask for the review? Thank you.

Free free to contribute. I will take a deeper look next week.
Thanks.

[reta]

@peterzhuamazon thanks a lot for merging the PRs, could you please ping us when the new images are available so we could alter the build pipelines to use the per-branch JDK specifications, thank you!

[peterzhuamazon]

@peterzhuamazon thanks a lot for merging the PRs, could you please ping us when the new images are available so we could alter the build pipelines to use the per-branch JDK specifications, thank you!

Will ping.

[peterzhuamazon]

@reta
docker pull opensearchstaging/ci-runner:centos7-x64-arm64-jdkmulti-node10.24.1-cypress6.9.1-20211019
docker pull opensearchstaging/ci-runner:al2-x64-arm64-jdkmulti-node10.24.1-cypress6.9.1-20211019

[peterzhuamazon]

https://hub.docker.com/r/opensearchstaging/ci-runner/tags?page=1&ordering=last_updated

[gaiksaya]

How did we build these images? Manually or is that automated somewhere?

[peterzhuamazon]

How did we build these images? Manually or is that automated somewhere?

Instructions in default README I will work on automation next week.

[reta]

@peternied @peterzhuamazon @dblock so we have images, manifests and Jenkinsfile updated, what are the next steps guys? I assume we should be switching the CI to use new pipelines (for PRs, etc, ...)?

[dblock]

Looks like we broke something - @kavilla said in a private chat: "seems like the docker stage will need an agent defined to do the work of reading the yaml file. which might defeat the purpose of just using a predefined docker agent to read the manifest for the docker agent" - someone will dig up the error(s) today and let's fix it

[dblock]

that Jenkins being still private, @peternied can help coordinate and make sure this all works next

[dblock]

I want to be able to close #74

[reta]

Looks like we broke something - @kavilla said in a private chat: "seems like the docker stage will need an agent defined to do the work of reading the yaml file. which might defeat the purpose of just using a predefined docker agent to read the manifest for the docker agent" - someone will dig up the error(s) today and let's fix it

I would expect that YAML manifest is read on master (since the agent is set to none), the docker stage comes in later. In any case, my apologies for breaking things, very difficult to make the changes right without the option to verify them. Please let me know if we need to adjust something, thank you.

[dblock]

Looks like we broke something - @kavilla said in a private chat: "seems like the docker stage will need an agent defined to do the work of reading the yaml file. which might defeat the purpose of just using a predefined docker agent to read the manifest for the docker agent" - someone will dig up the error(s) today and let's fix it

I would expect that YAML manifest is read on master (since the agent is set to none), the docker stage comes in later. In any case, my apologies for breaking things, very difficult to make the changes right without the option to verify them. Please let me know if we need to adjust something, thank you.

Definitely no need to apologize. It's on us that this infrastructure is private, we're working on it.

[peternied]

Looking into this 👀

If you are interested in contributing to that system checkout opensearch-ci

[dblock]

#790