Enable versioning for jenkins shared libraries

[gaiksaya]

Problem:

Since the jenkinsFile for individual client or any component repositories will be living in the respective GitHub repository, the share jenkins library stored in opensearch-build repo will act as upstream library for all actions.
If not versioned, a breaking change can be introduced causing the non-exercised release workflows in remote repositories to break.

Acceptance Criteria:

• Versioned jenkins shared libraries are published somewhere
• All jenkins files use a particular version of shared library and not just point to main or current commit

[gaiksaya]

Researched a bit on pulling different versions for remote jenkins shared libraries, and found this plugin https://plugins.jenkins.io/workflow-cps-global-lib-http/
Discussed with @prudhvigodithi, we can publish the zips to maven or any other artifact repository (s3 works as well) for different versions and accordingly pull the required version.

Few things to keep in mind:

• Jenkins Shared Library versions will be independent of opensearch and other product's version. (should we separate this out into new repository?)
• Version release notes, change log needs to be maintained to let the consumer of this library know what changed and if the next version can cause any breaking changes.
• Versioning should follow semver

[dblock]

+1 on separating the library into a new repository and automating away builds/releases

[prudhvigodithi]

This is even possible with git, separating out is good, if so then we should consider separating out the python workflows as well.

What is propose is?

Upon every merge to main or when we call for a date of release (Not linked to the Os and OSD release).
git clone -b main https://github.com/opensearch-project/opensearch-build.git --depth 1
git archive --format zip --output buildsetup-1.0.0.zip HEAD src/ vars/ tests/
(Optional with tag: git archive --format zip --output buildsetup-1.0.0.zip v1.2.1 vars/ tests/, where v1.2.1 is the gittag)
This will archive the shared libraries and and python code base workflows to a zip file.

Publish this zip file to maven org.opensearch.buildcode.buildsetup.1.0.0 or to any other location where it can be later imported and extracted to get the shared libraries and and python code to execute the builds.

Now for every Jenkinsfile (distribution builds or client releases), the 1st stage would always be getting the right compatible version zip and extracting it, so that the code base is now on local workspace which facilities the next stages of execution.

With this versioning is enabled for the main code (shared libraries and and python code), which eliminates the branching if organized properly what can go into the zip file.
Jenkinsfiles are like constants will always be on the main branch, but its shared libraries are versioned.

[gaiksaya]

I think we are complicating this by mixing 2 things, python build code and jenkins shared library.
I agree they are connected today in someway but that won't be the case in the long run. For example, python code receives frequent contributions which might need frequent releases compared to jenkins shared libraries which won't be that often.
This version history might be confusing for external consumers who do not care about python build code.
WDYT @bbarani ?

[prudhvigodithi]

All I'm thinking is versioning the entire code and inside that code we can choose if we want shared libraries or normal workflows, I dont deny its easy to separate the shared libraries and version it, but its always a hanging piece to maintain, from rest of the code base, with this I'm open for thoughts.
@dblock @bbarani

[bbarani]

I would recommend moving forward with separate repo for Jenkins shared library for now to unblock the progress and follow up on other items soon as required. @prudhvigodithi @gaiksaya

[gaiksaya]

Sounds good. In that case I don't think we need to publish at all as git tags can be used by remote repositories to use a particular version of shared libraries using dynamic retrieval method https://www.jenkins.io/doc/book/pipeline/shared-libraries/#dynamic-retrieval
As for python code we have issue #2490 opened to figure out that problem.
Thanks!

[gaiksaya]

Listing down the files/folders that need to be moved to new repository:

• https://github.com/opensearch-project/opensearch-build/vars
• https://github.com/opensearch-project/opensearch-build/tree/main/src/jenkins
• https://github.com/opensearch-project/opensearch-build/tree/main/tests/jenkins
• https://github.com/opensearch-project/opensearch-build/tree/main/publish

We need to make sure that the code inside this libraries are not cloning main branches of build repo else versioning is irrelevant.

Also we have been confusing these jenkins shared libraries as methods which is not right. Libraries should be generic.
All other code bases should go in the python code base and not groovy.

Once the libraries are moved, the current as well as future jenkins file will use this libraries as below:

library identifier: 'jenkins@<tag>', retriever: modernSCM(
  [$class: 'GitSCMSource',
    remote: 'https://github.com/opensearch-project/opensearch-build-shared-libraries.git'])

Resources to move this libraries while preserving the history: https://medium.com/@ayushya/move-directory-from-one-repository-to-another-preserving-git-history-d210fa049d4b

cc: @opensearch-project/engineering-effectiveness

[prudhvigodithi]

Just thinking out of the scope of this issue.

My thought earlier was to ship the entire build repo as a separate version, which I have proposed here , where each required use case can pick up the right version zip and then it has all the logic of shared libs, workflows etc. If we are separating out just entity of the build repo, we should then consider making it as loosely coupled as possible.

We have to take a call, once we separate out the the shared libs to a new repo, for build repo considering the jenkinsfiles will be hosted in the respective component repo as per the universal build plan, we dont need to go with proposed branching anymore. The shared libs which is possible for a frequent change are versioned based on tags in new repo.

We need to make sure that the code inside this libraries are not cloning main branches of build repo else versioning is irrelevant.

For this I agree we have to remove that logic where it looks for a specific branch, instead the libraries should call the code that is part of the workspace.
Example: buildManifest.groovy, if you see it uses the local workspace code that is part of the src/ folder, something like this should be good. This way it does not cause a problem and we should allow Jenkinsfile to decide which branch of build repo to clone.

In future If required we can create a new repo even for the entire workflow source code, but at this point its compatible to build with multiple versions. Image the workflow source code is separated into a new repo, then there would be 2 stages in jenkinsfile one is to load the shared libraries using modernSCM and next is to load the workflow source code using gitSCM followed by the actual build.

  gitSCM{
    userRemoteConfigs{
      userRemoteConfig{
        name("opensearch-build-workflows")
        url("https://github.com/opensearch-project/opensearch-build-workflows.git")
        refspec("tag")
      }

To summarize:

@dblock @bbarani @gaiksaya

[gaiksaya]

[Offline discussion] Apart from shared libraries, we would still need branching or code separation for build side of things. Example: OpenSearch makes some huge change in upcoming 3.x that changes the rpm spec file or tar-install.sh script. In that case the changes to build code should be only concerned to 3.x and not 2.x. We do not have a mechanism today to separate that out.

As for shared libraries, few jenkins libraries like rpmDistValidation needs to move out of the groovy code and converted to python code to make it easier for the any user to even run it locally which is not possible today.
Only CI related libraries need to be part of jenkins code base as stated before.

We will be moving ahead with separating the code base into new GitHub repo called opensearch-build-libraries.
Need to define the release process for the repo.
Some good examples that already exist out there: https://github.com/cloudogu/ces-build-lib
https://github.com/cloudogu/gitops-build-lib

[gaiksaya]

Jenkins shared libraries are now separate into new repository: https://github.com/opensearch-project/opensearch-build-libraries
We will be using the above repo for adding new libraries. How we manage release cycle for this new repo can be discussed in the issue opened here: opensearch-project/opensearch-build-libraries#5
Also the current jenkins files in this repo and other repos need to consume the new repo as remote library which can be tracked here #2646

Closing this issue!
Thanks!