Update lib location on jenkins/ to the remote repository opensearch-build-libraries

[jordarlu]

Description

Updating lib location on all jenkinsfiles/jobs at jenkins/ folder to use the lib located at new opensearch-build-libraries repository.
#2646

Issues Resolved

This PR is to update lib location of existing jenkinsfile to use new lib location at remote opensearch-build-libraries repository.
The update includes jenkinsfiles and regression files and BuildPipelineTest.groovy

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #2652 (8a376d1) into main (9697044) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##               main    #2652   +/-   ##
=========================================
  Coverage     93.94%   93.94%           
  Complexity       28       28           
=========================================
  Files           219      219           
  Lines          4494     4494           
  Branches         29       29           
=========================================
  Hits           4222     4222           
  Misses          266      266           
  Partials          6        6           

Impacted Files	Coverage Δ
...ests/jenkins/jobs/AssembleManifest_rpm_Jenkinsfile	100.00% <100.00%> (ø)
...ests/jenkins/jobs/AssembleManifest_tar_Jenkinsfile	100.00% <100.00%> (ø)
tests/jenkins/jobs/BuildManifest_Jenkinsfile	100.00% <100.00%> (ø)
tests/jenkins/jobs/BuildYumRepo_Jenkinsfile	100.00% <100.00%> (ø)
tests/jenkins/jobs/InputManifest_Jenkinsfile	100.00% <100.00%> (ø)
tests/jenkins/jobs/Messages_Jenkinsfile	100.00% <100.00%> (ø)
tests/jenkins/jobs/UploadIndexFile_Jenkinsfile	100.00% <100.00%> (ø)
...ts/jenkins/jobs/uploadMinSnapshotsToS3_Jenkinsfile	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[gaiksaya]

@jordarlu Groovy tests are failing. I saw this locally as well. Looks like the the below block of code needs change
https://github.com/opensearch-project/opensearch-build/blob/main/tests/jenkins/BuildPipelineTest.groovy#L37-L45
or override needs to be added for new shared library
Thanks!

[jordarlu]

Hi, we have all the relevant Test groovy files updated with using override to take lib version '1.0.0' on top of default version 'main' stated in BuildPipelineTest.groovy.

If you agree, @gaiksaya @prudhvigodithi , we can open another PR to update ./test/* and the rest who are still using the lib at local/opensearch-build repo.
After that, we can clean up/remove 'vars' in local/opensearch-build repo as a separated PR.

[gaiksaya]

Hi, we have all the relevant Test groovy files updated with using override to take lib version '1.0.0' on top of default version 'main' stated in BuildPipelineTest.groovy.

If you agree, @gaiksaya @prudhvigodithi , we can open another PR to update ./test/* and the rest who are still using the lib at local/opensearch-build repo. After that, we can clean up/remove 'vars' in local/opensearch-build repo as a separated PR.

Hi @jordarlu ,

I believe the changes to /tests should go in this PR and only removal or clean up for vars should go in next. Else there will be discrepancy in what tests are using and what the actual jenkinsFiles are using.
Thanks!

[jordarlu]

sounds good, @gaiksaya , let me do that and update! Thanks!

[jordarlu]

Hi, @gaiksaya , all relevant files in opensearch-build repo should be reflected with the new lib at opensearch-build-libraries repo. Please have a review, and if we are good to merge and move on to clean the vars at opensearch-build repo. Thanks.

[gaiksaya]

Enhancement to have this version number in some file from where it can be picked up and each jenkinsFile will default to that but can override to new version if required. More details:
opensearch-project/opensearch-build-libraries#7 (comment)

@opensearch-project/engineering-effectiveness Once this is merged we need to keep check on the Jenkins jobs that might break.

[peterzhuamazon]

Shall we remove legacy SCM since we dont use it anymore?

[jordarlu]

yep .. I think so ... as we are not going back to use legacySCM anymore ... let me go ahead and remove it, and then rerun the test .. thanks

[jordarlu]

Looks like we cannot remove this helper.registerAllowedMethod("legacySCM", [Closure.class], null) at the moment yet although the job does load the shared lib from the remote opensearch-build-libraries.git

I think when the job downloads/copies the remote lib (via the modernSCM method) from the opensearch-build-libraries repo into the local, and then finds the lib version is with "20211123" (for now), it then requires the legacySCM method to process.

Maybe we can keep the legacySCM regsiterAllowMethod for now, and come back to clean it up after we solve opensearch-project/opensearch-build-libraries#7 ? Thanks.

@peterzhuamazon @gaiksaya @prudhvigodithi

[peterzhuamazon]

Note:

lib = library(identifier: 'jenkins@1.0.0', retriever: modernSCM([
    $class: 'GitSCMSource',
    remote: 'https://github.com/opensearch-project/opensearch-build-libraries.git',
]))

In order to make it run correctly in tests:

import static com.lesfurets.jenkins.unit.global.lib.LibraryConfiguration.library
import static com.lesfurets.jenkins.unit.global.lib.GitSource.gitSource

helper.registerSharedLibrary(
            library().name('jenkins')
                .defaultVersion('main')
                .allowOverride(true)
                .implicit(true)
                .targetPath('vars')
                .retriever(gitSource('https://github.com/opensearch-project/opensearch-build-libraries.git'))
                .build()
        )

        helper.registerAllowedMethod("modernSCM", [Map.class], null)