Adding support to run integration tests in Jenkins

[saratvemulapalli]

Description

Adding support to run integration tests in Jenkins.

Parameters Integration Tests need:

1. Artifacts S3 Bucket - To pull build/bundle manifests, maven dependencies.
2. OpenSearch version - To pull build/bundle manifest.
3. Build ID - To map Build ID to a test run (Test Run ID) and to pull the right manifests.
4. CPU Architecture - To identify which agent node to run on.
5. Test Run ID - To publish results for a unique job ID

Issues Resolved

Closes #263

Check List

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #423 (da720a2) into main (1e66ff2) will increase coverage by 0.71%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #423      +/-   ##
==========================================
+ Coverage   64.82%   65.54%   +0.71%     
==========================================
  Files          48       47       -1     
  Lines        1288     1274      -14     
==========================================
  Hits          835      835              
+ Misses        453      439      -14     

Impacted Files	Coverage Δ
bundle-workflow/src/run_integ_test.py	0.00% <ø> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1e66ff2...da720a2. Read the comment docs.

[dblock]

1. It might not be possible, but a bundle defines the architecture for which it's built. This job lets me try and run a bundle for x86 on arm, which is not desirable. Can this be loaded from the bundle?
2. I'm worried that we're introducing test_integration.sh, which we have agreed to really be test.sh --suite integration or something like that. Can we do this, even if it's super hacky inside test.sh, in this PR so we don't have to go change it all over the place?

[peternied]

I was expecting to see all the legs of the integration tests executed, integration, bwc, perf is that coming in a future PR?

[saratvemulapalli]

I was expecting to see all the legs of the integration tests executed, integration, bwc, perf is that coming in a future PR?

Sure. This is a middle piece i.e bundle workflow -> test orchestrator -> individual test job.
This helps in keeping the jobs independent while the orchestrator combines all of them into one place and executes integ tests, bwc tests and perf tests.
This PR lays down the framework of how an individual test job can run.

[saratvemulapalli]

1. It might not be possible, but a bundle defines the architecture for which it's built. This job lets me try and run a bundle for x86 on arm, which is not desirable. Can this be loaded from the bundle?
2. I'm worried that we're introducing test_integration.sh, which we have agreed to really be test.sh --suite integration or something like that. Can we do this, even if it's super hacky inside test.sh, in this PR so we don't have to go change it all over the place?

1. It should be able to. Just didnt want to complicate the logic in Jenkins. But I agree and see the value that we are giving room to have mis-matched states while the truth already exists. I'll take care of this.
2. Absolutely, we went back and forth on this. I wanted to clear this up as part of Unify invoking the test workflows #393 while not holding this one up. I'll put something really simple for this PR and we could double down on the structure with Unify invoking the test workflows #393.

[peternied]

@saratvemulapalli During our discussion yesterday reading the manifest for architecture and reducing number of input parameters were mentioned - I'm waiting on those change

[dblock]

Which manifest?

[saratvemulapalli]

Not sure if the question was what manifest do we use for integ tests. If it is, it uses build and bundle manifest and it fetches it by itself.
I would like to have #456 update this readme to add the rest of the new parameters to this readme.

[setiah]

LGTM. Thanks @saratvemulapalli

[dblock]

Did you delete src/test.py?

[saratvemulapalli]

Yes, @setiah will add the test.sh in #456 as part of his PR.
Ref: #423 (comment)