Add CI bundle pattern to distribution download

[zelinh]

Signed-off-by: Zelin Hao zelinhao@amazon.com

Description

This PR is to add function to set up bundle distribution with the URL link from our CI build workflow.
setupDownloadServiceRepo function will download x64 tarball bundle distribution from CI build if property of bundleDownload is set and set to be true only.
This function would be used when we work on BWC tests running against test clusters with distribution bundles installed.

I will probably manually backport this PR to our 2.x, 2.4 and 1.3 branches as auto-backport might be failing once this is merged.

Issues Resolved

Part of opensearch-project/opensearch-build#2870

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7254/
• CommitID: cd134d7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.stats.IndexStatsIT.testFilterCacheStats

• URL: https://build.ci.opensearch.org/job/gradle-check/7255/
• CommitID: 7d3fb54
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov-commenter]

Codecov Report

Merging #5348 (1f1070f) into main (ff16ebd) will increase coverage by 0.07%.
The diff coverage is 45.45%.

@@             Coverage Diff              @@
##               main    #5348      +/-   ##
============================================
+ Coverage     70.93%   71.01%   +0.07%     
- Complexity    58268    58303      +35     
============================================
  Files          4733     4733              
  Lines        278241   278250       +9     
  Branches      40249    40251       +2     
============================================
+ Hits         197380   197599     +219     
+ Misses        64705    64465     -240     
- Partials      16156    16186      +30     

Impacted Files	Coverage Δ
.../opensearch/gradle/DistributionDownloadPlugin.java	87.23% <45.45%> (-5.71%)	⬇️
...cluster/coordination/PendingClusterStateStats.java	20.00% <0.00%> (-48.00%)	⬇️
...ain/java/org/opensearch/search/sort/MinAndMax.java	63.15% <0.00%> (-36.85%)	⬇️
...ensearch/client/indices/DetailAnalyzeResponse.java	20.54% <0.00%> (-34.25%)	⬇️
...nsearch/index/shard/IndexShardClosedException.java	66.66% <0.00%> (-33.34%)	⬇️
...min/cluster/snapshots/get/GetSnapshotsRequest.java	52.63% <0.00%> (-31.58%)	⬇️
...earch/client/indices/GetIndexTemplatesRequest.java	42.30% <0.00%> (-30.77%)	⬇️
...search/aggregations/pipeline/HoltWintersModel.java	21.47% <0.00%> (-30.21%)	⬇️
...opensearch/snapshots/SnapshotRestoreException.java	25.00% <0.00%> (-25.00%)	⬇️
...adcast/BroadcastShardOperationFailedException.java	55.55% <0.00%> (-22.23%)	⬇️
... and 456 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[dblock]

Should we be doing bcw against released versions of OpenSearch? Those aren't coming from ci.opensearch.org AFAIK, and certainly not from latest, or do they?

[zelinh]

Should we be doing bcw against released versions of OpenSearch? Those aren't coming from ci.opensearch.org AFAIK, and certainly not from latest, or do they?

We are thinking to run tests every time we generate a distribution bundle from the CI build workflow. Maybe this PR should be raised against 2.x or 2.4 branch first as main branch is targeting major version change, but I was thinking to backport this to 2.x/2.4 and tests based on these branches.

[peterzhuamazon]

Need to check how to use dynamic arch/platform/distro in the string pattern, instead of hardcoding x64/linux/tar.

[zelinh]

@peterzhuamazon This is probably not applicable as I did quite some research and didn't find a way to modify these pattern placeholder. The [classifier] in this case is linux-x64 but I'm not pretty aware how we could change to linux/x64 it as it's executed by gradle plugin. Open to any suggestions.

Also I see we have quite some bwcTests related are hardcoded to linux x64 and so were the BWC tests actively running in component repo are all hardcoded to use x64 architecture.

[saratvemulapalli]

@zelinh trying to understand whats the contention here? Is it with the format of linux-x64 vs linux/x64?
This won't work for other distributions (windows, linux arm64 etc)

[zelinh]

@saratvemulapalli Yes right. The current classifier is linux-x64 and can't be used for the path as linux/x64 here. The reason I hardcoded here is I see it's also hardcoded and used in component bwc plugin tests for example https://github.com/opensearch-project/anomaly-detection/blob/bae45dbb924d3b04e2838d9e8b1b62de95467ac0/build.gradle#L41
So we think we could kick off and give a try with x64 linux first.

[saratvemulapalli]

makes sense. We dont want to block this PR, but could you open an issue.
This should keep you going for the first phase

[zelinh]

@saratvemulapalli Thanks. I raised an issue regarding of this discussion here. #5535

[dblock]

Is bundleDownload necessary? Why can't this be always added?

[zelinh]

Is bundleDownload necessary? Why can't this be always added?

Hi @dblock The reason I use bundleDownload is to add the functionality without affecting current existing logics. For example, for bwc tests, we plan to have tests on both plugins and distribution bundle level. The one we have right now is on plugins level so the test cluster will only have OS min artifacts installed and plugin repo can install their own plugins to the cluster. If bundleDownload enabled, we would install the latest distribution bundle instead of just min.
From my perspective, if we add it always, we lost the functionality to just install min artifact for the test cluster.

[saratvemulapalli]

Why not just add ci/dbc to the pattern?

[zelinh]

Just updated to add it to the pattern.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7762/
• CommitID: 141dda4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7765/
• CommitID: b2f11d1
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7775/
• CommitID: b2f11d1
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7801/
• CommitID: b2f11d1
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dblock]

Is bundleDownload necessary? Why can't this be always added?

Hi @dblock The reason I use bundleDownload is to add the functionality without affecting current existing logics. For example, for bwc tests, we plan to have tests on both plugins and distribution bundle level. The one we have right now is on plugins level so the test cluster will only have OS min artifacts installed and plugin repo can install their own plugins to the cluster. If bundleDownload enabled, we would install the latest distribution bundle instead of just min. From my perspective, if we add it always, we lost the functionality to just install min artifact for the test cluster.

Ok now I understand. How about we rename this to customDistributionDownloadType (or similar) and default it to min, and make it an exclusive choice (either one or the other)?

[zelinh]

Is bundleDownload necessary? Why can't this be always added?

Hi @dblock The reason I use bundleDownload is to add the functionality without affecting current existing logics. For example, for bwc tests, we plan to have tests on both plugins and distribution bundle level. The one we have right now is on plugins level so the test cluster will only have OS min artifacts installed and plugin repo can install their own plugins to the cluster. If bundleDownload enabled, we would install the latest distribution bundle instead of just min. From my perspective, if we add it always, we lost the functionality to just install min artifact for the test cluster.

Ok now I understand. How about we rename this to customDistributionDownloadType (or similar) and default it to min, and make it an exclusive choice (either one or the other)?

Hi dB, I made the changes based on your suggestions. Now we default the customDistributionDownloadType to be min and only download distribution bundle when we specify it as bundle.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7969/
• CommitID: a7af13f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/8046/
• CommitID: 03710cd
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/8071/
• CommitID: 1f1070f

[saratvemulapalli]

@zelinh lets add this to the change log and lets add some documentation to Testing.md[1]?
@dblock @peterzhuamazon any concerns?

[1] https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md

[zelinh]

@zelinh lets add this to the change log and lets add some documentation to Testing.md[1]? @dblock @peterzhuamazon any concerns?

[1] https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md

Thanks. Just added some documentations for instructions.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/8078/
• CommitID: 9a22463
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

The backport to 2.4 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.4 2.4
# Navigate to the new working tree
pushd ../.worktrees/backport-2.4
# Create a new branch
git switch --create backport/backport-5348-to-2.4
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fe8fd67884160e346def8c8503d09fb4b5eb5363
# Push it to GitHub
git push --set-upstream origin backport/backport-5348-to-2.4
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.4

Then, create a pull request where the base branch is 2.4 and the compare/head branch is backport/backport-5348-to-2.4.

[peternied]

I think there is an issue causing less frequently updated RC builds from being favored over snapshot builds, but I'm unsure if this is the best direction to move, since snapshot builds would always be available. Getting insight on the intention of this change might help determine the path forward on the PR I've just created

• Prioritize snapshot builds in distribution downloader #8731