Add gradle check test for github workflows (Testing again)

[peterzhuamazon]

Signed-off-by: Peter Zhu zhujiaxi@amazon.com

Description

Add gradle check test for github workflows (Testing again)

Issues Resolved

opensearch-project/opensearch-build#851

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dreamer-89]

testBeforeIndexShardClosed_CancelsOngoingReplications test failure.

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testBeforeIndexShardClosed_CancelsOngoingReplications" -Dtests.seed=7063FEA65C9F22BE -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=is-IS -Dtests.timezone=Europe/San_Marino -Druntime.java=17

org.opensearch.indices.replication.SegmentReplicationTargetServiceTests > testBeforeIndexShardClosed_CancelsOngoingReplications FAILED
    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=663, name=opensearch[org.opensearch.indices.replication.SegmentReplicationTargetServiceTests][generic][T#1], state=RUNNABLE, group=TGRP-SegmentReplicationTargetServiceTests]
        at __randomizedtesting.SeedInfo.seed([7063FEA65C9F22BE:208DF90D4830F75]:0)

        Caused by:
        java.lang.NullPointerException: Cannot invoke "org.opensearch.indices.replication.common.ReplicationCollection$ReplicationRef.get()" because "replicationRef" is null
            at __randomizedtesting.SeedInfo.seed([7063FEA65C9F22BE]:0)
            at org.opensearch.indices.replication.SegmentReplicationTargetService.start(SegmentReplicationTargetService.java:175)
            at org.opensearch.indices.replication.SegmentReplicationTargetService$ReplicationRunner.run(SegmentReplicationTargetService.java:169)
            at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:739)
            at java.****/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
            at java.****/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
            at java.****/java.lang.Thread.run(Thread.java:833)

repository-s3 20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials test. Seems flaky as well. Has anything changed related to s3 credentials ?

  2> REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}" -Dtests.seed=7063FEA65C9F22BE -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=lt -Dtests.timezone=PST8PDT -Druntime.java=17 -Dtests.rest.denylist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*,repository_s3/60_repository_eks_credentials/*
  2> java.lang.AssertionError: Failure at [repository_s3/20_repository_permanent_credentials:257]: expected [2xx] status code but api [snapshot.delete] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"repository_exception","reason":"[repository_permanent] concurrent modification of the index-N file, expected current generation [6311] but it was not found in the repository","stack_trace":"RepositoryException[[repository_permanent] concurrent modification of the index-N file, expected current generation [6311] but it was not found in the repository]\n\tat 

[peterzhuamazon]

Gradle check on main branch success after changing to 128CPU setup:
https://github.com/opensearch-project/OpenSearch/runs/7081855652?check_suite_focus=true / https://build.ci.opensearch.org/job/gradle-check/14/

2.0 as well: https://build.ci.opensearch.org/job/gradle-check/13/console

[dreamer-89]

LGTM!

[opensearch-ci-bot]

❌   Gradle Check failure 5f599f6
Log 6384

Reports 6384

[owaiskazi19]

❌   Gradle Check failure 5f599f6 Log 6384

Reports 6384

@peterzhuamazon are we planning to remove ci-bot comments as a part of this PR?

[peterzhuamazon]

❌ Gradle Check failure 5f599f6 Log 6384
Reports 6384

@peterzhuamazon are we planning to remove ci-bot comments as a part of this PR?

This is old workflow @owaiskazi19 have nothing to do with the new one.

[owaiskazi19]

❌ Gradle Check failure 5f599f6 Log 6384
Reports 6384

@peterzhuamazon are we planning to remove ci-bot comments as a part of this PR?

This is old workflow @owaiskazi19 have nothing to do with the new one.

Yes. So are we planning to remove the old workflow or do we have any issue to track it?

[peterzhuamazon]

@dreamer-89 new workflow has much higher stable rate compares to old one, success again.
https://build.ci.opensearch.org/job/gradle-check/15/console

The old one is failure miserably during a test, flaky integTest.

[peterzhuamazon]

❌ Gradle Check failure 5f599f6 Log 6384
Reports 6384

@peterzhuamazon are we planning to remove ci-bot comments as a part of this PR?

This is old workflow @owaiskazi19 have nothing to do with the new one.

Yes. So are we planning to remove the old workflow or do we have any issue to track it?

Per our offline conversation I already stated I am going to disable it once we merge this.

[peterzhuamazon]

Seems like still need to present on these old branches, backport them.

[opensearch-trigger-bot]

The backport to 1.3 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.3 1.3
# Navigate to the new working tree
cd .worktrees/backport-1.3
# Create a new branch
git switch --create backport/backport-3717-to-1.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 660cc11612ee8d9380f6944a212cfdd66a86d36a
# Push it to GitHub
git push --set-upstream origin backport/backport-3717-to-1.3
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.3

Then, create a pull request where the base branch is 1.3 and the compare/head branch is backport/backport-3717-to-1.3.

[opensearch-trigger-bot]

The backport to 2.0 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.0 2.0
# Navigate to the new working tree
cd .worktrees/backport-2.0
# Create a new branch
git switch --create backport/backport-3717-to-2.0
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 660cc11612ee8d9380f6944a212cfdd66a86d36a
# Push it to GitHub
git push --set-upstream origin backport/backport-3717-to-2.0
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.0

Then, create a pull request where the base branch is 2.0 and the compare/head branch is backport/backport-3717-to-2.0.

[opensearch-trigger-bot]

The backport to 1.2 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.2 1.2
# Navigate to the new working tree
cd .worktrees/backport-1.2
# Create a new branch
git switch --create backport/backport-3717-to-1.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 660cc11612ee8d9380f6944a212cfdd66a86d36a
# Push it to GitHub
git push --set-upstream origin backport/backport-3717-to-1.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.2

Then, create a pull request where the base branch is 1.2 and the compare/head branch is backport/backport-3717-to-1.2.

[opensearch-trigger-bot]

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.x 1.x
# Navigate to the new working tree
cd .worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-3717-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 660cc11612ee8d9380f6944a212cfdd66a86d36a
# Push it to GitHub
git push --set-upstream origin backport/backport-3717-to-1.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-3717-to-1.x.

[dblock]

Could we try https://github.com/marketplace/actions/trigger-jenkins-job instead of huge amount of boilerplate scripting here?