[Segment Replication] Use deterministic mechanism to have concurrent invocation of segment replication

[dreamer-89]

Description

Fixes SegmentReplicationTargetServiceTests.testShardAlreadyReplicating flaky test by ensuring first round of segment replication is not completed before invoking next round.

Related Issues

Resolves #8928

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21181/
• CommitID: 0984b60
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dreamer-89]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21181/
• CommitID: 0984b60
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

NRTReplicationReaderManagerTests.testUpdateSegmentsWhileRefreshing -> #8941
SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication -> #8059

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21200/
• CommitID: 0984b60
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates

• URL: https://build.ci.opensearch.org/job/gradle-check/21201/
• CommitID: 21f3036
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8937 (21f3036) into main (91bc891) will decrease coverage by 0.10%.
Report is 4 commits behind head on main.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #8937      +/-   ##
============================================
- Coverage     71.05%   70.95%   -0.10%     
- Complexity    57198    57215      +17     
============================================
  Files          4758     4758              
  Lines        269863   269863              
  Branches      39484    39484              
============================================
- Hits         191755   191491     -264     
- Misses        61982    62283     +301     
+ Partials      16126    16089      -37     

see 469 files with indirect coverage changes