Segment Replication - Allow search idle with no replicas

[mch2]

This is a re-open of #7736.

Description
With Segment Replication enabled we generally disable shard idle. This change ensures there are replicas in the index before disabling shard idle with SR. It will warn if a user attempts to update the search.idle.after interval with replicas configured.

Related Issues
#7761

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/18004/
• CommitID: 5d11cdd
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8173 (fde8361) into main (6a19660) will decrease coverage by 0.02%.
The diff coverage is 55.55%.

@@             Coverage Diff              @@
##               main    #8173      +/-   ##
============================================
- Coverage     70.88%   70.87%   -0.02%     
- Complexity    57196    57200       +4     
============================================
  Files          4771     4771              
  Lines        270312   270323      +11     
  Branches      39505    39509       +4     
============================================
- Hits         191609   191585      -24     
- Misses        62600    62616      +16     
- Partials      16103    16122      +19     

Impacted Files	Coverage Δ
.../main/java/org/opensearch/index/IndexSettings.java	85.25% <0.00%> (-0.43%)	⬇️
...earch/action/search/AbstractSearchAsyncAction.java	77.03% <25.00%> (-1.50%)	⬇️
...in/java/org/opensearch/index/shard/IndexShard.java	69.65% <100.00%> (+0.44%)	⬆️
...ices/replication/RemoteStoreReplicationSource.java	94.28% <100.00%> (+0.73%)	⬆️

... and 476 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/18005/
• CommitID: 455ff46
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocationWithSegRepFailure
      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

• URL: https://build.ci.opensearch.org/job/gradle-check/18027/
• CommitID: dbdcff7
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/18051/
• CommitID: 85343e9
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[mch2]

@andrross @Bukhtawar Any concerns on this? I think this is still valuable particularly with remote store and we can explore enabling search idle across the board with SegRep in a separate effort.

[andrross]

Since "search idle" is a performance optimization I think we have a lot of options to change and improve the behavior going forward, so I don't see any one-way doors here. I also think this is an improvement over the current behavior of disabling search idle when there are no replicas, so I'm good with this change even as we explore improving the search idle behavior in all cases.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20209/
• CommitID: 9ab9fb5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20210/
• CommitID: 0a539ac
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[mch2]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20210/
• CommitID: 0a539ac
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

#8699

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/20224/
• CommitID: fde8361