Enhance CheckpointState to support no-op replication

[ashking94]

Description

The PR constitutes of the following changes -

• Modified checkpoint state to support no-op replication and allowing other Transport Replication Action on need basis as well.
• Tracked and Insync definition stays the same across the replication tracker and the master nodes.
• Global checkpoint computation made dependent on (replica) shards that have translog stored locally.
• PRRL checks and reliance on (replica) shards that have translog stored locally.

Issues Resolved

This would solve #5033 completely and #3706 partially.

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• FAILURES:

      6 org.opensearch.script.expression.MoreExpressionIT.classMethod
      4 org.opensearch.search.aggregations.DelayedShardAggregationClientYamlTestSuiteIT.test {yaml=delayed_aggs/10_basic/Delayed Aggs}
      4 org.opensearch.index.mapper.murmur3.MapperMurmur3ClientYamlTestSuiteIT.test {yaml=mapper_murmur3/10_basic/Mapper Murmur3}
      4 org.opensearch.example.expertscript.ExpertScriptClientYamlTestSuiteIT.test {yaml=script_expert_scoring/20_score/document scoring}
      4 org.opensearch.example.customsuggester.CustomSuggesterClientYamlTestSuiteIT.test {yaml=custom-suggester/20_suggest/test custom suggester}
      1 org.opensearch.test.rest.WaitForRefreshAndCloseIT.testUpdateAndThenClose
      1 org.opensearch.test.rest.WaitForRefreshAndCloseIT.testIndexAndThenClose
      1 org.opensearch.test.rest.WaitForRefreshAndCloseIT.testDeleteAndThenClose
      1 org.opensearch.test.rest.RequestsWithoutContentIT.testSimulatePipelineMissingBody
      1 org.opensearch.test.rest.RequestsWithoutContentIT.testPutSettingsMissingBody

• URL: https://build.ci.opensearch.org/job/gradle-check/6963/
• CommitID: 5bb021d
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• FAILURES:

• URL: https://build.ci.opensearch.org/job/gradle-check/6989/
• CommitID: 4c6282c
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• FAILURES:

• URL: https://build.ci.opensearch.org/job/gradle-check/7026/
• CommitID: 4913941
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• FAILURES:

• URL: https://build.ci.opensearch.org/job/gradle-check/7025/
• CommitID: 4667680
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7034/
• CommitID: 46903d5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7128/
• CommitID: 1b57504
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7129/
• CommitID: dbcdae3
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7130/
• CommitID: 40d6c8d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7132/
• CommitID: 4f6b1a7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7133/
• CommitID: 76c9e18
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[Bukhtawar]

Why can't we implement a ReplicasProxy which doesn't send request to the replicas?

[ashking94]

There are multiple cases we need to support -

• primary-primary peer recovery where the replication should work as is.
• Ability to support certain transport replication actions like publish checkpoint action.
• and then No-Op replication has to be supported using existing transport shard bulk action.

Let me see if all of these cases can be supported with replicas proxy with keeping the code readable, maintainable and extensible.

[ashking94]

@Bukhtawar - When we do not want the replication calls to fan out, the replica proxy comes in only at this point.

OpenSearch/server/src/main/java/org/opensearch/action/support/replication/ReplicationOperation.java

Line 312 in 0210b76

replicasProxy.performOn(shard, replicaRequest, primaryTerm, globalCheckpoint, maxSeqNoOfUpdatesOrDeletes, listener);

and the below piece of code would get executed as well, while this is not required in case of request to not fan out to replication group -

OpenSearch/server/src/main/java/org/opensearch/action/support/replication/ReplicationOperation.java

Lines 243 to 258 in 0210b76

	if (logger.isTraceEnabled()) {
	logger.trace("[{}] sending op [{}] to replica {} for request [{}]", shard.shardId(), opType, shard, replicaRequest);
	}
	totalShards.incrementAndGet();
	pendingActions.incrementAndGet();
	final ActionListener<ReplicaResponse> replicationListener = new ActionListener<ReplicaResponse>() {
	@Override
	public void onResponse(ReplicaResponse response) {
	successfulShards.incrementAndGet();
	try {
	updateCheckPoints(shard, response::localCheckpoint, response::globalCheckpoint);
	} finally {
	decPendingAndFinishIfNeeded();
	}
	}

[Bukhtawar]

Sorry couldn't follow you, but are you suggesting it might not be possible to create a proxy to control replication

[ashking94]

It is possible, but I think it would 2 things which can be avoided -

• Executes some code which is totally unnecessary (mentioned in my earlier comment)
• It would require create redundant replicaProxy on each of the TransportReplicationAction. IMO, this would just add lot of conditional logic in each of the TransportReplicationAction which the solution is this PR is handling in just one place itself.

[Bukhtawar]

Sorry but what prevents reuse since you mentioned we need redundant code
Also why can't we wire replication actions to use a flavor of replica proxy

[ashking94]

We would need to then handle the execution of unnecessary code that would get executed if we trigger the performOnReplicas method which then hits the replicaProxy.peform method. Not just that, but the added logic for each of transport replication actions.
Do you see any reason that why we must let the call pass through even with the controls that has been put up (allowing certain transport replication to proceed - publish checkpoint action & transport shard bulk action)?

[ashking94]

Done @Bukhtawar

[ashking94]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7132/
• CommitID: 4f6b1a7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Genuine failure. Trying to fix it at the earliest and proceed with other steps.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/7729/
• CommitID: 7b0ce5c

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/7732/
• CommitID: 3c3fb27

[Bukhtawar]

Should this class should be moved outside the tracker?

[ashking94]

Ack

[Bukhtawar]

This needs to be cleaned up now, possibly dead code

[ashking94]

Yes, making the change.

[Bukhtawar]

nit: could use == true instead

[ashking94]

Ack, making the change.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/7734/
• CommitID: bfbb71f

[Bukhtawar]

Thanks LGTM