Remove SEARCH_PIPELINE feature flag

[noCharger]

Description

Remove Search Pipeline SEARCH_PIPELINE feature flag.

Instead of flip the flag, remove the gating logic with some considerations:

1. We should be quite certain that having this feature enabled doesn’t negatively impact unrelated operations
2. We can always reset this PR for rollback
3. It's confusing to users by fliping the flag opensearch.experimental.feature.search_pipeline.enabled, from its naming the feature is still in experimental
4. We can also update the flag name and then set default to True, but it seems not a common practice. Open to feedbacks

Will have a follow up PR to remove the flag on bwc test once the this PR is backported

Related Issues

#8489

Check List

• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19349/
• CommitID: fcc9aea
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19363/
• CommitID: 00d46cd
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19360/
• CommitID: 573b88e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19364/
• CommitID: 13a0fe0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19365/
• CommitID: bae0180
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19371/
• CommitID: b9c366b
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[noCharger]

BWC test failed https://build.ci.opensearch.org/job/gradle-check/19371/testReport/

 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test basic pipeline crud}	0.26 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test Get All Pipelines}	0.22 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test invalid config}	0.23 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test Put Versioned Pipeline}	0.22 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test basic pipeline crud}	1.2 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test Get All Pipelines}	0.68 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test invalid config}	0.37 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test Put Versioned Pipeline}	0.23 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test basic pipeline crud}	1.2 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test Get All Pipelines}	0.68 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test invalid config}	0.36 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test Put Versioned Pipeline}	0.22 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test basic pipeline crud}	1.2 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test Get All Pipelines}	0.68 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test invalid config}	0.36 sec	1
 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search_pipeline/10_basic/Test Put Versioned Pipeline}	0.22 sec	1

While same test case passed test.rest suite

test {p0=search_pipeline/10_basic/Test Get All Pipelines} | 0.3 sec | Passed
test {p0=search_pipeline/10_basic/Test Put Versioned Pipeline} | 0.66 sec | Passed
test {p0=search_pipeline/10_basic/Test basic pipeline crud} | 0.6 sec | Passed
test {p0=search_pipeline/10_basic/Test invalid config} | 0.21 sec | Passed

@msfroh any ideas to bypass the bwc test?

[andrross]

Maybe you need to keep the setting 'opensearch.experimental.feature.search_pipeline.enabled', 'true' setting for the mixed cluster so that setting gets applied against the 2.x version which still requires the feature flag? (at least until after this change is backported)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20350/
• CommitID: 1bb0954
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20351/
• CommitID: 61be29c
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20353/
• CommitID: ac53d93
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[noCharger]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20353/
• CommitID: ac53d93
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Flaky test https://build.ci.opensearch.org/job/gradle-check/20353/testReport/org.opensearch.action.admin.cluster.node.tasks/ResourceAwareTasksTests/testBasicTaskResourceTracking/

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20359/
• CommitID: ac53d93
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20362/
• CommitID: ac53d93
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[noCharger]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20359/
• CommitID: ac53d93
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

»  org.opensearch.bootstrap.StartupException: SettingsException[unknown setting [opensearch.experimental.feature.search_pipeline.enabled] did you mean any of [opensearch.experimental.feature.remote_store.enabled, opensearch.experimental.feature.extensions.enabled, opensearch.experimental.feature.identity.enabled, opensearch.experimental.feature.telemetry.enabled]?]

This is expected to be addressed after this PR backported. See #8513 (comment) for details

[macohen]

I don’t see any flags left over and the backport will fix the 2.x issue.

[noCharger]

 ./gradlew ':qa:mixed-cluster:v2.10.0#mixedClusterTest'
BUILD SUCCESSFUL in 9m 13s

Local test passed after 2.x and 2.9 merged in

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

• URL: https://build.ci.opensearch.org/job/gradle-check/20423/
• CommitID: 3170953
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8513 (a224a56) into main (8bd6b7c) will decrease coverage by 0.03%.
The diff coverage is 40.00%.

@@             Coverage Diff              @@
##               main    #8513      +/-   ##
============================================
- Coverage     71.03%   71.00%   -0.03%     
+ Complexity    57235    57227       -8     
============================================
  Files          4771     4771              
  Lines        270323   270311      -12     
  Branches      39509    39506       -3     
============================================
- Hits         192016   191934      -82     
- Misses        62142    62202      +60     
- Partials      16165    16175      +10     

Impacted Files	Coverage Δ
...pensearch/common/settings/FeatureFlagSettings.java	50.00% <ø> (ø)
.../java/org/opensearch/common/util/FeatureFlags.java	91.66% <ø> (-0.65%)	⬇️
server/src/main/java/org/opensearch/node/Node.java	86.02% <ø> (-0.02%)	⬇️
...opensearch/search/builder/SearchSourceBuilder.java	69.56% <0.00%> (-0.44%)	⬇️
...nsearch/search/pipeline/SearchPipelineService.java	85.58% <0.00%> (+0.58%)	⬆️
.../main/java/org/opensearch/action/ActionModule.java	96.33% <100.00%> (-0.01%)	⬇️
.../main/java/org/opensearch/index/IndexSettings.java	85.13% <100.00%> (-0.12%)	⬇️

... and 443 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush
      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringFetchPhase

• URL: https://build.ci.opensearch.org/job/gradle-check/20424/
• CommitID: a224a56
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8513-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 57d5e90cd10033eb67e13731728af25ce76f484d
# Push it to GitHub
git push --set-upstream origin backport/backport-8513-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8513-to-2.x.

[opensearch-trigger-bot]

The backport to 2.9 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.9 2.9
# Navigate to the new working tree
pushd ../.worktrees/backport-2.9
# Create a new branch
git switch --create backport/backport-8513-to-2.9
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 57d5e90cd10033eb67e13731728af25ce76f484d
# Push it to GitHub
git push --set-upstream origin backport/backport-8513-to-2.9
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.9

Then, create a pull request where the base branch is 2.9 and the compare/head branch is backport/backport-8513-to-2.9.