Time series based workload desc order optimization through reverse segment read

[gashutos]

Description

EDIT -> After incorporating comments/suggestions from Nick, Andriya & Andorss, we decided to limit this change only to @timestamp field indices and also we changed default segment search order for these indices as Descending (from latest to the oldest segments), and also safe guarded rare ASC queries on time series based workload not to traverse on default DESC order to avoid any regression. The perf gains still remains similar.

As described in issue OpenSearch-6814, desc order sort queries are taking more time (3x or more) compare to asc order queries on same data set (time series based).
That is because all latest documents in time series workload are in latest segments when our IndexSearcher search documents in order of oldest segments to newest segments.

In this PR, we changed the leafReaderContext (segments) order in ContextIndexSearcher in reverse if query type is descending order sort. This will by default behaviour for all type of workloads and not just for time series based workloads. However it will only benefit to workload which has ever increasing data values. And for other type of workload, it wont regress or impact since data is randomly distributed across all segments.
But as a precaution, in case any user/workload gets search latency hit, I have added index level setting to disable this optimization.

Optimization gains

We are seeing 3x to 5x gains on below configuration.

• On http_logs workload with disabled force merge, so each shard has around 100-150 segments.
• instance type, r6g.2xlarge.
• units are in ms.

Sort Type	hits = 5	hits = 100	hits = 1000
Desc	150	462	621
Desc (with this change)	49	112	125

Issues Resolved

OpenSearch-6814

Next action item

On same segment of time series based workload, ASC queiry is faster compare to descending order. We need to look further to that.

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[gashutos]

@nknize @reta , check once you get time, if this appraoch looks good. The optimization gain is pretty good.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14293/
• CommitID: a776ebb
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/14294/
• CommitID: caed403
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7244-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 4c98b3d38064b94a87f6d7a1359e623849459bac
# Push it to GitHub
git push --set-upstream origin backport/backport-7244-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7244-to-2.x.