Reduce memory copy in zstd compression

[luyuncheng]

As i see #3577 there is a zstd compression mode.
i think in some scenarios, like init compress buffer, we can save memory copy in ArrayUtil.grow instead of using ArrayUtil.growNoCopy

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16010/
• CommitID: a1b7871
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[reta]

@luyuncheng could you please fix DCO check https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin

[dblock]

Curious what the impact on this change is, any benchmarks or ballpark numbers?

cc: @backslasht @mulugetam who were doing some numbers

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16155/
• CommitID: 1edf16f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/16163/
• CommitID: 2f6334e
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #7681 (01bb220) into main (0921ad7) will decrease coverage by 0.22%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #7681      +/-   ##
============================================
- Coverage     70.89%   70.67%   -0.22%     
+ Complexity    56242    56130     -112     
============================================
  Files          4682     4682              
  Lines        266102   266100       -2     
  Branches      39070    39070              
============================================
- Hits         188658   188076     -582     
- Misses        61478    62130     +652     
+ Partials      15966    15894      -72     

Impacted Files	Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java	92.50% <ø> (-0.36%)	⬇️
.../index/codec/customcodecs/ZstdCompressionMode.java	84.09% <100.00%> (ø)
.../codec/customcodecs/ZstdNoDictCompressionMode.java	76.71% <100.00%> (ø)

... and 469 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16186/
• CommitID: 432f1ea
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16192/
• CommitID: 432f1ea
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu
      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=tasks.list/10_basic/tasks_list test}

• URL: https://build.ci.opensearch.org/job/gradle-check/16305/
• CommitID: 432f1ea
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[reta]

@dblock looks like a legit fix (of not doing unneeded array copy), wdyt?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringQueryPhase

• URL: https://build.ci.opensearch.org/job/gradle-check/16415/
• CommitID: 01bb220
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[Bukhtawar]

While this one looks straight forward but its always a good practise to understand the optimisation and check if there are regressions if any with performance

[dblock]

@luyuncheng do you have any numbers?

[luyuncheng]

@luyuncheng do you have any numbers?

@dblock i think this optimize just reduce memory copy, mini benchmark can not show the exactly performance. AND it certainly not introduce any performance regression.

i do some mini benchmark with rally: a little promote
Track: http_logs , jvm 1G(less memory trigger more gc), 8core, 16G Mem, codec:zstd

Metric	Task	Baseline	Contender	Diff	Unit	Diff%
Cumulative indexing time of primary shards		260.726	253.455	-7.271	min	-2.79%
Cumulative merge time of primary shards		97.1903	95.1604	-2.02988	min	-2.09%
Cumulative merge count of primary shards		585	578	-7		-1.20%
Cumulative merge throttle time of primary shards		11.4248	10.4442	-0.98063	min	-8.58%
Cumulative refresh time of primary shards		23.4759	23.6735	0.1976	min	+0.84%
Cumulative refresh count of primary shards		2085	2047	-38		-1.82%
Cumulative flush time of primary shards		2.22883	2.26052	0.03168	min	+1.42%
Cumulative flush count of primary shards		122	121	-1		-0.82%
Total Young Gen GC time		263.871	256.28	-7.591	s	-2.88%
Total Young Gen GC count		16495	16437	-58		-0.35%
Total Old Gen GC time		52.601	47.93	-4.671	s	-8.88%
Total Old Gen GC count		606	591	-15		-2.48%
Min Throughput	index-append	89497.2	94867.8	5370.6	docs/s	+6.00%
Mean Throughput	index-append	96260.5	97990.2	1729.73	docs/s	+1.80%
Median Throughput	index-append	96184.2	97639.2	1455.02	docs/s	+1.51%
99th percentile latency	index-append	1271.33	1250.93	-20.4042	ms	-1.60%
99.9th percentile latency	index-append	1861	1860.47	-0.52277	ms	-0.03%
99.99th percentile latency	index-append	2799.13	2337.49	-461.639	ms	-16.49%
99th percentile service time	index-append	1271.33	1250.93	-20.4042	ms	-1.60%
99.9th percentile service time	index-append	1861	1860.47	-0.52277	ms	-0.03%
99.99th percentile service time	index-append	2799.13	2337.49	-461.639	ms	-16.49%

[dblock]

Looks innocent enough. I merged it. Thanks @luyuncheng!

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7681-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a0a8a181551c536228fc96bb0f27030b892ae51a
# Push it to GitHub
git push --set-upstream origin backport/backport-7681-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7681-to-2.x.

[reta]

@luyuncheng mind please creating a manual backport to 2.x? thank you

[reta]

@luyuncheng mind please creating a manual backport to 2.x? thank you

Backported

[luyuncheng]

@luyuncheng mind please creating a manual backport to 2.x? thank you

Backported

Thanks !!!