CardinalityIT/NestedIT test failures with concurrent search enabled and AssertingCodec #8303

sohami · 2023-06-27T23:14:29Z

Description

The tests were failing because during the concurrent segment search for each slice the codec producers for the leafs were initialized by the slice thread. Later in reduce phase, the post collection happens over those codec producers on the search thread. With AssertingCodec it verifies that all access is done by the same thread causing the failures. More details are in #8095

Related Issues

Resolves #8095

Check List

New functionality includes testing.
- All tests pass
[] New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
[] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-06-27T23:22:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18515/
CommitID: 7628e7d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-28T00:23:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication

URL: https://build.ci.opensearch.org/job/gradle-check/18519/
CommitID: d9b107e
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-06-28T00:25:58Z

Codecov Report

Merging #8303 (8429693) into main (68ddf24) will increase coverage by 0.06%.
The diff coverage is 67.34%.

❗ Current head 8429693 differs from pull request most recent head e01b743. Consider uploading reports for the commit e01b743 to get more accurate results

@@             Coverage Diff              @@
##               main    #8303      +/-   ##
============================================
+ Coverage     70.93%   71.00%   +0.06%     
+ Complexity    57184    57108      -76     
============================================
  Files          4770     4760      -10     
  Lines        270280   269725     -555     
  Branches      39499    39460      -39     
============================================
- Hits         191726   191520     -206     
+ Misses        62439    62069     -370     
- Partials      16115    16136      +21

Impacted Files	Coverage Δ
...nsearch/search/internal/FilteredSearchContext.java	`8.92% <0.00%> (-0.25%)`	⬇️
...arch/aggregations/DefaultAggregationProcessor.java	`13.95% <50.00%> (+1.75%)`	⬆️
.../search/aggregations/BucketCollectorProcessor.java	`65.62% <65.62%> (ø)`
.../org/opensearch/search/internal/SearchContext.java	`38.00% <66.66%> (+1.82%)`	⬆️
...va/org/opensearch/search/DefaultSearchContext.java	`78.90% <100.00%> (+0.33%)`	⬆️
...arch/aggregations/AggregationCollectorManager.java	`82.75% <100.00%> (+10.03%)`	⬆️
...h/aggregations/ConcurrentAggregationProcessor.java	`81.25% <100.00%> (+1.25%)`	⬆️
...ensearch/search/internal/ContextIndexSearcher.java	`68.23% <100.00%> (+0.18%)`	⬆️

... and 579 files with indirect coverage changes

sohami · 2023-06-28T02:00:12Z

@reta Can you please help to review ?

server/src/main/java/org/opensearch/search/internal/ContextIndexSearcher.java

github-actions · 2023-07-11T20:24:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19968/
CommitID: f493a09
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sohami · 2023-07-11T23:33:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌

URL: https://build.ci.opensearch.org/job/gradle-check/19968/

CommitID: f493a09
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Failed tests are unrelated:

[org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/19968/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testStaleCommitDeletionWithInvokeFlush/)
[org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure](https://build.ci.opensearch.org/job/gradle-check/19968/testReport/junit/org.opensearch.remotestore/RemoteStoreRefreshListenerIT/testRemoteRefreshRetryOnFailure/)
[org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush](https://build.ci.opensearch.org/job/gradle-check/19968/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testStaleCommitDeletionWithInvokeFlush/)

server/src/main/java/org/opensearch/search/aggregations/BucketCollectorProcessor.java

server/src/main/java/org/opensearch/search/internal/ContextIndexSearcher.java

server/src/main/java/org/opensearch/search/aggregations/BucketCollectorProcessor.java

reta · 2023-07-12T17:40:55Z

@sohami could you please add changelog (may be using different phrasing), it seems we are changing not the tests but core functionality.

github-actions · 2023-07-12T17:41:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled

URL: https://build.ci.opensearch.org/job/gradle-check/20062/
CommitID: 8429693
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

sohami · 2023-07-12T22:30:23Z

@sohami could you please add changelog (may be using different phrasing), it seems we are changing not the tests but core functionality.

Added the changelog entry

github-actions · 2023-07-12T22:51:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20084/
CommitID: b623bc4
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sohami · 2023-07-13T16:26:30Z

@reta Can this be merged ?

reta · 2023-07-13T16:29:39Z

@reta Can this be merged ?

@sohami Green build? Please rebase may be

…nd AssertingCodec The tests were failing because during the concurrent segment search for each slice the codec producers for the leafs were initialized by the slice thread. Later in reduce phase, the post collection happens over those codec producers on the search thread. With AssertingCodec it verifies that all access is done by the same thread causing the failures Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

sohami · 2023-07-13T16:33:58Z

@reta Can this be merged ?

@sohami Green build? Please rebase may be

I did rebase yesterday as part of final commit and gradle seems to be failing for bwc2.9 jar. Let me rebase again to see if that fixes below issue.

* What went wrong:
Execution failed for task ':distribution:bwc:minor:buildBwcLinuxTar'.
> Building 2.9.0 didn't generate expected file /var/jenkins/workspace/gradle-check/search/distribution/bwc/minor/build/bwc/checkout-2.x/distribution/archives/linux-tar/build/distributions/opensearch-min-2.9.0-SNAPSHOT-linux-x64.tar.gz

github-actions · 2023-07-13T17:08:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/20124/
CommitID: e01b743

opensearch-trigger-bot · 2023-07-13T17:10:31Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8303-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 064f265362f6ae8296bfeed09aaaf48be86e6a4d
# Push it to GitHub
git push --set-upstream origin backport/backport-8303-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8303-to-2.x.

reta · 2023-07-13T17:12:23Z

Hehe @sohami please backport manually to 2.x, thank you

…nd AssertingCodec (opensearch-project#8303) * CardinalityIT/NestedIT test failures with concurrent search enabled and AssertingCodec The tests were failing because during the concurrent segment search for each slice the codec producers for the leafs were initialized by the slice thread. Later in reduce phase, the post collection happens over those codec producers on the search thread. With AssertingCodec it verifies that all access is done by the same thread causing the failures Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Address review comments Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> --------- Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

sohami · 2023-07-13T19:36:19Z

Hehe @sohami please backport manually to 2.x, thank you

@reta Created backport pr #8687. Thanks!

…nd AssertingCodec (#8303) (#8687) * CardinalityIT/NestedIT test failures with concurrent search enabled and AssertingCodec The tests were failing because during the concurrent segment search for each slice the codec producers for the leafs were initialized by the slice thread. Later in reduce phase, the post collection happens over those codec producers on the search thread. With AssertingCodec it verifies that all access is done by the same thread causing the failures * Address review comments --------- Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

…nd AssertingCodec (opensearch-project#8303) * CardinalityIT/NestedIT test failures with concurrent search enabled and AssertingCodec The tests were failing because during the concurrent segment search for each slice the codec producers for the leafs were initialized by the slice thread. Later in reduce phase, the post collection happens over those codec producers on the search thread. With AssertingCodec it verifies that all access is done by the same thread causing the failures Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Address review comments Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> --------- Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> Signed-off-by: sahil buddharaju <sahilbud@amazon.com>

…nd AssertingCodec (opensearch-project#8303) * CardinalityIT/NestedIT test failures with concurrent search enabled and AssertingCodec The tests were failing because during the concurrent segment search for each slice the codec producers for the leafs were initialized by the slice thread. Later in reduce phase, the post collection happens over those codec producers on the search thread. With AssertingCodec it verifies that all access is done by the same thread causing the failures Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Address review comments Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> --------- Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

…nd AssertingCodec (opensearch-project#8303) * CardinalityIT/NestedIT test failures with concurrent search enabled and AssertingCodec The tests were failing because during the concurrent segment search for each slice the codec producers for the leafs were initialized by the slice thread. Later in reduce phase, the post collection happens over those codec producers on the search thread. With AssertingCodec it verifies that all access is done by the same thread causing the failures Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> * Address review comments Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> --------- Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

sohami requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah, dbwiddis and sachinpkale as code owners June 27, 2023 23:14

sohami force-pushed the issue-8095 branch from 7628e7d to d9b107e Compare June 27, 2023 23:48

reta reviewed Jun 28, 2023

View reviewed changes

server/src/main/java/org/opensearch/search/internal/ContextIndexSearcher.java Outdated Show resolved Hide resolved

sohami force-pushed the issue-8095 branch from d9b107e to f493a09 Compare July 11, 2023 20:16

jed326 mentioned this pull request Jul 11, 2023

[Concurrent Segment Search] Fix SignificantTerms related ITs #8509

Closed

reta reviewed Jul 12, 2023

View reviewed changes

server/src/main/java/org/opensearch/search/aggregations/BucketCollectorProcessor.java Outdated Show resolved Hide resolved

reta reviewed Jul 12, 2023

View reviewed changes

server/src/main/java/org/opensearch/search/internal/ContextIndexSearcher.java Show resolved Hide resolved

reta reviewed Jul 12, 2023

View reviewed changes

server/src/main/java/org/opensearch/search/aggregations/BucketCollectorProcessor.java Show resolved Hide resolved

sohami force-pushed the issue-8095 branch from 8429693 to b623bc4 Compare July 12, 2023 22:28

reta approved these changes Jul 12, 2023

View reviewed changes

sohami added 2 commits July 13, 2023 09:32

Address review comments

e01b743

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

sohami force-pushed the issue-8095 branch from b623bc4 to e01b743 Compare July 13, 2023 16:34

reta merged commit 064f265 into opensearch-project:main Jul 13, 2023
7 checks passed

reta added the backport 2.x Backport to 2.x branch label Jul 13, 2023

sohami mentioned this pull request Jul 13, 2023

CardinalityIT/NestedIT test failures with concurrent search enabled and AssertingCodec #8687

Merged

5 tasks

sohami mentioned this pull request Aug 17, 2023

Evaluate using LeafCollector::finish API in lucene for aggregation postCollection processing #9411

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CardinalityIT/NestedIT test failures with concurrent search enabled and AssertingCodec #8303

CardinalityIT/NestedIT test failures with concurrent search enabled and AssertingCodec #8303

sohami commented Jun 27, 2023

github-actions bot commented Jun 27, 2023

github-actions bot commented Jun 28, 2023

codecov bot commented Jun 28, 2023 •

edited

Loading

sohami commented Jun 28, 2023

github-actions bot commented Jul 11, 2023

sohami commented Jul 11, 2023

Gradle Check (Jenkins) Run Completed with:

reta commented Jul 12, 2023

github-actions bot commented Jul 12, 2023

sohami commented Jul 12, 2023

github-actions bot commented Jul 12, 2023

sohami commented Jul 13, 2023

reta commented Jul 13, 2023

sohami commented Jul 13, 2023

github-actions bot commented Jul 13, 2023

opensearch-trigger-bot bot commented Jul 13, 2023

reta commented Jul 13, 2023

sohami commented Jul 13, 2023

CardinalityIT/NestedIT test failures with concurrent search enabled and AssertingCodec #8303

CardinalityIT/NestedIT test failures with concurrent search enabled and AssertingCodec #8303

Conversation

sohami commented Jun 27, 2023

Description

Related Issues

Check List

github-actions bot commented Jun 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 28, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Jun 28, 2023 • edited Loading

Codecov Report

sohami commented Jun 28, 2023

github-actions bot commented Jul 11, 2023

Gradle Check (Jenkins) Run Completed with:

sohami commented Jul 11, 2023

Gradle Check (Jenkins) Run Completed with:

reta commented Jul 12, 2023

github-actions bot commented Jul 12, 2023

Gradle Check (Jenkins) Run Completed with:

sohami commented Jul 12, 2023

github-actions bot commented Jul 12, 2023

Gradle Check (Jenkins) Run Completed with:

sohami commented Jul 13, 2023

reta commented Jul 13, 2023

sohami commented Jul 13, 2023

github-actions bot commented Jul 13, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Jul 13, 2023

reta commented Jul 13, 2023

sohami commented Jul 13, 2023

codecov bot commented Jun 28, 2023 •

edited

Loading