[Concurrent Segment Search][BUG] org.opensearch.search.profile.query.QueryProfilerIT.testProfileQuery {p0={"search.concurrent_segment_search.enabled":"true"} p1=true} is flaky

[reta]

Describe the bug
The test case org.opensearch.search.profile.query.QueryProfilerIT.testProfileQuery {p0={"search.concurrent_segment_search.enabled":"true"} p1=true} is flaky

Failed to execute phase [query], all shards failed; shardFailures {[cWz5tn-wSD6HkZtOCYeenQ][test][0]: RemoteTransportException[[node_s1][127.0.0.1:42349][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute concurrent segment search thread]]; nested: AssertionError; }
	at __randomizedtesting.SeedInfo.seed([B1A9A3487FC5243D:4A8940A482451BDC]:0)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:677)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:373)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:716)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:485)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:295)
	at app//org.opensearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:104)
	at app//org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:75)
	at app//org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:749)
	at app//org.opensearch.transport.TransportService$6.handleException(TransportService.java:875)
	at app//org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1483)
	at app//org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1597)
	at app//org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1571)
	at app//org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:80)
	at app//org.opensearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:72)
	at app//org.opensearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:70)
	at app//org.opensearch.action.ActionRunnable.onFailure(ActionRunnable.java:104)
	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:54)
	at app//org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at app//org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
	at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908)
	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base@20.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base@20.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base@20.0.2/java.lang.Thread.run(Thread.java:1623)
Caused by: OpenSearchException; nested: AssertionError;
	at app//org.opensearch.OpenSearchException.guessRootCauses(OpenSearchException.java:708)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:371)
	... 23 more
Caused by: java.lang.AssertionError
	at org.opensearch.search.profile.query.AbstractQueryProfileTree.startRewriteTime(AbstractQueryProfileTree.java:48)
	at org.opensearch.search.profile.query.QueryProfiler.startRewriteTime(QueryProfiler.java:78)
	at org.opensearch.search.internal.ContextIndexSearcher.rewrite(ContextIndexSearcher.java:189)
	at org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.rewrite(AbstractMultiTermQueryConstantScoreWrapper.java:174)
	at org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight$1.get(AbstractMultiTermQueryConstantScoreWrapper.java:269)
	at org.opensearch.search.profile.query.ProfileWeight$1.get(ProfileWeight.java:96)
	at org.opensearch.search.profile.query.ProfileWeight.scorer(ProfileWeight.java:72)
	at org.apache.lucene.search.Weight.bulkScorer(Weight.java:166)
	at org.opensearch.search.profile.query.ProfileWeight.bulkScorer(ProfileWeight.java:123)
	at org.opensearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:380)
	at org.opensearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:321)
	at org.opensearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:280)
	at org.apache.lucene.search.IndexSearcher.lambda$search$1(IndexSearcher.java:723)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 8 more

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.profile.query.QueryProfilerIT" -Dtests.method="testProfileQuery {p0={"search.concurrent_segment_search.enabled":"true"} p1=true}" -Dtests.seed=B1A9A3487FC5243D 

Expected behavior
Test must always pass

Plugins
Standard

Screenshots
N/A

Host/Environment (please complete the following information):

• CI

Additional context
https://build.ci.opensearch.org/job/gradle-check/24531/testReport/junit/org.opensearch.search.profile.query/QueryProfilerIT/testProfileQuery__p0___search_concurrent_segment_search_enabled___true___p1_true_/

[reta]

@ticheng-aws seems to be caused by #9248, could you please take a look? thank you

[jed326]

@reta This looks like a dupe of #9815?

[reta]

@reta This looks like a dupe of #9815?

@jed326 not really, the test is the same but failure mode is very different (AssertionError vs NPE)

[ticheng-aws]

@reta This looks like a dupe of #9815?

@jed326 not really, the test is the same but failure mode is very different (AssertionError vs NPE)

+1 This is a random query test.