[Concurrent Segment Search][BUG] org.opensearch.search.profile.query.QueryProfilerIT.testProfileQuery {p0={"search.concurrent_segment_search.enabled":"true"} p1=true} is flaky

[reta]

Describe the bug

The test case org.opensearch.search.profile.query.QueryProfilerIT.testProfileQuery {p0={"search.concurrent_segment_search.enabled":"true"} p1=true} is flaky:

Failed to execute phase [query], all shards failed; shardFailures {[NozaKKq8RO29Gdqj-Bi9WQ][test][0]: RemoteTransportException[[node_s4][127.0.0.1:41257][indices:data/read/search[phase/query]]]; nested: NullPointerException[Cannot invoke "java.lang.Long.longValue()" because the return value of "java.util.Map.get(Object)" is null]; }{[jmngMMEcTbqleNF9QtB2ZA][test][1]: RemoteTransportException[[node_s3][127.0.0.1:37065][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute concurrent segment search thread]]; nested: NotSerializableExceptionWrapper[assertion_error: null]; }{[NozaKKq8RO29Gdqj-Bi9WQ][test][2]: RemoteTransportException[[node_s4][127.0.0.1:41257][indices:data/read/search[phase/query]]]; nested: NullPointerException[Cannot invoke "java.lang.Long.longValue()" because the return value of "java.util.Map.get(Object)" is null]; }

Failed to execute phase [query], all shards failed; shardFailures {[NozaKKq8RO29Gdqj-Bi9WQ][test][0]: RemoteTransportException[[node_s4][127.0.0.1:41257][indices:data/read/search[phase/query]]]; nested: NullPointerException[Cannot invoke "java.lang.Long.longValue()" because the return value of "java.util.Map.get(Object)" is null]; }{[jmngMMEcTbqleNF9QtB2ZA][test][1]: RemoteTransportException[[node_s3][127.0.0.1:37065][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute concurrent segment search thread]]; nested: NotSerializableExceptionWrapper[assertion_error: null]; }{[NozaKKq8RO29Gdqj-Bi9WQ][test][2]: RemoteTransportException[[node_s4][127.0.0.1:41257][indices:data/read/search[phase/query]]]; nested: NullPointerException[Cannot invoke "java.lang.Long.longValue()" because the return value of "java.util.Map.get(Object)" is null]; }
	at __randomizedtesting.SeedInfo.seed([E0A359380AF3E50D:1B83BAD4F773DAEC]:0)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:677)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:373)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:716)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:485)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:295)
	at app//org.opensearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:104)
	at app//org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:75)
	at app//org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:749)
	at app//org.opensearch.transport.TransportService$6.handleException(TransportService.java:875)
	at app//org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1483)
	at app//org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:421)
	at app//org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:341)
	at app//org.opensearch.transport.InboundHandler.handleException(InboundHandler.java:419)
	at app//org.opensearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:411)
	at app//org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:159)
	at app//org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:115)
	at app//org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:767)
	at app//org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175)
	at app//org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150)
	at app//org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115)
	at app//org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:341)
	at app//org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246)
	at app//org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59)
	at app//org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152)
	at app//org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167)
	at app//org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438)
	at app//org.opensearch.nio.NioSelector.processKey(NioSelector.java:264)
	at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191)
	at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148)
	at java.base@20.0.2/java.lang.Thread.run(Thread.java:1623)
Caused by: OpenSearchException[Cannot invoke "java.lang.Long.longValue()" because the return value of "java.util.Map.get(Object)" is null]; nested: NullPointerException[Cannot invoke "java.lang.Long.longValue()" because the return value of "java.util.Map.get(Object)" is null];
	at app//org.opensearch.OpenSearchException.guessRootCauses(OpenSearchException.java:708)
	at app//org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:371)
	... 28 more
Caused by: java.lang.NullPointerException: Cannot invoke "java.lang.Long.longValue()" because the return value of "java.util.Map.get(Object)" is null
	at org.opensearch.search.profile.query.ConcurrentQueryProfileBreakdown.buildSliceLevelBreakdown(ConcurrentQueryProfileBreakdown.java:185)
	at org.opensearch.search.profile.query.ConcurrentQueryProfileBreakdown.toBreakdownMap(ConcurrentQueryProfileBreakdown.java:83)
	at org.opensearch.search.profile.query.ConcurrentQueryProfileTree.createProfileResult(ConcurrentQueryProfileTree.java:43)
	at org.opensearch.search.profile.query.ConcurrentQueryProfileTree.createProfileResult(ConcurrentQueryProfileTree.java:24)
	at org.opensearch.search.profile.AbstractInternalProfileTree.doGetTree(AbstractInternalProfileTree.java:183)
	at org.opensearch.search.profile.AbstractInternalProfileTree.doGetTree(AbstractInternalProfileTree.java:174)
	at org.opensearch.search.profile.AbstractInternalProfileTree.doGetTree(AbstractInternalProfileTree.java:174)
	at org.opensearch.search.profile.AbstractInternalProfileTree.getTree(AbstractInternalProfileTree.java:155)
	at org.opensearch.search.profile.query.ConcurrentQueryProfileTree.getTree(ConcurrentQueryProfileTree.java:73)
	at org.opensearch.search.profile.AbstractProfiler.getTree(AbstractProfiler.java:69)
	at org.opensearch.search.profile.SearchProfileShardResults.buildShardResults(SearchProfileShardResults.java:202)
	at org.opensearch.search.query.QueryPhase.execute(QueryPhase.java:162)
	at org.opensearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:546)
	at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:610)
	at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:579)
	at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74)
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1623)

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.profile.query.QueryProfilerIT" -Dtests.method="testProfileMatchesRegular {p0={"search.concurrent_segment_search.enabled":"true"} p1=true}" -Dtests.seed=E0A359380AF3E50D 

Expected behavior
A clear and concise description of what you expected to happen.

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context
Add any other context about the problem here.

@ticheng-aws this is another one, but this time it is NPE, please take a look

[ticheng-aws]

Taking a look

[reta]

@ticheng-aws I am reopening this one, since we muted tests but not fixed them