[BUG] Flaky test failure org.opensearch.index.WaitUntilRefreshIT.classMethod

[dreamer-89]

Gradle link

https://build.ci.opensearch.org/job/gradle-check/13858

Identified in PR

#7094 (comment)

Failure trace

org.opensearch.index.WaitUntilRefreshIT > classMethod FAILED
    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=574, name=opensearch[node_s0][generic][T#7], state=RUNNABLE, group=TGRP-WaitUntilRefreshIT]

        Caused by:
        org.opensearch.core.concurrency.OpenSearchRejectedExecutionException: rejected execution of java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@77e3f6a0[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@53ad6424[Wrapped task = [threaded] fail_stale_replica]] on org.opensearch.threadpool.Scheduler$SafeScheduledThreadPoolExecutor@9ec95c2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 463]
            at __randomizedtesting.SeedInfo.seed([8BFF65004B7D5420]:0)
            at app//org.opensearch.common.util.concurrent.OpenSearchAbortPolicy.rejectedExecution(OpenSearchAbortPolicy.java:67)
            at java.****@19.0.2/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
            at java.****@19.0.2/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
            at java.****@19.0.2/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
            at app//org.opensearch.threadpool.ThreadPool.schedule(ThreadPool.java:421)
            at app//org.opensearch.common.util.concurrent.AbstractAsyncTask.rescheduleIfNecessary(AbstractAsyncTask.java:109)
            at app//org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:174)
            at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
            at java.****@19.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
            at java.****@19.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
            at java.****@19.0.2/java.lang.Thread.run(Thread.java:1589)

[reta]

Another variation here [1]:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1037, name=opensearch[node_s2][generic][T#7], state=RUNNABLE, group=TGRP-SearchFieldsIT]
Caused by: org.opensearch.core.concurrency.OpenSearchRejectedExecutionException: rejected execution of java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6ee84ba2[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@2526cc0a[Wrapped task = [threaded] fail_stale_replica]] on org.opensearch.threadpool.Scheduler$SafeScheduledThreadPoolExecutor@24c49208[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 181]
	at __randomizedtesting.SeedInfo.seed([37F8D9EE5752E8C5]:0)
	at app//org.opensearch.common.util.concurrent.OpenSearchAbortPolicy.rejectedExecution(OpenSearchAbortPolicy.java:67)
	at java.base@20.0.1/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
	at java.base@20.0.1/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
	at java.base@20.0.1/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
	at app//org.opensearch.threadpool.ThreadPool.schedule(ThreadPool.java:438)
	at app//org.opensearch.common.util.concurrent.AbstractAsyncTask.rescheduleIfNecessary(AbstractAsyncTask.java:109)
	at app//org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:174)
	at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
	at java.base@20.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base@20.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base@20.0.1/java.lang.Thread.run(Thread.java:1623)

[1] https://build.ci.opensearch.org/job/gradle-check/17035/testReport/junit/org.opensearch.search.fields/SearchFieldsIT/classMethod/

[sejli]

@MightyTQ, could you pick this up for OSCI? Thanks!

[dhwanilpatel]

Ran with 500 Iterations, it didn't failed. I see there is not any recent PR has refereed this failure as well.

Closing the issue.