Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Flaky test: org.opensearch.search.query.QueryProfilePhaseTests.testMinScoreDisablesCountOptimization #10518

Closed
peteralfonsi opened this issue Oct 10, 2023 · 3 comments · Fixed by #11104
Closed

[BUG] Flaky test: org.opensearch.search.query.QueryProfilePhaseTests.testMinScoreDisablesCountOptimization #10518

peteralfonsi opened this issue Oct 10, 2023 · 3 comments · Fixed by #11104
Assignees
@ticheng-aws
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run untriaged

Comments

@peteralfonsi
Copy link
Contributor

Describe the bug
QueryProfilePhaseTests.testMinScoreDisablesCountOptimization is flaky. See this CI failure.

To Reproduce
Run ./gradlew ':server:test' --tests "org.opensearch.search.query.QueryProfilePhaseTests"

Expected behavior
Test should succeed when run.

Additional context
Found through failed Jenkins build, reproduced on main branch.
@peteralfonsi peteralfonsi added bug Something isn't working untriaged labels Oct 10, 2023
@peteralfonsi peteralfonsi mentioned this issue Oct 10, 2023
Closed
5 tasks
@noCharger noCharger added the flaky-test Random test failure that succeeds on second run label Oct 13, 2023
@ticheng-aws
Copy link
Contributor

Taking a look

@ticheng-aws
Copy link
Contributor

Hi @peteralfonsi, the Jenkins build you provided is org.opensearch.search.query.QueryProfilePhaseTests.testMaxScore not testMinScoreDisablesCountOptimization. testMaxScore has fixed in PR #10547. Could you provide us the correct link?

@ticheng-aws
Copy link
Contributor

The issue happened on both concurrent and non-concurrent search cases.

  • p0=0
REPRODUCE WITH: ./gradlew 'null' --tests "org.opensearch.search.query.QueryProfilePhaseTests" -Dtests.method="testMinScoreDisablesCountOptimization [p0=0 p1=org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher@74c79fa2 seed=[DF8361490E483F60:5308BA3E89989F7]]" -Dtests.seed=DF8361490E483F60 -Dtests.locale=vi-VN -Dtests.timezone=America/Louisville -Druntime.java=17

java.lang.AssertionError: 
Expected: a value equal to or greater than <100L>
     but: <42L> was less than <100L>

	at __randomizedtesting.SeedInfo.seed([DF8361490E483F60:5308BA3E89989F7]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.junit.Assert.assertThat(Assert.java:930)
	at org.opensearch.search.query.QueryProfilePhaseTests.lambda$testMinScoreDisablesCountOptimization$10(QueryProfilePhaseTests.java:341)
	at org.opensearch.search.query.QueryProfilePhaseTests.lambda$assertProfileData$55(QueryProfilePhaseTests.java:1472)
	at org.opensearch.search.query.QueryProfilePhaseTests.assertProfileData(QueryProfilePhaseTests.java:1523)
	at org.opensearch.search.query.QueryProfilePhaseTests.assertProfileData(QueryProfilePhaseTests.java:1478)
	at org.opensearch.search.query.QueryProfilePhaseTests.assertProfileData(QueryProfilePhaseTests.java:1469)
	at org.opensearch.search.query.QueryProfilePhaseTests.testMinScoreDisablesCountOptimization(QueryProfilePhaseTests.java:339)
	at jdk.internal.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:833)
  • p0=5
REPRODUCE WITH: ./gradlew 'null' --tests "org.opensearch.search.query.QueryProfilePhaseTests" -Dtests.method="testMinScoreDisablesCountOptimization [p0=5 p1=org.opensearch.search.query.ConcurrentQueryPhaseSearcher@5cde6747 seed=[DF8361490E483F60:D96508ED9CF3686E]]" -Dtests.seed=DF8361490E483F60 -Dtests.locale=vi-VN -Dtests.timezone=America/Louisville -Druntime.java=17

java.lang.AssertionError: 
Expected: a value equal to or greater than <100L>
     but: <83L> was less than <100L>

	at __randomizedtesting.SeedInfo.seed([DF8361490E483F60:D96508ED9CF3686E]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.junit.Assert.assertThat(Assert.java:930)
	at org.opensearch.search.query.QueryProfilePhaseTests.lambda$testMinScoreDisablesCountOptimization$10(QueryProfilePhaseTests.java:341)
	at org.opensearch.search.query.QueryProfilePhaseTests.lambda$assertProfileData$55(QueryProfilePhaseTests.java:1472)
	at org.opensearch.search.query.QueryProfilePhaseTests.assertProfileData(QueryProfilePhaseTests.java:1523)
	at org.opensearch.search.query.QueryProfilePhaseTests.assertProfileData(QueryProfilePhaseTests.java:1478)
	at org.opensearch.search.query.QueryProfilePhaseTests.assertProfileData(QueryProfilePhaseTests.java:1469)
	at org.opensearch.search.query.QueryProfilePhaseTests.testMinScoreDisablesCountOptimization(QueryProfilePhaseTests.java:339)
	at jdk.internal.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:833)

@ticheng-aws ticheng-aws mentioned this issue Nov 6, 2023
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ticheng-aws ticheng-aws

Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run untriaged
Projects
Concurrent Search
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix flaky testMinScoreDisablesCountOptimization in query profile phase tests ticheng-aws/OpenSearch
4 participants
@yigithub @peteralfonsi @noCharger @ticheng-aws