Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] SearchableSnapshotsIntegTests testCreateAndRestoreSearchableSnapshot failing #116377

Closed
elasticsearchmachine opened this issue Nov 7, 2024 · 6 comments
Closed

[CI] SearchableSnapshotsIntegTests testCreateAndRestoreSearchableSnapshot failing #116377

elasticsearchmachine opened this issue Nov 7, 2024 · 6 comments
Assignees
@javanna
Labels
needs:risk Requires assignment of a risk label (low, medium, blocker) :Search Foundations/Search Catch all for Search Foundations Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

Build Scans:

Reproduction Line:

./gradlew ":x-pack:plugin:searchable-snapshots:internalClusterTest" --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsIntegTests.testCreateAndRestoreSearchableSnapshot" -Dtests.seed=BBCFB57310F3D344 -Dtests.locale=mas -Dtests.timezone=Europe/Monaco -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.Exception: Test abandoned because suite timeout was reached.

Issue Reasons:

  • [main] 2 failures in test testCreateAndRestoreSearchableSnapshot (0.4% fail rate in 459 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. >test-failure Triaged test failures from CI labels Nov 7, 2024
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 2 failures in test testCreateAndRestoreSearchableSnapshot (0.4% fail rate in 459 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Nov 7, 2024
@elasticsearchmachine
Mute org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsI…
3bc094f 
…ntegTests testCreateAndRestoreSearchableSnapshot #116377
@elasticsearchmachine elasticsearchmachine added needs:triage Requires assignment of a team area label needs:risk Requires assignment of a risk label (low, medium, blocker) labels Nov 7, 2024
kderusso pushed a commit to kderusso/elasticsearch that referenced this issue Nov 7, 2024
@elasticsearchmachine @kderusso
Mute org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsI…
50174c4 
…ntegTests testCreateAndRestoreSearchableSnapshot elastic#116377
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Nov 11, 2024
@original-brownbear
Unmute a lot of fixed tests from search race condition
d8a36b6 
closes elastic#115664 elastic#113430 elastic#115717 elastic#115705 elastic#115970 elastic#115988 elastic#115810 elastic#116027
 elastic#115754 elastic#116097 elastic#115818 elastic#116377 elastic#114824
@original-brownbear original-brownbear mentioned this issue Nov 11, 2024
Merged
original-brownbear added a commit that referenced this issue Nov 11, 2024
@original-brownbear
Unmute a lot of fixed tests from search race condition (#116587)
a21d375 
These are all fixed by #116264

closes #115664 #113430 #115717 #115705 #115970 #115988 #115810 #116027
 #115754 #116097 #115818 #116377 #114824
jozala pushed a commit that referenced this issue Nov 13, 2024
@elasticsearchmachine @jozala
Mute org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsI…
e3dbf9d 
…ntegTests testCreateAndRestoreSearchableSnapshot #116377
jozala pushed a commit that referenced this issue Nov 13, 2024
@original-brownbear @jozala
Unmute a lot of fixed tests from search race condition (#116587)
d2e8a30 
These are all fixed by #116264

closes #115664 #113430 #115717 #115705 #115970 #115988 #115810 #116027
 #115754 #116097 #115818 #116377 #114824
afoucret pushed a commit to afoucret/elasticsearch that referenced this issue Nov 14, 2024
@original-brownbear @afoucret
Unmute a lot of fixed tests from search race condition (elastic#116587)
776dfba 
These are all fixed by elastic#116264

closes elastic#115664 elastic#113430 elastic#115717 elastic#115705 elastic#115970 elastic#115988 elastic#115810 elastic#116027
 elastic#115754 elastic#116097 elastic#115818 elastic#116377 elastic#114824
@pxsalehi pxsalehi added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. labels Nov 21, 2024
@elasticsearchmachine elasticsearchmachine added Team:Distributed Coordination Meta label for Distributed Coordination team and removed needs:triage Requires assignment of a team area label labels Nov 21, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

@ywangd
Copy link
Member

ywangd commented Nov 21, 2024

The underlying failure is

[2024-11-07T07:11:37,867][ERROR][o.e.t.LeakTracker        ] [[Cleaner-0]] LEAK: resource was not cleaned up before it was garbage-collected.
Recent access records: 
Created at:
	in [Thread-76][testCreateAndRestoreSearchableSnapshot]
	org.elasticsearch.action.search.ArraySearchPhaseResults.<init>(ArraySearchPhaseResults.java:28)
	org.elasticsearch.action.search.QueryPhaseResultConsumer.<init>(QueryPhaseResultConsumer.java:101)
	org.elasticsearch.action.search.SearchPhaseController.newSearchPhaseResults(SearchPhaseController.java:824)
	org.elasticsearch.action.search.TransportSearchAction$AsyncSearchActionProvider.newSearchPhase(TransportSearchAction.java:1498)
	org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1301)
	org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:1034)
	org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$4(TransportSearchAction.java:361)
	org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247)
	org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)
	org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)
	org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:527)
	org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:317)
	org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:121)
	org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:135)
	org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:54)
	org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)
	org.elasticsearch.action.support.TransportAction.handleExecution(TransportAction.java:96)
	org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:59)
	org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:197)
	org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:106)
	org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:84)
	org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:140)
	org.elasticsearch.client.internal.FilterClient.doExecute(FilterClient.java:56)
	org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:140)
	org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:126)
	org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:37)
	org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:44)
	org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:17)
	org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertResponse(ElasticsearchAssertions.java:382)
	org.elasticsearch.xpack.searchablesnapshots.BaseSearchableSnapshotsIntegTestCase.lambda$assertTotalHits$3(BaseSearchableSnapshotsIntegTestCase.java:300)
	java.base/java.lang.Thread.run(Thread.java:1575)
[2024-11-07T07:11:37,868][ERROR][o.e.t.LeakTracker        ] [[Cleaner-0]] LEAK: resource was not cleaned up before it was garbage-collected.
Recent access records: 
Created at:
	in [Thread-76][testCreateAndRestoreSearchableSnapshot]
	org.elasticsearch.action.search.ArraySearchPhaseResults.<init>(ArraySearchPhaseResults.java:28)
	org.elasticsearch.action.search.SearchDfsQueryThenFetchAsyncAction.<init>(SearchDfsQueryThenFetchAsyncAction.java:70)
	org.elasticsearch.action.search.TransportSearchAction$AsyncSearchActionProvider.newSearchPhase(TransportSearchAction.java:1511)
	org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1301)
	org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:1034)
	org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$4(TransportSearchAction.java:361)
	org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247)
	org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)
	org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)
	org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:527)
	org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:317)
	org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:121)
	org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:135)
	org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:54)
	org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:132)
	org.elasticsearch.action.support.TransportAction.handleExecution(TransportAction.java:96)
	org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:59)
	org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:197)
	org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:106)
	org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:84)
	org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:140)
	org.elasticsearch.client.internal.FilterClient.doExecute(FilterClient.java:56)
	org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:140)
	org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:126)
	org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:37)
	org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:44)
	org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:17)
	org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertResponse(ElasticsearchAssertions.java:382)
	org.elasticsearch.xpack.searchablesnapshots.BaseSearchableSnapshotsIntegTestCase.lambda$assertTotalHits$3(BaseSearchableSnapshotsIntegTestCase.java:300)
	java.base/java.lang.Thread.run(Thread.java:1575)

Seems fairly similar to #115056. The build does have the fix #116060. But the issue has since been re-openned. I am rerouting this to the search team to take a look.

@ywangd ywangd added :Search Foundations/Search Catch all for Search Foundations and removed :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs labels Nov 21, 2024
@elasticsearchmachine elasticsearchmachine added Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch and removed Team:Distributed Coordination Meta label for Distributed Coordination team labels Nov 21, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-search-foundations (Team:Search Foundations)

@ywangd
Copy link
Member

ywangd commented Nov 21, 2024

Looks like it meant to be closed by #116587. @original-brownbear Could you please take a look and close it if it's already been fixed?

@javanna javanna self-assigned this Nov 22, 2024
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this issue Nov 28, 2024
@elasticsearchmachine @alexey-ivanov-es
Mute org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsI…
6336198 
…ntegTests testCreateAndRestoreSearchableSnapshot elastic#116377
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this issue Nov 28, 2024
@original-brownbear @alexey-ivanov-es
Unmute a lot of fixed tests from search race condition (elastic#116587)
94ffee1 
These are all fixed by elastic#116264

closes elastic#115664 elastic#113430 elastic#115717 elastic#115705 elastic#115970 elastic#115988 elastic#115810 elastic#116027
 elastic#115754 elastic#116097 elastic#115818 elastic#116377 elastic#114824
@javanna
Copy link
Member

javanna commented Dec 9, 2024

Indeed, this is no longer muted.

@javanna javanna closed this as completed Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@javanna javanna

Labels
needs:risk Requires assignment of a risk label (low, medium, blocker) :Search Foundations/Search Catch all for Search Foundations Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@javanna @ywangd @pxsalehi @elasticsearchmachine