Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fault-tolerant tests have flaky init #14035

Closed
findepi opened this issue Sep 7, 2022 · 5 comments
Closed

Fault-tolerant tests have flaky init #14035

findepi opened this issue Sep 7, 2022 · 5 comments
Labels
bug Something isn't working test

Comments

@findepi
Copy link
Member

findepi commented Sep 7, 2022

Fault-tolerant / faulttolerant. For example TestHiveTaskFailureRecoveryTest.init failed in https://github.com/trinodb/trino/runs/8228073689?check_suite_focus=true with

Error:  io.trino.faulttolerant.hive.TestHiveTaskFailureRecoveryTest.init  Time elapsed: 225.711 s  <<< FAILURE!
java.lang.RuntimeException: 
Error fetching next at http://127.0.0.1:40205/v1/statement/executing/20220907_135132_00013_vg4hy/y379558b510cc93ad493389051f1603855f39b07b/1 returned an invalid response: JsonResponse{statusCode=500, statusMessage=Internal Server Error, headers={content-length=[1016], content-type=[text/plain], date=[Wed, 07 Sep 2022 13:51:34 GMT]}, hasValue=false} [Error: java.util.concurrent.CancellationException: Task was cancelled.
	at com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:1543)
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:586)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547)
	at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:240)
	at com.google.common.util.concurrent.Futures.getDone(Futures.java:1181)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1130)
	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
]
	at io.trino.client.StatementClientV1.requestFailedException(StatementClientV1.java:453)
	at io.trino.client.StatementClientV1.advance(StatementClientV1.java:392)
	at io.trino.testing.AbstractTestingTrinoClient.execute(AbstractTestingTrinoClient.java:99)
	at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:480)
	at io.trino.plugin.hive.HiveQueryRunner.copyTableBucketed(HiveQueryRunner.java:375)
	at io.trino.plugin.hive.HiveQueryRunner.copyTpchTablesBucketed(HiveQueryRunner.java:339)
	at io.trino.plugin.hive.HiveQueryRunner$Builder.populateData(HiveQueryRunner.java:279)
	at io.trino.plugin.hive.HiveQueryRunner$Builder.build(HiveQueryRunner.java:258)
	at io.trino.plugin.hive.s3.S3HiveQueryRunner$Builder.build(S3HiveQueryRunner.java:150)
	at io.trino.faulttolerant.hive.TestHiveTaskFailureRecoveryTest.createQueryRunner(TestHiveTaskFailureRecoveryTest.java:71)
	at io.trino.faulttolerant.BaseFailureRecoveryTest.createQueryRunner(BaseFailureRecoveryTest.java:107)
	at io.trino.testing.AbstractTestQueryFramework.init(AbstractTestQueryFramework.java:95)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:515)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:217)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:144)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:169)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
	Suppressed: java.lang.Exception: SQL: CREATE TABLE orders WITH (bucketed_by=array['custkey'], bucket_count=11) AS SELECT * FROM tpch.tiny.orders
		at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:483)
		... 21 more
@findepi findepi added bug Something isn't working test labels Sep 7, 2022
@findepi
Copy link
Member Author

findepi commented Sep 7, 2022

@linzebing @arhimondr @losipiuk is it a duplicate of any other existing issue?
I couldn't find any, but you may know better

@findepi findepi mentioned this issue Sep 7, 2022
Merged
@ebyhr
Copy link
Member

ebyhr commented Sep 7, 2022

https://github.com/trinodb/trino/runs/8229808681

@ebyhr ebyhr mentioned this issue Sep 7, 2022
Merged
@ebyhr
Copy link
Member

ebyhr commented Sep 8, 2022

https://github.com/trinodb/trino/runs/8243366011

@losipiuk losipiuk mentioned this issue Sep 14, 2022
Merged
@findepi
Copy link
Member Author

findepi commented Sep 23, 2022

https://github.com/trinodb/trino/actions/runs/3112718741/jobs/5046822323

Error:  Tests run: 933, Failures: 1, Errors: 0, Skipped: 932, Time elapsed: 166.37 s <<< FAILURE! - in TestSuite
Error:  io.trino.faulttolerant.hive.TestHiveFaultTolerantExecutionJoinQueries.init  Time elapsed: 127.453 s  <<< FAILURE!
java.lang.RuntimeException: 
Error fetching next at http://127.0.0.1:34337/v1/statement/executing/20220923_133957_00006_2wfp6/y4ae55a208cec9acab0a48c9398e5aa65873d5273/1 returned an invalid response: JsonResponse{statusCode=500, statusMessage=Internal Server Error, headers={content-length=[1016], content-type=[text/plain], date=[Fri, 23 Sep 2022 13:39:58 GMT]}, hasValue=false} [Error: java.util.concurrent.CancellationException: Task was cancelled.
	at com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:1543)
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:586)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547)
	at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:240)
	at com.google.common.util.concurrent.Futures.getDone(Futures.java:1181)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1130)
	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
]
	at io.trino.client.StatementClientV1.requestFailedException(StatementClientV1.java:453)
	at io.trino.client.StatementClientV1.advance(StatementClientV1.java:392)
	at io.trino.testing.AbstractTestingTrinoClient.execute(AbstractTestingTrinoClient.java:99)
	at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:480)
	at io.trino.testing.QueryAssertions.copyTable(QueryAssertions.java:339)
	at io.trino.testing.QueryAssertions.copyTable(QueryAssertions.java:331)
	at io.trino.testing.QueryAssertions.copyTpchTables(QueryAssertions.java:323)
	at io.trino.plugin.hive.HiveQueryRunner$Builder.populateData(HiveQueryRunner.java:273)
	at io.trino.plugin.hive.HiveQueryRunner$Builder.build(HiveQueryRunner.java:258)
	at io.trino.faulttolerant.hive.TestHiveFaultTolerantExecutionJoinQueries.createQueryRunner(TestHiveFaultTolerantExecutionJoinQueries.java:53)
	at io.trino.testing.AbstractTestFaultTolerantExecutionJoinQueries.createQueryRunner(AbstractTestFaultTolerantExecutionJoinQueries.java:25)
	at io.trino.testing.AbstractTestQueryFramework.init(AbstractTestQueryFramework.java:95)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:515)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:217)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:144)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:169)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
	Suppressed: java.lang.Exception: SQL: CREATE TABLE IF NOT EXISTS lineitem AS SELECT * FROM tpch.tiny.lineitem
		at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:483)
		... 21 more

@findepi findepi mentioned this issue Sep 23, 2022
Merged
@arhimondr
Copy link
Contributor

Closing as a duplicate of #13946

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working test
Milestone
No milestone
Development

No branches or pull requests
3 participants
@findepi @arhimondr @ebyhr