Flaky TestSecuredPinotIntegrationConnectorSmokeTest #14277

martint · 2022-09-24T16:02:20Z

Error:  io.trino.plugin.pinot.TestSecuredPinotIntegrationConnectorSmokeTest.testBrokerColumnMappingForSelectQueries  Time elapsed: 0.691 s  <<< FAILURE!
java.lang.AssertionError: Execution of 'actual' query failed: SELECT price, vendor FROM "SELECT price, vendor FROM my_table WHERE vendor != 'vendor7'"
	at org.testng.Assert.fail(Assert.java:83)
	at io.trino.testing.QueryAssertions.assertQuery(QueryAssertions.java:150)
	at io.trino.testing.QueryAssertions.assertQuery(QueryAssertions.java:106)
	at io.trino.testing.AbstractTestQueryFramework.assertQuery(AbstractTestQueryFramework.java:252)
	at io.trino.plugin.pinot.BasePinotIntegrationConnectorSmokeTest.testBrokerColumnMappingForSelectQueries(BasePinotIntegrationConnectorSmokeTest.java:1036)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.trino.testing.QueryFailedException: java.io.UncheckedIOException: Failed communicating with server: http://localhost:49179/tables/my_table/instances
	at io.trino.testing.AbstractTestingTrinoClient.execute(AbstractTestingTrinoClient.java:123)
	at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:480)
	at io.trino.testing.QueryAssertions.assertQuery(QueryAssertions.java:147)
	... 16 more
	Suppressed: java.lang.Exception: SQL: SELECT price, vendor FROM "SELECT price, vendor FROM my_table WHERE vendor != 'vendor7'"
		at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:483)
		... 17 more
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.io.UncheckedIOException: Failed communicating with server: http://localhost:49179/tables/my_table/instances
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2055)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4950)
	at com.google.common.cache.ForwardingLoadingCache.get(ForwardingLoadingCache.java:46)
	at io.trino.plugin.pinot.client.PinotClient.getBrokerHost(PinotClient.java:379)
	at io.trino.plugin.pinot.client.PinotClient.lambda$submitBrokerQueryJson$6(PinotClient.java:535)
	at io.trino.plugin.pinot.client.PinotClient.doWithRetries(PinotClient.java:621)
	at io.trino.plugin.pinot.client.PinotClient.submitBrokerQueryJson(PinotClient.java:533)
	at io.trino.plugin.pinot.client.PinotClient.createResultIterator(PinotClient.java:578)
	at io.trino.plugin.pinot.PinotBrokerPageSource.getNextPage(PinotBrokerPageSource.java:112)
	at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:311)
	at io.trino.operator.Driver.processInternal(Driver.java:411)
	at io.trino.operator.Driver.lambda$process$10(Driver.java:314)
	at io.trino.operator.Driver.tryWithLock(Driver.java:706)
	at io.trino.operator.Driver.process(Driver.java:306)
	at io.trino.operator.Driver.processForDuration(Driver.java:277)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:736)
	at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:164)
	at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:515)
	at io.trino.$gen.Trino_testversion____20220924_022540_1981.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.UncheckedIOException: Failed communicating with server: http://localhost:49179/tables/my_table/instances
	at io.airlift.http.client.ResponseHandlerUtils.propagate(ResponseHandlerUtils.java:22)
	at io.airlift.http.client.JsonResponseHandler.handleException(JsonResponseHandler.java:62)
	at io.airlift.http.client.jetty.JettyHttpClient.execute(JettyHttpClient.java:599)
	at io.trino.plugin.pinot.client.PinotClient.doHttpActionWithHeadersJson(PinotClient.java:198)
	at io.trino.plugin.pinot.client.PinotClient.sendHttpGetToControllerJson(PinotClient.java:220)
	at io.trino.plugin.pinot.client.PinotClient.getAllBrokersForTable(PinotClient.java:357)
	at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:169)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
	... 23 more
Caused by: java.io.EOFException: HttpConnectionOverHTTP@64483ed3::SocketChannelEndPoint@214abc3e{l=/127.0.0.1:42888,r=localhost/127.0.0.1:49179,ISHUT,fill=-,flush=-,to=67/300000}{io=0/0,kio=0,kro=1}->HttpConnectionOverHTTP@64483ed3(l:/127.0.0.1:42888 <-> r:localhost/127.0.0.1:49179,closed=false)=>HttpChannelOverHTTP@4d3be8f9(exchange=HttpExchange@1021b8c5{req=HttpRequest[GET /tables/my_table/instances HTTP/1.1]@35bab2b4[TERMINATED/null] res=HttpResponse[null 0 null]@41654371[PENDING/null]})[send=HttpSenderOverHTTP@538b0784(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@4ef17aa0{s=START}],recv=HttpReceiverOverHTTP@2ebe5ed8(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of -1}]]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:376)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1620)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:259)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:181)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:79)
	at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131)
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:172)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	... 1 more

[INFO] 
[INFO] Results:
[INFO] 
Error:  Failures: 
Error:    TestSecuredPinotIntegrationConnectorSmokeTest>BasePinotIntegrationConnectorSmokeTest.testBrokerColumnMappingForSelectQueries:1036->AbstractTestQueryFramework.assertQuery:252 Execution of 'actual' query failed: SELECT price, vendor FROM "SELECT price, vendor FROM my_table WHERE vendor != 'vendor7'"

The text was updated successfully, but these errors were encountered:

findepi · 2022-09-26T09:07:52Z

cc @elonazoulay @xiangfu0

xiangfu0 · 2022-09-26T10:01:53Z

Hmm, I tried but cannot reproduce it locally.

Is there a failed CI link? Or any environment are you on?

xiangfu0 · 2022-09-26T22:59:35Z

#14304

ebyhr · 2022-09-27T03:12:50Z

https://github.com/trinodb/trino/actions/runs/3127676861/jobs/5075173404

xiangfu0 · 2022-09-27T08:10:50Z

One round of 50 stress test passed with master branch: https://github.com/trinodb/trino/actions/runs/3132624061

xiangfu0 · 2022-09-27T23:41:31Z

Another 50 rounds stress test passed on master branch: https://github.com/trinodb/trino/actions/runs/3137939987

xiangfu0 · 2022-09-28T00:02:39Z

Found one flaky one: https://github.com/trinodb/trino/actions/runs/3137952024/jobs/5098915969

Error:  io.trino.plugin.pinot.TestSecuredPinotIntegrationConnectorSmokeTest.testMaxLimitForPassthroughQueries  Time elapsed: 0.41 s  <<< FAILURE!
java.lang.AssertionError: Execution of 'actual' query failed: SELECT string_col, updated_at_seconds  FROM  "SELECT updated_at_seconds, string_col FROM too_many_broker_rows  WHERE string_col != 'string_12'  LIMIT 12"
	at org.testng.Assert.fail(Assert.java:83)
	at io.trino.testing.QueryAssertions.assertQuery(QueryAssertions.java:150)
	at io.trino.testing.QueryAssertions.assertQuery(QueryAssertions.java:106)
	at io.trino.testing.AbstractTestQueryFramework.assertQuery(AbstractTestQueryFramework.java:252)
	at io.trino.plugin.pinot.BasePinotIntegrationConnectorSmokeTest.testMaxLimitForPassthroughQueries(BasePinotIntegrationConnectorSmokeTest.java:1323)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.trino.testing.QueryFailedException: java.io.UncheckedIOException: Failed communicating with server: http://localhost:49180/tables/too_many_broker_rows/instances
	at io.trino.testing.AbstractTestingTrinoClient.execute(AbstractTestingTrinoClient.java:123)
	at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:480)
	at io.trino.testing.QueryAssertions.assertQuery(QueryAssertions.java:147)
	... 16 more
	Suppressed: java.lang.Exception: SQL: SELECT string_col, updated_at_seconds  FROM  "SELECT updated_at_seconds, string_col FROM too_many_broker_rows  WHERE string_col != 'string_12'  LIMIT 12"
		at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:483)
		... 17 more
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.io.UncheckedIOException: Failed communicating with server: http://localhost:49180/tables/too_many_broker_rows/instances
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2055)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4950)
	at com.google.common.cache.ForwardingLoadingCache.get(ForwardingLoadingCache.java:46)
	at io.trino.plugin.pinot.client.PinotClient.getBrokerHost(PinotClient.java:379)
	at io.trino.plugin.pinot.client.PinotClient.lambda$submitBrokerQueryJson$6(PinotClient.java:535)
	at io.trino.plugin.pinot.client.PinotClient.doWithRetries(PinotClient.java:621)
	at io.trino.plugin.pinot.client.PinotClient.submitBrokerQueryJson(PinotClient.java:533)
	at io.trino.plugin.pinot.client.PinotClient.createResultIterator(PinotClient.java:578)
	at io.trino.plugin.pinot.PinotBrokerPageSource.getNextPage(PinotBrokerPageSource.java:112)
	at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:311)
	at io.trino.operator.Driver.processInternal(Driver.java:411)
	at io.trino.operator.Driver.lambda$process$10(Driver.java:314)
	at io.trino.operator.Driver.tryWithLock(Driver.java:706)
	at io.trino.operator.Driver.process(Driver.java:306)
	at io.trino.operator.Driver.processForDuration(Driver.java:277)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:736)
	at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:164)
	at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:515)
	at io.trino.$gen.Trino_testversion____20220927_234833_2169.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.UncheckedIOException: Failed communicating with server: http://localhost:49180/tables/too_many_broker_rows/instances
	at io.airlift.http.client.ResponseHandlerUtils.propagate(ResponseHandlerUtils.java:22)
	at io.airlift.http.client.JsonResponseHandler.handleException(JsonResponseHandler.java:62)
	at io.airlift.http.client.jetty.JettyHttpClient.execute(JettyHttpClient.java:599)
	at io.trino.plugin.pinot.client.PinotClient.doHttpActionWithHeadersJson(PinotClient.java:198)
	at io.trino.plugin.pinot.client.PinotClient.sendHttpGetToControllerJson(PinotClient.java:220)
	at io.trino.plugin.pinot.client.PinotClient.getAllBrokersForTable(PinotClient.java:357)
	at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:169)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
	... 23 more
Caused by: java.io.EOFException: HttpConnectionOverHTTP@14997ad3::SocketChannelEndPoint@564b8428{l=/127.0.0.1:44498,r=localhost/127.0.0.1:49180,ISHUT,fill=-,flush=-,to=0/300000}{io=0/0,kio=0,kro=1}->HttpConnectionOverHTTP@14997ad3(l:/127.0.0.1:44498 <-> r:localhost/127.0.0.1:49180,closed=false)=>HttpChannelOverHTTP@45d921a1(exchange=HttpExchange@183f64df{req=HttpRequest[GET /tables/too_many_broker_rows/instances HTTP/1.1]@472fcae9[TERMINATED/null] res=HttpResponse[null 0 null]@4d4f6ce0[PENDING/null]})[send=HttpSenderOverHTTP@419798c1(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@226d15b0{s=START}],recv=HttpReceiverOverHTTP@1b2cfb96(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of -1}]]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:376)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1620)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:259)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:181)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:79)
	at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131)
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:172)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	... 1 more

Seems that the http request may fail. Shall we add a default 3 retries?

xiangfu0 · 2022-10-19T23:14:12Z

I think this is ready: #14304

martint changed the title ~~Flaky TestSecuredPinotIntegrationConnectorSmokeTest.testBrokerColumnMappingForSelectQueries~~ Flaky TestSecuredPinotIntegrationConnectorSmokeTest Sep 24, 2022

xiangfu0 mentioned this issue Sep 26, 2022

Retry on Pinot Controller/Broker Rest calls and check data readiness for BasePinotIntegrationConnectorSmokeTest #14304

Closed

xiangfu0 mentioned this issue Oct 10, 2022

Flaky TestPinotWithoutAuthenticationIntegrationLatestVersionConnectorSmokeTest.init: Container startup failed: Timed out waiting for container port to open #14546

Closed

hashhar added the test label Oct 13, 2022

xiangfu0 self-assigned this Oct 19, 2022

hashhar mentioned this issue Nov 1, 2022

Minor cleanups #14839

Merged

losipiuk mentioned this issue May 9, 2023

Remove FixedCountNodeAllocatorService #17333

Merged

hashhar closed this as completed Sep 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flaky TestSecuredPinotIntegrationConnectorSmokeTest #14277

Flaky TestSecuredPinotIntegrationConnectorSmokeTest #14277

martint commented Sep 24, 2022

findepi commented Sep 26, 2022

xiangfu0 commented Sep 26, 2022

xiangfu0 commented Sep 26, 2022

ebyhr commented Sep 27, 2022

xiangfu0 commented Sep 27, 2022 •

edited

Loading

xiangfu0 commented Sep 27, 2022 •

edited

Loading

xiangfu0 commented Sep 28, 2022

xiangfu0 commented Oct 19, 2022

Flaky TestSecuredPinotIntegrationConnectorSmokeTest #14277

Flaky TestSecuredPinotIntegrationConnectorSmokeTest #14277

Comments

martint commented Sep 24, 2022

findepi commented Sep 26, 2022

xiangfu0 commented Sep 26, 2022

xiangfu0 commented Sep 26, 2022

ebyhr commented Sep 27, 2022

xiangfu0 commented Sep 27, 2022 • edited Loading

xiangfu0 commented Sep 27, 2022 • edited Loading

xiangfu0 commented Sep 28, 2022

xiangfu0 commented Oct 19, 2022

xiangfu0 commented Sep 27, 2022 •

edited

Loading

xiangfu0 commented Sep 27, 2022 •

edited

Loading