Skip to content

perf opt

perf opt #102

Triggered via push October 27, 2023 19:25
Status Cancelled
Total duration 41m 45s
Artifacts 29
This run and associated checks have been archived and are scheduled for deletion. Learn more about checks retention

build_main.yml

on: push
Run  /  Check changes
42s
Run / Check changes
Run  /  Base image build
59s
Run / Base image build
Run  /  Breaking change detection with Buf (branch-3.5)
1m 10s
Run / Breaking change detection with Buf (branch-3.5)
Run  /  Run TPC-DS queries with SF=1
39m 9s
Run / Run TPC-DS queries with SF=1
Run  /  Run Docker integration tests
40m 42s
Run / Run Docker integration tests
Run  /  Run Spark on Kubernetes Integration test
38m 48s
Run / Run Spark on Kubernetes Integration test
Matrix: Run / build
Matrix: Run / java-other-versions
Run  /  Build modules: sparkr
0s
Run / Build modules: sparkr
Run  /  Linters, licenses, dependencies and documentation generation
27m 5s
Run / Linters, licenses, dependencies and documentation generation
Matrix: Run / pyspark
Fit to window
Zoom out
Zoom in

Annotations

57 errors and 10 warnings
Run / Linters, licenses, dependencies and documentation generation
Process completed with exit code 1.
Run / Build modules: pyspark-connect
Process completed with exit code 19.
Run / Build modules: pyspark-pandas-slow
The run was canceled by @ahshahid.
Run / Build modules: pyspark-pandas-connect-part0
The run was canceled by @ahshahid.
Run / Build modules: sparkr
The run was canceled by @ahshahid.
Run / Build modules: pyspark-pandas-connect-part1
The run was canceled by @ahshahid.
Run / Build modules: pyspark-pandas-connect-part1
The operation was canceled.
Run / Java 17 build with Maven
The run was canceled by @ahshahid.
Run / Java 17 build with Maven
The operation was canceled.
Run / Java 21 build with Maven
The run was canceled by @ahshahid.
Run / Java 21 build with Maven
The operation was canceled.
Run / Run Spark on Kubernetes Integration test
The run was canceled by @ahshahid.
Run / Run Spark on Kubernetes Integration test
The operation was canceled.
Run / Build modules: hive - slow tests
The run was canceled by @ahshahid.
Run / Build modules: hive - slow tests
The operation was canceled.
Run / Build modules: sql - other tests
The run was canceled by @ahshahid.
Run / Build modules: sql - other tests
The operation was canceled.
Run / Build modules: pyspark-core, pyspark-streaming
The run was canceled by @ahshahid.
Run / Build modules: pyspark-core, pyspark-streaming
The operation was canceled.
Run / Build modules: mllib-local,mllib
The run was canceled by @ahshahid.
Run / Build modules: mllib-local,mllib
The operation was canceled.
Run / Build modules: hive - other tests
The run was canceled by @ahshahid.
Run / Build modules: hive - other tests
The operation was canceled.
Run / Build modules: pyspark-pandas-connect-part2
The run was canceled by @ahshahid.
Run / Build modules: pyspark-pandas-connect-part2
The operation was canceled.
Run / Build modules: pyspark-mllib, pyspark-ml, pyspark-ml-connect
The operation was canceled.
Run / Build modules: sql - slow tests
The run was canceled by @ahshahid.
Run / Build modules: sql - slow tests
The operation was canceled.
Run / Build modules: sql - extended tests
The run was canceled by @ahshahid.
Run / Build modules: sql - extended tests
The operation was canceled.
Run / Build modules: api, catalyst, hive-thriftserver
The run was canceled by @ahshahid.
Run / Build modules: api, catalyst, hive-thriftserver
The operation was canceled.
Run / Build modules: pyspark-pandas
The run was canceled by @ahshahid.
Run / Build modules: pyspark-pandas
The operation was canceled.
Run / Run TPC-DS queries with SF=1
The run was canceled by @ahshahid.
Run / Run TPC-DS queries with SF=1
The operation was canceled.
Run / Build modules: pyspark-pandas-connect-part3
The run was canceled by @ahshahid.
Run / Build modules: pyspark-pandas-connect-part3
The operation was canceled.
Run / Run Docker integration tests
The run was canceled by @ahshahid.
Run / Run Docker integration tests
aborting artifact upload
Run / Run Docker integration tests
An error was encountered when uploading unit-tests-log-docker-integration--17-hadoop3-hive2.3. There were 1 items that failed to upload.
Run / Run Docker integration tests
The operation was canceled.
PredictorSuite.(It is not a test it is a sbt.testing.SuiteSelector): PredictorSuite#L30
java.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.
VectorSizeHintSuite.(It is not a test it is a sbt.testing.SuiteSelector): VectorSizeHintSuite#L27
java.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.
RWrapperUtilsSuite.(It is not a test it is a sbt.testing.SuiteSelector): RWrapperUtilsSuite#L24
java.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.
ALSStorageSuite.(It is not a test it is a sbt.testing.SuiteSelector): ALSStorageSuite#L1075
java.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.
LinearRegressionSuite.(It is not a test it is a sbt.testing.SuiteSelector): LinearRegressionSuite#L52
java.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.
KolmogorovSmirnovTestSuite.1 sample Kolmogorov-Smirnov test: apache commons math3 implementation equivalence: KolmogorovSmirnovTestSuite#L97
org.apache.spark.SparkException: Job 3 cancelled because SparkContext was shut down
KolmogorovSmirnovTestSuite.1 sample Kolmogorov-Smirnov test: R implementation equivalence: KolmogorovSmirnovTestSuite#L126
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at: org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1094) org.apache.spark.mllib.util.MLlibTestSparkContext.beforeAll(MLlibTestSparkContext.scala:40) org.apache.spark.mllib.util.MLlibTestSparkContext.beforeAll$(MLlibTestSparkContext.scala:36) org.apache.spark.ml.stat.KolmogorovSmirnovTestSuite.beforeAll(KolmogorovSmirnovTestSuite.scala:30) org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212) org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69) org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321) org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517) sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.java:840) The currently active SparkContext was created at: org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1094) org.apache.spark.mllib.util.MLlibTestSparkContext.beforeAll(MLlibTestSparkContext.scala:40) org.apache.spark.mllib.util.MLlibTestSparkContext.beforeAll$(MLlibTestSparkContext.scala:36) org.apache.spark.ml.stat.KolmogorovSmirnovTestSuite.beforeAll(KolmogorovSmirnovTestSuite.scala:30) org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212) org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69) org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321) org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517) sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.base/java.lang.Thread.run(Thread.java:840)
HypothesisTestSuite.(It is not a test it is a sbt.testing.SuiteSelector): HypothesisTestSuite#L33
java.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.
Run / Build modules: pyspark-errors
No files were found with the provided path: **/target/test-reports/*.xml. No artifacts will be uploaded.
Run / Build modules: pyspark-pandas-connect-part1
No files were found with the provided path: **/target/unit-tests.log. No artifacts will be uploaded.
Run / Build modules: pyspark-pandas-connect-part1
No files were found with the provided path: **/target/test-reports/*.xml. No artifacts will be uploaded.
Run / Build modules: pyspark-pandas-connect-part2
No files were found with the provided path: **/target/test-reports/*.xml. No artifacts will be uploaded.
Run / Build modules: pyspark-pandas-connect-part2
No files were found with the provided path: **/target/unit-tests.log. No artifacts will be uploaded.
Run / Build modules: sql - extended tests
No files were found with the provided path: **/target/test-reports/*.xml. No artifacts will be uploaded.
Run / Run TPC-DS queries with SF=1
No files were found with the provided path: **/target/test-reports/*.xml. No artifacts will be uploaded.
Run / Build modules: pyspark-pandas-connect-part3
No files were found with the provided path: **/target/unit-tests.log. No artifacts will be uploaded.
Run / Build modules: pyspark-pandas-connect-part3
No files were found with the provided path: **/target/test-reports/*.xml. No artifacts will be uploaded.
Run / Run Docker integration tests
Aborting upload for /home/runner/work/spark/spark/connector/docker-integration-tests/target/unit-tests.log due to failure

Artifacts

Produced during runtime
Name Size
spark-on-kubernetes-it-log Expired
170 KB
test-results-api, catalyst, hive-thriftserver--17-hadoop3-hive2.3 Expired
2.11 MB
test-results-core, unsafe, kvstore, avro, utils, network-common, network-shuffle, repl, launcher, examples, sketch, graphx--17-hadoop3-hive2.3 Expired
781 KB
test-results-docker-integration--17-hadoop3-hive2.3 Expired
72.3 KB
test-results-hive-- other tests-17-hadoop3-hive2.3 Expired
70.6 KB
test-results-hive-- slow tests-17-hadoop3-hive2.3 Expired
318 KB
test-results-mllib-local,mllib--17-hadoop3-hive2.3 Expired
591 KB
test-results-pyspark-connect--17-hadoop3-hive2.3 Expired
17.9 KB
test-results-pyspark-core, pyspark-streaming--17-hadoop3-hive2.3 Expired
27.5 KB
test-results-pyspark-mllib, pyspark-ml, pyspark-ml-connect--17-hadoop3-hive2.3 Expired
65.3 KB
test-results-pyspark-pandas--17-hadoop3-hive2.3 Expired
590 KB
test-results-sql-- other tests-17-hadoop3-hive2.3 Expired
379 KB
test-results-sql-- slow tests-17-hadoop3-hive2.3 Expired
315 KB
test-results-streaming, sql-kafka-0-10, streaming-kafka-0-10, yarn, kubernetes, hadoop-cloud, spark-ganglia-lgpl, connect, protobuf--17-hadoop3-hive2.3 Expired
41.5 KB
unit-tests-log-api, catalyst, hive-thriftserver--17-hadoop3-hive2.3 Expired
7.01 MB
unit-tests-log-core, unsafe, kvstore, avro, utils, network-common, network-shuffle, repl, launcher, examples, sketch, graphx--17-hadoop3-hive2.3 Expired
21.6 MB
unit-tests-log-docker-integration--17-hadoop3-hive2.3 Expired
3.43 MB
unit-tests-log-hive-- other tests-17-hadoop3-hive2.3 Expired
13.5 MB
unit-tests-log-hive-- slow tests-17-hadoop3-hive2.3 Expired
53.8 MB
unit-tests-log-mllib-local,mllib--17-hadoop3-hive2.3 Expired
155 MB
unit-tests-log-pyspark-connect--17-hadoop3-hive2.3 Expired
12 MB
unit-tests-log-pyspark-core, pyspark-streaming--17-hadoop3-hive2.3 Expired
15.9 MB
unit-tests-log-pyspark-mllib, pyspark-ml, pyspark-ml-connect--17-hadoop3-hive2.3 Expired
21.1 MB
unit-tests-log-pyspark-pandas--17-hadoop3-hive2.3 Expired
90.4 MB
unit-tests-log-sql-- extended tests-17-hadoop3-hive2.3 Expired
3.28 MB
unit-tests-log-sql-- other tests-17-hadoop3-hive2.3 Expired
19.2 MB
unit-tests-log-sql-- slow tests-17-hadoop3-hive2.3 Expired
112 MB
unit-tests-log-streaming, sql-kafka-0-10, streaming-kafka-0-10, yarn, kubernetes, hadoop-cloud, spark-ganglia-lgpl, connect, protobuf--17-hadoop3-hive2.3 Expired
17.1 MB
unit-tests-log-tpcds--17-hadoop3-hive2.3 Expired
6.42 MB