Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot run K-means and PCA algorithm with oap-mllib on Google Dataproc #122

Closed
HongW2019 opened this issue Aug 16, 2021 · 1 comment · Fixed by #124
Closed

Cannot run K-means and PCA algorithm with oap-mllib on Google Dataproc #122

HongW2019 opened this issue Aug 16, 2021 · 1 comment · Fixed by #124
Labels
bug Something isn't working

Comments

@HongW2019
Copy link
Contributor

With the conda package built on branch-1.2 latest commit, we cannot run K-means and PCA algorithm with oap-mllib on Google Dataproc.

21/08/13 08:32:37 INFO org.sparkproject.jetty.util.log: Logging initialized @2321ms to org.sparkproject.jetty.util.log.Slf4jLog
21/08/13 08:32:37 INFO org.sparkproject.jetty.server.Server: jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 1.8.0_292-b10
21/08/13 08:32:37 INFO org.sparkproject.jetty.server.Server: Started @2462ms
21/08/13 08:32:37 INFO org.sparkproject.jetty.server.AbstractConnector: Started ServerConnector@76911385{HTTP/1.1, (http/1.1)}{0.0.0.0:42875}
21/08/13 08:32:38 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at cluster-bigdata-m/10.128.15.226:8032
21/08/13 08:32:38 INFO org.apache.hadoop.yarn.client.AHSProxy: Connecting to Application History server at cluster-bigdata-m/10.128.15.226:10200
21/08/13 08:32:38 INFO org.apache.hadoop.conf.Configuration: resource-types.xml not found
21/08/13 08:32:38 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'.
21/08/13 08:32:39 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1628832281795_0010
21/08/13 08:32:40 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at cluster-bigdata-m/10.128.15.226:8030
21/08/13 08:32:42 INFO com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl: Ignoring exception of type GoogleJsonResponseException; verified object already exists with desired state.
21/08/13 08:32:46 INFO org.apache.hadoop.mapred.FileInputFormat: Total input files to process : 8
21/08/13 08:35:25 INFO util.OneDAL: Processing partitions with 2 executors
21/08/13 08:36:47 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 5.0 in stage 7.0 (TID 535) (cluster-bigdata-w-1.c.mlp-das-bdf-oap-11228062.internal executor 3): java.lang.UnsatisfiedLinkError: com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.dInit(JI)J
	at com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.dInit(Native Method)
	at com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.initHomogenNumericTable(Unknown Source)
	at com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.<init>(Unknown Source)
	at com.intel.daal.data_management.data.HomogenNumericTable.<init>(Unknown Source)
	at com.intel.daal.data_management.data.Matrix.<init>(Unknown Source)
	at org.apache.spark.ml.util.OneDAL$.vectorsToDenseNumericTable(OneDAL.scala:373)
	at org.apache.spark.ml.util.OneDAL$.$anonfun$rddVectorToMergedTables$3(OneDAL.scala:445)
	at org.apache.spark.ml.util.OneDAL$.$anonfun$rddVectorToMergedTables$3$adapted(OneDAL.scala:437)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
	at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
	at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
	at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1423)
	at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1350)
	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1414)
	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1237)
	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

21/08/13 08:36:47 ERROR org.apache.spark.scheduler.TaskSetManager: Task 10 in stage 7.0 failed 4 times; aborting job
21/08/13 08:36:47 INFO org.sparkproject.jetty.server.AbstractConnector: Stopped Spark@76911385{HTTP/1.1, (http/1.1)}{0.0.0.0:0}
21/08/13 08:36:47 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 7.0 in stage 7.0 (TID 537) (cluster-bigdata-w-1.c.mlp-das-bdf-oap-11228062.internal executor 1): TaskKilled (Stage cancelled)
21/08/13 08:36:47 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 3.0 in stage 7.0 (TID 533) (cluster-bigdata-w-1.c.mlp-das-bdf-oap-11228062.internal executor 1): TaskKilled (Stage cancelled)
21/08/13 08:36:47 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 9.2 in stage 7.0 (TID 550) (cluster-bigdata-w-0.c.mlp-das-bdf-oap-11228062.internal executor 2): TaskKilled (Stage cancelled)
21/08/13 08:36:47 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 4.0 in stage 7.0 (TID 534) (cluster-bigdata-w-0.c.mlp-das-bdf-oap-11228062.internal executor 2): TaskKilled (Stage cancelled)
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 10 in stage 7.0 failed 4 times, most recent failure: Lost task 10.3 in stage 7.0 (TID 553) (cluster-bigdata-w-0.c.mlp-das-bdf-oap-11228062.internal executor 4): java.lang.UnsatisfiedLinkError: com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.dInit(JI)J
	at com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.dInit(Native Method)
	at com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.initHomogenNumericTable(Unknown Source)
	at com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.<init>(Unknown Source)
	at com.intel.daal.data_management.data.HomogenNumericTable.<init>(Unknown Source)
	at com.intel.daal.data_management.data.Matrix.<init>(Unknown Source)
	at org.apache.spark.ml.util.OneDAL$.vectorsToDenseNumericTable(OneDAL.scala:373)
	at org.apache.spark.ml.util.OneDAL$.$anonfun$rddVectorToMergedTables$3(OneDAL.scala:445)
	at org.apache.spark.ml.util.OneDAL$.$anonfun$rddVectorToMergedTables$3$adapted(OneDAL.scala:437)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
	at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
	at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
	at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1423)
	at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1350)
	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1414)
	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1237)
	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2259)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2208)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2207)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2207)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1079)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1079)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1079)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2446)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2388)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2377)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:868)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2196)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2217)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2236)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2261)
	at org.apache.spark.rdd.RDD.count(RDD.scala:1253)
	at org.apache.spark.ml.util.OneDAL$.rddVectorToMergedTables(OneDAL.scala:449)
	at org.apache.spark.ml.feature.PCADALImpl.train(PCADALImpl.scala:39)
	at org.apache.spark.ml.feature.PCA.fit(PCA.scala:106)
	at com.intel.hibench.sparkbench.ml.PCAExample$.run(PCAExample.scala:71)
	at com.intel.hibench.sparkbench.ml.PCAExample$.main(PCAExample.scala:49)
	at com.intel.hibench.sparkbench.ml.PCAExample.main(PCAExample.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.UnsatisfiedLinkError: com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.dInit(JI)J
	at com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.dInit(Native Method)
	at com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.initHomogenNumericTable(Unknown Source)
	at com.intel.daal.data_management.data.HomogenNumericTableByteBufferImpl.<init>(Unknown Source)
	at com.intel.daal.data_management.data.HomogenNumericTable.<init>(Unknown Source)
	at com.intel.daal.data_management.data.Matrix.<init>(Unknown Source)
	at org.apache.spark.ml.util.OneDAL$.vectorsToDenseNumericTable(OneDAL.scala:373)
	at org.apache.spark.ml.util.OneDAL$.$anonfun$rddVectorToMergedTables$3(OneDAL.scala:445)
	at org.apache.spark.ml.util.OneDAL$.$anonfun$rddVectorToMergedTables$3$adapted(OneDAL.scala:437)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
	at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
	at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
	at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1423)
	at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1350)
	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1414)
	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1237)
	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML-123][Core] Improve locality handling for native lib loading xwu99/oap-mllib
3 participants
@xwu99 @HongW2019 @zhixingheyi-tian