Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TensorFlow library not loading with GraalVM Native Image #67

Closed
murphye opened this issue May 11, 2020 · 13 comments
Closed

TensorFlow library not loading with GraalVM Native Image #67

murphye opened this issue May 11, 2020 · 13 comments
Labels
bug Something isn't working

Comments

@murphye
Copy link

murphye commented May 11, 2020

Description

I am working on an implementation of the covid19-detection example code, but using Quarkus to serve the model and also support GraalVM Native Image.

The project is located here:
https://github.com/murphye/djl-demo/tree/master/covid19-detection-quarkus

The application runs fine on the JVM, but when running in native mode, the TensorFlow libraries are not being loaded (i.e. System.loadLibrary).

Important: I cannot find a reference in the DJL code for System.loadLibrary and I do not understand how the TensorFlow libraries are actually loaded. If I better understood how the mechanism worked, I could better diagnose it. It does seem to be related to Bytedeco which I am not familiar with.

Here is the code that I am running to demonstrate the issue:

        LibUtils.loadLibrary(); // Forcing the library to load to demo error
        System.out.println("Library Path: " + System.getProperty("org.bytedeco.javacpp.platform.preloadpath"));

        // See if TF loaded correctly or not. If not, expect java.lang.UnsatisfiedLinkError
        TfEngine.getInstance().debugEnvironment();

Error Message

Here is the output of the error when running this code. The TF libraries are downloaded and placed in /Users/ermurphy/.tensorflow/cache/2.1.0-a-SNAPSHOT-cpu-osx-x86_64 correctly.

__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2020-05-11 19:22:10,136 INFO  [io.quarkus] (main) covid19-detection-quarkus 1.0-SNAPSHOT (powered by Quarkus 1.4.2.Final) started in 0.056s. Listening on: http://0.0.0.0:8080
2020-05-11 19:22:10,137 INFO  [io.quarkus] (main) Profile prod activated. 
2020-05-11 19:22:10,137 INFO  [io.quarkus] (main) Installed features: [cdi, resteasy, resteasy-jackson]
2020-05-11 19:22:23,584 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libstdc++.6.dylib ...
2020-05-11 19:22:24,752 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libjnitensorflow.dylib ...
2020-05-11 19:22:24,987 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libgcc_s.1.dylib ...
2020-05-11 19:22:25,203 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading THIRD_PARTY_TF_JNI_LICENSES ...
2020-05-11 19:22:25,448 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libtensorflow.2.dylib ...
2020-05-11 19:22:37,506 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libjnimklml.dylib ...
2020-05-11 19:22:37,656 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libiomp5.dylib ...
2020-05-11 19:22:38,182 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libmkldnn.0.dylib ...
2020-05-11 19:22:38,934 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading LICENSE ...
2020-05-11 19:22:39,038 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libmklml.dylib ...
2020-05-11 19:22:42,655 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libjnimkldnn.dylib ...
2020-05-11 19:22:42,833 INFO  [ai.djl.ten.eng.LibUtils] (executor-thread-1) Downloading libgomp.1.dylib ...
Library Path: /Users/ermurphy/.tensorflow/cache/2.1.0-a-SNAPSHOT-cpu-osx-x86_64
2020-05-11 19:22:42,979 INFO  [ai.djl.eng.Engine] (executor-thread-1) Engine name: TensorFlow
2020-05-11 19:22:42,980 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /predict failed, error id: 45121a3e-fbf8-4684-911c-4e9250ed8f41-1: java.lang.UnsatisfiedLinkError: org.tensorflow.internal.c_api.global.tensorflow.TF_Version()Lorg/bytedeco/javacpp/BytePointer; [symbol: Java_org_tensorflow_internal_c_1api_global_tensorflow_TF_1Version or Java_org_tensorflow_internal_c_1api_global_tensorflow_TF_1Version__]
        at com.oracle.svm.jni.access.JNINativeLinkage.getOrFindEntryPoint(JNINativeLinkage.java:145)
        at com.oracle.svm.jni.JNIGeneratedMethodSupport.nativeCallAddress(JNIGeneratedMethodSupport.java:57)
        at org.tensorflow.internal.c_api.global.tensorflow.TF_Version(tensorflow.java)
        at org.tensorflow.TensorFlow.version(TensorFlow.java:37)
        at ai.djl.tensorflow.engine.TfEngine.getVersion(TfEngine.java:64)
        at ai.djl.engine.Engine.debugEnvironment(Engine.java:171)
        at com.examples.ExampleService.<init>(ExampleService.java:42)

Expected Behavior

Running in GraalVM Native Image executable, the libaries should be loaded and usable through JNI bridge. I have proven this in the past with this PoC:
https://github.com/murphye/quarkus-tensorflow-inception/blob/master/src/main/java/io/quarkus/tensorflow/LoadTensorFlow.java

Next Steps

I need some guidance on how TensorFlow is loaded in DJL if it's not using System.loadLibrary as shown here:
https://github.com/murphye/quarkus-tensorflow-inception/blob/master/src/main/java/io/quarkus/tensorflow/LoadTensorFlow.java#L98

How else does the TensorFlow library get loaded, and how can I further diagnose the issue when running in Native mode?

@murphye murphye added the bug Something isn't working label May 11, 2020
@stu1130
Copy link
Contributor

stu1130 commented May 12, 2020

Thanks for reporting the issue @murphye
We don't explicitly call System.loadLibrary() as we load the TensorFlow native library via JavaCpp.
I am able to reproduce the error shown above, will take a look.

@frankfliu
Copy link
Contributor

@murphye
We are using javacpp to load TF native library, you can find more detail here: https://github.com/tensorflow/java

I noticed static initialization block is not executed in GraalVM Native Image. Both DJL and JavaCPP rely on static initialization code.

I'm not familiar with GraalVM, would you please help on:

  1. How can I add --initialize-at-run-time flag to GraalVM?
  2. How can I add system properties to native runner?
  3. HOw can I debug native runner?

@stu1130
Copy link
Contributor

stu1130 commented May 13, 2020

I tried to add --initialize-at-run-time in quarkus.native.additional-build-args and also in application.properties. It ends up showing the following error

* What went wrong:
Execution failed for task ':quarkusBuild'.
> io.quarkus.builder.BuildException: Build failure: Build failed due to errors
        [error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.RuntimeException: Failed to build native image
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:353)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:931)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at org.jboss.threads.JBossThread.run(JBossThread.java:479)
  Caused by: java.lang.RuntimeException: Image generation failed. Exit code: 1
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.imageGenerationFailed(NativeImageBuildStep.java:369)
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:339)
        ... 12 more

@murphye
Copy link
Author

murphye commented May 14, 2020

@frankfliu What classes (with packages) contain the static initialization block(s)? Can we offer alternative that doesn't do this?

--initialize-at-run-time is sort of a workaround, in my opinion. That is a reason why I created my own TensorFlow loader because TF had static initialization as well. This doesn't require --initialize-at-run-time
https://github.com/murphye/quarkus-tensorflow-inception/blob/master/src/main/java/io/quarkus/tensorflow/LoadTensorFlow.java

@murphye
Copy link
Author

murphye commented May 14, 2020

@stu1130 Can you turn on debug output to get additional error output? Whatever the equivalent is for Maven -X. I am not versed with Gradle.

@murphye
Copy link
Author

murphye commented May 14, 2020

@frankfliu

  1. See https://github.com/murphye/djl-demo/blob/master/covid19-detection-quarkus/src/main/resources/application.properties#L5 to where to add --initialize-at-run-time

  2. You can add System properties as shown here build/covid19-detection-quarkus-1.0-SNAPSHOT-runner -Dai.djl.repository.zoo.location=models/saved_model1

  3. I will find out from Quarkus team if there is support for native debugging yet. This was something that Oracle was only supporting with their EE version of GraalVM. There is support to turn on debug symbols, however:
    https://quarkus.io/guides/building-native-image#quarkus-native-pkg-native-config_quarkus.native.debug-symbols

Update: Native debug support will be in the next version of GraalVM CE

@murphye
Copy link
Author

murphye commented May 14, 2020

In searching through JavaCPP repo, here are static initialization blocks. For my error message, it stems from BytePointer

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/Loader.java#L90

https://github.com/bytedeco/javacpp/blob/734e30e8a2ae15539bb4af5a00aa649804ca0296/src/main/java/org/bytedeco/javacpp/indexer/Raw.java#L32

https://github.com/bytedeco/javacpp/blob/6db23a20d149599b9b2f07b47971a6c978319b8f/src/main/java/org/bytedeco/javacpp/indexer/UnsafeRaw.java#L36

Only Calling Loader.load()

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/BytePointer.java#L45

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/CLongPointer.java#L43

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/CharPointer.java#L38

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/DoublePointer.java#L38

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/FloatPointer.java#L38

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/IntPointer.java#L38

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/LongPointer.java#L38

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/ShortPointer.java#L38

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/ShortPointer.java#L38

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/SizeTPointer.java#L42

https://github.com/bytedeco/javacpp/blob/40560d7a3ddae99a805268ffb39e5e00b1a17f5c/src/main/java/org/bytedeco/javacpp/PointerPointer.java#L42

@stu1130
Copy link
Contributor

stu1130 commented May 14, 2020

@MarkAtwood I add -S to show full stack trace

* What went wrong:
Execution failed for task ':quarkusBuild'.
> io.quarkus.builder.BuildException: Build failure: Build failed due to errors
        [error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.RuntimeException: Failed to build native image
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:353)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:931)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at org.jboss.threads.JBossThread.run(JBossThread.java:479)
  Caused by: java.lang.RuntimeException: Image generation failed. Exit code: 1
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.imageGenerationFailed(NativeImageBuildStep.java:369)
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:339)
        ... 12 more


* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':quarkusBuild'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:205)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:263)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:203)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:184)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:114)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:62)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:372)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:359)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:352)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:338)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
        [error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.RuntimeException: Failed to build native image
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:353)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:931)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at org.jboss.threads.JBossThread.run(JBossThread.java:479)
Caused by: java.lang.RuntimeException: Image generation failed. Exit code: 1
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.imageGenerationFailed(NativeImageBuildStep.java:369)
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:339)
        ... 12 more

        at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:247)
        at io.quarkus.runner.bootstrap.AugmentActionImpl.createProductionApplication(AugmentActionImpl.java:87)
        at io.quarkus.gradle.tasks.QuarkusBuild.buildQuarkus(QuarkusBuild.java:79)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:49)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:42)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:727)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:694)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.run(ExecuteActionsTaskExecuter.java:568)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:402)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:394)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:92)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:553)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:536)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$300(ExecuteActionsTaskExecuter.java:109)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.executeWithPreviousOutputFiles(ExecuteActionsTaskExecuter.java:276)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:265)
        at org.gradle.internal.execution.steps.ExecuteStep.lambda$execute$1(ExecuteStep.java:33)
        at java.base/java.util.Optional.orElseGet(Optional.java:369)
        at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:33)
        at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:26)
        at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:67)
        at org.gradle.internal.execution.steps.CleanupOutputsStep.execute(CleanupOutputsStep.java:36)
        at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:49)
        at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:34)
        at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:43)
        at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:73)
        at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:54)
        at org.gradle.internal.execution.steps.CatchExceptionStep.execute(CatchExceptionStep.java:34)
        at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:44)
        at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:54)
        at org.gradle.internal.execution.steps.SnapshotOutputsStep.execute(SnapshotOutputsStep.java:38)
        at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:49)
        at org.gradle.internal.execution.steps.CacheStep.executeWithoutCache(CacheStep.java:159)
        at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:72)
        at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:43)
        at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:44)
        at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:33)
        at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:38)
        at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:24)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:92)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$0(SkipUpToDateStep.java:85)
        at java.base/java.util.Optional.map(Optional.java:265)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:39)
        at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:76)
        at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:37)
        at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:36)
        at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:26)
        at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:94)
        at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:49)
        at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:79)
        at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:53)
        at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:74)
        at org.gradle.internal.execution.steps.SkipEmptyWorkStep.lambda$execute$2(SkipEmptyWorkStep.java:78)
        at java.base/java.util.Optional.orElseGet(Optional.java:369)
        at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:78)
        at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:34)
        at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:39)
        at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:40)
        at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:28)
        at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:33)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:192)
        ... 33 more
Caused by: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
        [error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.RuntimeException: Failed to build native image
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:353)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:931)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at org.jboss.threads.JBossThread.run(JBossThread.java:479)
Caused by: java.lang.RuntimeException: Image generation failed. Exit code: 1
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.imageGenerationFailed(NativeImageBuildStep.java:369)
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:339)
        ... 12 more

        at io.quarkus.builder.Execution.run(Execution.java:115)
        at io.quarkus.builder.BuildExecutionBuilder.execute(BuildExecutionBuilder.java:79)
        at io.quarkus.deployment.QuarkusAugmentor.run(QuarkusAugmentor.java:156)
        at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:245)
        ... 104 more
Caused by: java.lang.RuntimeException: Failed to build native image
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:353)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:931)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at org.jboss.threads.JBossThread.run(JBossThread.java:479)
Caused by: java.lang.RuntimeException: Image generation failed. Exit code: 1
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.imageGenerationFailed(NativeImageBuildStep.java:369)
        at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:339)
        ... 12 more

and -d
output.log

@murphye
Copy link
Author

murphye commented May 14, 2020

@stu1130 Unfortunately your error message isn't very helpful. I opened a GitHub issue on that matter as it may be related to the Quarkus Gradle plugin: quarkusio/quarkus#9306

Can you put your current version of the code/config in GitHub so I can link to the exact commit version? That would be helpful.

@murphye
Copy link
Author

murphye commented May 14, 2020

@stu1130 @frankfliu I believe I have enough information to play around with --initialize-at-runtime by including all of the classes above that have the static initialization block. Theoretically that should overcome the issue, even though it's not ideal.

I believe the end goal should be to create a Quarkus extension for DJL that would provide a preconfigured DJL config for the developer.

@frankfliu
Copy link
Contributor

Here is one more class using static initializer, it comes from TF-java dependencies:
org.tensorflow.EagerSession

@murphye
Copy link
Author

murphye commented May 14, 2020

@stu1130 @frankfliu

Here is the format you need to follow:

--initialize-at-run-time=org.tensorflow.EagerSession\\,org.bytedeco.javacpp.Loader\\, (Add more as needed)

This is painful for the moment, but a better way can potentially be implemented for a Quarkus extension to handle this. For now, let's go with --initialize-at-run-time

@stu1130
Copy link
Contributor

stu1130 commented May 14, 2020

@murphye Let us know if you encounter other issues when using --initialize-at-run-time and feel free to reopen it if you have any other question

@stu1130 stu1130 closed this as completed May 15, 2020
@renovate renovate bot mentioned this issue May 24, 2021
1 task
Lokiiiiii pushed a commit to Lokiiiiii/djl that referenced this issue Oct 10, 2023
* [serving] Refactor ModelDefinition class

* [serving] Add async model loading on startup

* [serving] Start HTTP listener while model is loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants