Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

arrow dependency must put after arrow installation #248

Closed
weiting-chen opened this issue Apr 14, 2021 · 2 comments · Fixed by #259
Closed

arrow dependency must put after arrow installation #248

weiting-chen opened this issue Apr 14, 2021 · 2 comments · Fixed by #259
Labels
bug Something isn't working

Comments

@weiting-chen
Copy link
Collaborator

Found one issue in a clear environment.
It would report the below error:
[ERROR] Failed to execute goal on project spark-arrow-datasource-common: Could not resolve dependencies for project com.intel.oap:spark-arrow-datasource-common:jar:1.1.0: Failure to find org.apache.arrow:arrow-dataset:jar:3.0.0 in http://scala-tools.org/repo-releases was cached in the local repository

The root cause is the dependency check for arrow-dataset is under parent pom and before arrow installation.
We should move the arrow-dataset dependency check after arrow installation.

@zhouyuan
Copy link
Collaborator

there are also some redundant libs in our package

jar -tvf ~/nativesql_jars/spark-columnar-core-1.1.0-jar-with-dependencies.jar  | grep libarrow
26877488 Fri Apr 02 01:20:46 CST 2021 libarrow_dataset_jni.so.300.0.0
26877488 Fri Apr 02 01:20:46 CST 2021 libarrow_dataset_jni.so
26877488 Fri Apr 02 01:20:46 CST 2021 libarrow_dataset_jni.so.300
21190912 Fri Apr 02 01:22:28 CST 2021 libarrow.so.300
21190912 Fri Apr 02 01:22:28 CST 2021 libarrow.so

@zhouyuan zhouyuan added the bug Something isn't working label Apr 15, 2021
@weiting-chen
Copy link
Collaborator Author

there are also some redundant libs in our package

jar -tvf ~/nativesql_jars/spark-columnar-core-1.1.0-jar-with-dependencies.jar  | grep libarrow
26877488 Fri Apr 02 01:20:46 CST 2021 libarrow_dataset_jni.so.300.0.0
26877488 Fri Apr 02 01:20:46 CST 2021 libarrow_dataset_jni.so
26877488 Fri Apr 02 01:20:46 CST 2021 libarrow_dataset_jni.so.300
21190912 Fri Apr 02 01:22:28 CST 2021 libarrow.so.300
21190912 Fri Apr 02 01:22:28 CST 2021 libarrow.so

Thanks @zhouyuan , has submit this to another issue(#254 ) for tracking.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants