-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[R] Error: package or namespace load failed for ‘arrow’ in dyn.load(file, DLLpath = DLLpath, ...): #39206
Comments
FWIW, I'm encountering the same with versions newer than 13.0.0.1 14.0.0.2 (fails)install.packages('arrow'); library('arrow')
#> Installing package into '/Users/psadil/git/arrow/renv/library/R-4.2/aarch64-apple-darwin20'
#> (as 'lib' is unspecified)
#>
#> The downloaded binary packages are in
#> /var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//Rtmp9NnRRv/downloaded_packages
#> Error: package or namespace load failed for 'arrow' in dyn.load(file, DLLpath = DLLpath, ...):
#> unable to load shared object '/Users/psadil/git/arrow/renv/library/R-4.2/aarch64-apple-darwin20/arrow/libs/arrow.so':
#> dlopen(/Users/psadil/git/arrow/renv/library/R-4.2/aarch64-apple-darwin20/arrow/libs/arrow.so, 0x0006): symbol not found in flat namespace '__ZN5arrow12ArrayBuilder12AppendScalarERKNS_6ScalarEx' Created on 2023-12-12 with reprex v2.0.2 14.0.0.1 (fails)remotes::install_version("arrow", "14.0.0.1")
#> Downloading package from url: https://packagemanager.posit.co/cran/latest/src/contrib/Archive/arrow/arrow_14.0.0.1.tar.gz
#> Installing package into '/Users/psadil/git/arrow/renv/library/R-4.2/aarch64-apple-darwin20'
#> (as 'lib' is unspecified)
#> Warning in i.p(...): installation of package
#> '/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpFv7S6a/remotes13556236975bd/arrow'
#> had non-zero exit status Longer Install Log> remotes::install_version('arrow', "14.0.0.1")
Downloading package from url: https://packagemanager.posit.co/cran/latest/src/contrib/Archive/arrow/arrow_14.0.0.1.tar.gz
Installing package into ‘/Users/psadil/git/arrow/renv/library/R-4.2/aarch64-apple-darwin20’
(as ‘lib’ is unspecified)
* installing *source* package ‘arrow’ ...
** package ‘arrow’ successfully unpacked and MD5 sums checked
** using staged installation
*** OpenSSL not found
*** Unable to find libcurl and openssl
*** Found local C++ source: 'tools/cpp'
*** Building libarrow from source
For build options and troubleshooting, see the install guide:
https://arrow.apache.org/docs/r/articles/install.html
**** cmake
**** pkg-config not installed, setting ARROW_DEPENDENCY_SOURCE=BUNDLED
**** S3/GCS support requires libcurl-devel (rpm) or libcurl4-openssl-dev (deb) ; building with ARROW_S3=OFF and ARROW_GCS=OFF
**** arrow
**** Error building Arrow C++.
**** Printing contents of build log because the build failed while ARROW_R_DEV was set to FALSE
++ pwd
+ : /private/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T/RtmpaBXKm8/remotes1351f3d741409/arrow
+ : tools/cpp
+ : /var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpjgVixX/file138d4f47f4aa
+ : libarrow/arrow-14.0.0
+ : /var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpjgVixX/file138d4449acdf6/cmake-3.26.4-macos-universal/bin/cmake
++ cd tools/cpp
++ pwd
+ SOURCE_DIR=/private/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T/RtmpaBXKm8/remotes1351f3d741409/arrow/tools/cpp
++ mkdir -p libarrow/arrow-14.0.0
++ cd libarrow/arrow-14.0.0
++ pwd
+ DEST_DIR=/private/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T/RtmpaBXKm8/remotes1351f3d741409/arrow/libarrow/arrow-14.0.0
+ '[' 2 = '' ']'
+ '[' false '!=' '' ']'
++ echo false
++ tr '[:upper:]' '[:lower:]'
+ LIBARROW_MINIMAL=false
+ '[' false = false ']'
+ ARROW_DEFAULT_PARAM=ON
+ case "$CXX" in
+ mkdir -p /var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpjgVixX/file138d4f47f4aa
+ pushd /var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpjgVixX/file138d4f47f4aa
/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T/RtmpjgVixX/file138d4f47f4aa /private/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T/RtmpaBXKm8/remotes1351f3d741409/arrow
+ /var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpjgVixX/file138d4449acdf6/cmake-3.26.4-macos-universal/bin/cmake -DARROW_BOOST_USE_SHARED=OFF -DARROW_BUILD_TESTS=OFF -DARROW_BUILD_SHARED=OFF -DARROW_BUILD_STATIC=ON -DARROW_ACERO=ON -DARROW_COMPUTE=ON -DARROW_CSV=ON -DARROW_DATASET=ON -DARROW_DEPENDENCY_SOURCE=BUNDLED -DAWSSDK_SOURCE= -DARROW_FILESYSTEM=ON -DARROW_GCS=OFF -DARROW_JEMALLOC=ON -DARROW_MIMALLOC=ON -DARROW_JSON=ON -DARROW_PARQUET=ON -DARROW_S3=OFF -DARROW_WITH_BROTLI=ON -DARROW_WITH_BZ2=ON -DARROW_WITH_LZ4=ON -DARROW_WITH_RE2=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_UTF8PROC=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON -DARROW_VERBOSE_THIRDPARTY_BUILD=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_FIND_DEBUG_MODE=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_INSTALL_PREFIX=/private/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T/RtmpaBXKm8/remotes1351f3d741409/arrow/libarrow/arrow-14.0.0 -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON -DCMAKE_UNITY_BUILD=OFF -Dxsimd_SOURCE= -Dzstd_SOURCE= -G 'Unix Makefiles' /private/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T/RtmpaBXKm8/remotes1351f3d741409/arrow/tools/cpp
inst/build_arrow_static.sh: line 67: /var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpjgVixX/file138d4449acdf6/cmake-3.26.4-macos-universal/bin/cmake: No such file or directory
**** Complete build log may still be present at /var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpjgVixX/file138d4584a69f5.log
Warning messages:
1: In rep(yes, length.out = len) : 'x' is NULL so the result will be NULL
2: In system(cmake_cmd, ignore.stdout = TRUE, ignore.stderr = TRUE) :
error in running command
------------------------- NOTE ---------------------------
There was an issue preparing the Arrow C++ libraries.
See https://arrow.apache.org/docs/r/articles/install.html
----------------------------------------------------------
Test compile error: <stdin>:1:10: fatal error: 'arrow/api.h' file not found
#include <arrow/api.h>
^~~~~~~~~~~~~
1 error generated.
Failing compile command: clang++ -arch arm64 -E -I/opt/R/arm64/include -falign-functions=64 -Wall -g -O2 -std=gnu++17 -xc++ -
PKG_CFLAGS=
PKG_LIBS=
ERROR: configuration failed for package ‘arrow’
* removing ‘/Users/psadil/git/arrow/renv/library/R-4.2/aarch64-apple-darwin20/arrow’
* restoring previous ‘/Users/psadil/git/arrow/renv/library/R-4.2/aarch64-apple-darwin20/arrow’
Warning message:
In i.p(...) :
installation of package ‘/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpaBXKm8/remotes1351f3d741409/arrow’ had non-zero exit status Created on 2023-12-12 with reprex v2.0.2 13.0.0.1 (works)remotes::install_version('arrow', "13.0.0.1")
#> Downloading package from url: https://packagemanager.posit.co/cran/latest/src/contrib/Archive/arrow/arrow_13.0.0.1.tar.gz
#> Installing package into '/Users/psadil/git/arrow/renv/library/R-4.2/aarch64-apple-darwin20'
#> (as 'lib' is unspecified) Created on 2023-12-12 with reprex v2.0.2 Session infosessionInfo()
#> R version 4.2.3 (2023-03-15)
#> Platform: aarch64-apple-darwin20 (64-bit)
#> Running under: macOS 14.1.1
#>
#> Matrix products: default
#> BLAS: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRblas.0.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib
#>
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> loaded via a namespace (and not attached):
#> [1] digest_0.6.33 withr_2.5.2 R.methodsS3_1.8.2 lifecycle_1.0.4
#> [5] magrittr_2.0.3 reprex_2.0.2 evaluate_0.23 rlang_1.1.2
#> [9] cli_3.6.2 remotes_2.4.2.1 rstudioapi_0.15.0 fs_1.6.3
#> [13] R.utils_2.12.3 R.oo_1.25.0 vctrs_0.6.5 styler_1.10.2
#> [17] rmarkdown_2.25 tools_4.2.3 R.cache_0.16.0 glue_1.6.2
#> [21] purrr_1.0.2 xfun_0.41 yaml_2.3.8 fastmap_1.1.1
#> [25] compiler_4.2.3 htmltools_0.5.7 knitr_1.45 |
@psadil using the package version 13.0.0.1 worked for me as well! It did the trick! I don't know how I did not think about installing an other version before, probably the killing overload before the christmas break killed me ! 🤦♂️ 😶🌫️ Thanks for the quick response, sharing and support @psadil !! However, I will let the issue open to get the apache-arrow support team be aware about the issue with the package version > 13.0.0.1! |
hm this reads like maybe the precompiled binaries where build without s3/gcs but the package was (correctly) build with them enabled? I don't have a mac so I will have to look at the binary but maybe @paleolimbot can test it? (which we did prior to release so it's weird that it's not working now...) |
For the original source build in the post, I see you are building from R installed via Homebrew. I don't think we test or support that...perhaps R installed from the official CRAN page will work? (Even though we don't test or support installation via Homebrew, this error may point to something else that is incorrect about our installation process on MacOS, so thank you for reporting!). I suspect that in your case the linked libraries are different than those by CRAN-provided R, so I'm not sure our precompiled binaries will work (although a source build should). For the On one of my MacOS M1s I get a somewhat concerning: (edit: This was because Arrow was already loaded in my session...restarting R and then installing rectified the error)
On my other MacOS M1, I don't have any issues. Those are both on R 4.3.1 (i.e., "release"). |
I can reproduce @psadil's error by installing from Posit package manager:
Perhaps since you're using renv that's where the binary is coming from? |
I was working with
CRAN <- "https://cloud.r-project.org/"
PPM <- "https://packagemanager.posit.co/cran/latest"
# fails -----
install.packages("arrow", repos = PPM)
#> Installing package into '/Users/psadil/git/arrow/renv/library/R-4.3/aarch64-apple-darwin20'
#> (as 'lib' is unspecified)
#>
#> The downloaded binary packages are in
#> /var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpyLusWt/downloaded_packages
callr::r(\(x) library("arrow"))
#> Error: ! in callr subprocess.
#> Caused by error:
#> ! package or namespace load failed for ‘arrow’ in dyn.load(file, DLLpath = DLLpath, ...):
#> unable to load shared object '/Users/psadil/git/arrow/renv/library/R-4.3/aarch64-apple-darwin20/arrow/libs/arrow.so':
#> dlopen(/Users/psadil/git/arrow/renv/library/R-4.3/aarch64-apple-darwin20/arrow/libs/arrow.so, 0x0006): symbol not found in flat namespace '__ZN5arrow12ArrayBuilder12AppendScalarERKNS_6ScalarEx'
# fails -----
remotes::install_version("arrow", "14.0.0.2", repos = PPM)
#> Downloading package from url: https://packagemanager.posit.co/cran/latest/src/contrib/arrow_14.0.0.2.tar.gz
#> Installing package into '/Users/psadil/git/arrow/renv/library/R-4.3/aarch64-apple-darwin20'
#> (as 'lib' is unspecified)
#> Warning in i.p(...): installation of package
#> '/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpyLusWt/remotes1cf77b6510ae/arrow'
#> had non-zero exit status
remotes::install_version("arrow", "14.0.0.2", repos = CRAN)
#> Downloading package from url: https://cloud.r-project.org/src/contrib/arrow_14.0.0.2.tar.gz
#> Installing package into '/Users/psadil/git/arrow/renv/library/R-4.3/aarch64-apple-darwin20'
#> (as 'lib' is unspecified)
#> Warning in i.p(...): installation of package
#> '/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpyLusWt/remotes1cf7134a78d1/arrow'
#> had non-zero exit status
# succeeds -----
install.packages("arrow", repos = CRAN)
#> Installing package into '/Users/psadil/git/arrow/renv/library/R-4.3/aarch64-apple-darwin20'
#> (as 'lib' is unspecified)
#>
#> The downloaded binary packages are in
#> /var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T//RtmpyLusWt/downloaded_packages
callr::r(\(x) library("arrow"))
#> [1] "arrow" "stats" "graphics" "grDevices" "datasets" "utils"
#> [7] "methods" "base"
remotes::install_version("arrow", "13.0.0.1", repos = PPM)
#> Downloading package from url: https://packagemanager.posit.co/cran/latest/src/contrib/Archive/arrow/arrow_13.0.0.1.tar.gz
#> Installing package into '/Users/psadil/git/arrow/renv/library/R-4.3/aarch64-apple-darwin20'
#> (as 'lib' is unspecified) Created on 2023-12-13 with reprex v2.0.2 Session infosessionInfo()
#> R version 4.3.2 (2023-10-31)
#> Platform: aarch64-apple-darwin20 (64-bit)
#> Running under: macOS Sonoma 14.1.1
#>
#> Matrix products: default
#> BLAS: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRblas.0.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.11.0
#>
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#>
#> time zone: America/New_York
#> tzcode source: internal
#>
#> attached base packages:
#> [1] stats graphics grDevices datasets utils methods base
#>
#> loaded via a namespace (and not attached):
#> [1] vctrs_0.6.5 cli_3.6.2 knitr_1.45 rlang_1.1.2
#> [5] xfun_0.41 processx_3.8.3 purrr_1.0.2 styler_1.10.2
#> [9] renv_1.0.3 glue_1.6.2 htmltools_0.5.7 ps_1.7.5
#> [13] rmarkdown_2.25 R.cache_0.16.0 evaluate_0.23 fastmap_1.1.1
#> [17] yaml_2.3.8 lifecycle_1.0.4 compiler_4.3.2 fs_1.6.3
#> [21] rstudioapi_0.15.0 R.oo_1.25.0 R.utils_2.12.3 digest_0.6.33
#> [25] R6_2.5.1 reprex_2.0.2 callr_3.7.3 magrittr_2.0.3
#> [29] R.methodsS3_1.8.2 tools_4.3.2 withr_2.5.2 remotes_2.4.2.1 |
I can repro on github actions: |
The non-working PPM binary is caused by the issue reported by Jeroen in #38855 where the host system is used to download the pre-compiled binary which is of course not suiteable for the target system (mac):
|
The failing source build is likely #38777 which I still have to investigate |
I've been running into this issue on my M2 Mac as well when I use renv, and have been using 13.0.0.1 for the time being. Just wanted to confirm that I was able to install and use 14.0.0.2 using CRAN. In the console I ran:
and the package loaded normally afterwards. |
We believe we've now fixed this for arrow 14.0.0.2 in Package Manager. Older arrow versions with the issue haven't been rebuilt yet, but we'll get those fixed soon as well. @sj-io If you're able to reinstall that arrow binary from Package Manager, could you provide the output of running |
@glin I believe it's working. I removed the package and uninstalled/reinstalled a couple different ways (including using
When I load it outside of a project with renv it lists "Repository: CRAN". |
@sj-io Thanks for confirming that. We've also now rebuilt arrow with the fix for 14.0.0 and 14.0.0.1, so installing these older versions should work as well. |
Sorry, but I still get errors (I think is the same as before). Am I doing something wrong? I tried uninstalled previous version before installing the new one, but with no luck either.
|
I think there are two different issues in this thread, one about installing arrow from source on Homebrew R, and one about installing the precompiled arrow package binaries from Posit Package Manager (https://packagemanager.posit.co/cran/latest). We've addressed the issues with installing precompiled arrow from Posit Package Manager, but I don't think there was a resolution with installing arrow from Homebrew R. I'm not sure what the Homebrew issue may be, but @juanfcocontreras you can try installing arrow from https://packagemanager.posit.co/cran/latest instead of https://cran.rstudio.com/. CRAN binary packages on macOS typically aren't compatible with Homebrew R, but the Package Manager binary packages may happen to work on Homebrew R. |
Hello @glin , Thank you very much for your detailed reply, I wasn't aware of it, until now (and until version 13 of arrow) everything was working correctly from CRAN. I just tried to install from packagemanager.postit:
But I'm still getting errors. Did I do something wrong or it just doesn't work from pacakgemanager either?
|
@juanfcocontreras Sorry, I forgot that Homebrew R has binary package installation disabled entirely, not just from CRAN, so trying to install binaries from Posit Package Manager won't work out of the box either. That is still attempting a source installation where R tries to compile arrow, rather than installing the precompiled binary from Package Manager. I'd recommend switching to the R installation from CRAN if possible, rather than Homebrew, as you'll have a much better package installation experience in general, being able to use all of CRAN's precompiled packages. If that's not feasible, I think it may still be possible to force a binary package installation on Homebrew R, but it's not straightforward. You can install directly from the binary files using: install.packages("https://packagemanager.posit.co/cran/latest/bin/macosx/big-sur-arm64/contrib/4.3/arrow_14.0.0.2.tgz", repos = NULL) Where you can find the binary package file URL from the Package Manager website, for a given R version and macOS arch (x86 or arm64): https://packagemanager.posit.co/client/#/repos/cran/packages/overview?search=arrow. This example is for R 4.3, macOS arm64. This method won't install dependencies by default, so you'll have to individually install each dependency manually. For why Homebrew R isn't compatible with CRAN, here is some background: https://stat.ethz.ch/pipermail/r-sig-mac/2016-March/011890.html |
@juanfcocontreras I am going to close this issue as the actual matter is closed (Thanks for that and the brew help too @glin!) As @glin said homebrew R is not recommended for arrow specifically and also overall ;) If you want the ease of a packagemanager to update and switch R versions I would highly recommend https://github.com/r-libs/rig! You can of course open another issue if the problems persists after following the suggestions :) |
Just as a side note, while Homebrew's R formula has the issues mentioned above, the Cask version works great (it's just the CRAN So while this gives you a problematic version of R:
This gives you the CRAN version of R: brew install --cask r And comes with the benefit that Homebrew will automatically upgrade it for you when you run |
Thank you so much for the suggestion! It's working great!!
|
I have had success with the --cask method from @juanfcocontreras and @amoeba. Thank you. I did have to remove the previous "formula"-based homebrew install of R before arrow would properly install. After that it worked great. |
Describe the bug, including details regarding any error messages, version, and platform.
Hi there - I am trying to install the R package arrow on my Mac OS M1.
I have tried to re-install the apache-arrow library with home brew and multiple other solution I read from different github and stackoverflow issue but it still does not solve the issue.
Could you help me out there?
Here is the log below:
Component(s)
R
The text was updated successfully, but these errors were encountered: