improve OpenMP detection on macOS #6034

kevinushey · 2024-03-29T17:42:21Z

Potential solution for #4348. Testing would be appreciated!

codecov · 2024-03-29T17:52:19Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.53%. Comparing base (566bff0) to head (3e970e0).
Report is 1 commits behind head on master.

❗ Current head 3e970e0 differs from pull request most recent head 603e778. Consider uploading reports for the commit 603e778 to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6034   +/-   ##
=======================================
  Coverage   97.53%   97.53%           
=======================================
  Files          80       80           
  Lines       14915    14916    +1     
=======================================
+ Hits        14547    14548    +1     
  Misses        368      368

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

MichaelChirico · 2024-03-29T18:06:13Z

Well on my Linux at least it works as expected :)

* checking if R installation supports OpenMP without any extra hints... yes

Anyone with a Mac able to test this? (I guess it has to be a newer Mac?)

Anirban166 · 2024-03-29T22:59:57Z

Great job! I did a devtools::load_all() with my working directory set to this branch of data.table and it seems to be working as it was able to detect the installation of the libomp brew bottle in my system:

ℹ Loading data.table
ℹ Re-compiling data.table (debug build)
── R CMD INSTALL ────────────────────────────────────────────────────────────────────────────────
─  installing *source* package ‘data.table’ ...
   ** using staged installation
   zlib 1.2.11 is available ok
─  checking if R installation supports OpenMP with "-Xclang -fopenmp" ... no
─  checking if libomp installation at /opt/homebrew/opt/libomp can be used... yes (941ms)
   ** libs
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include  -I/opt/homebrew/opt/libomp/include -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c assign.c -o assign.o
   In file included from assign.c:1:
   In file included from ./data.table.h:1:
   In file included from ./myomp.h:10:
   /opt/homebrew/opt/libomp/include/omp.h:54:9: warning: ISO C restricts enumerator values to range of 'int' (2147483648 is too large) [-Wpedantic]
           omp_sched_monotonic = 0x80000000
           ^                     ~~~~~~~~~~
   /opt/homebrew/opt/libomp/include/omp.h:409:7: warning: ISO C restricts enumerator values to range of 'int' (18446744073709551615 is too large) [-Wpedantic]
         KMP_ALLOCATOR_MAX_HANDLE = UINTPTR_MAX
         ^                          ~~~~~~~~~~~
   /opt/homebrew/opt/libomp/include/omp.h:425:7: warning: ISO C restricts enumerator values to range of 'int' (18446744073709551615 is too large) [-Wpedantic]
         KMP_MEMSPACE_MAX_HANDLE = UINTPTR_MAX
         ^                         ~~~~~~~~~~~
   /opt/homebrew/opt/libomp/include/omp.h:469:39: warning: ISO C restricts enumerator values to range of 'int' (18446744073709551615 is too large) [-Wpedantic]
       typedef enum omp_event_handle_t { KMP_EVENT_MAX_HANDLE = UINTPTR_MAX } omp_event_handle_t;
                                         ^                      ~~~~~~~~~~~
   4 warnings generated.

   ...

   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include  -I/opt/homebrew/opt/libomp/include -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c wrappers.c -o wrappers.o
   In file included from wrappers.c:1:
   In file included from ./data.table.h:1:
   In file included from ./myomp.h:10:
   /opt/homebrew/opt/libomp/include/omp.h:54:9: warning: ISO C restricts enumerator values to range of 'int' (2147483648 is too large) [-Wpedantic]
           omp_sched_monotonic = 0x80000000
           ^                     ~~~~~~~~~~
   /opt/homebrew/opt/libomp/include/omp.h:409:7: warning: ISO C restricts enumerator values to range of 'int' (18446744073709551615 is too large) [-Wpedantic]
         KMP_ALLOCATOR_MAX_HANDLE = UINTPTR_MAX
         ^                          ~~~~~~~~~~~
   /opt/homebrew/opt/libomp/include/omp.h:425:7: warning: ISO C restricts enumerator values to range of 'int' (18446744073709551615 is too large) [-Wpedantic]
         KMP_MEMSPACE_MAX_HANDLE = UINTPTR_MAX
         ^                         ~~~~~~~~~~~
   /opt/homebrew/opt/libomp/include/omp.h:469:39: warning: ISO C restricts enumerator values to range of 'int' (18446744073709551615 is too large) [-Wpedantic]
       typedef enum omp_event_handle_t { KMP_EVENT_MAX_HANDLE = UINTPTR_MAX } omp_event_handle_t;
                                         ^                      ~~~~~~~~~~~
   4 warnings generated.
   clang -arch arm64 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/opt/R/arm64/lib -o data.table.so assign.o between.o bmerge.o chmatch.o cj.o coalesce.o dogroups.o fastmean.o fcast.o fifelse.o fmelt.o forder.o frank.o fread.o freadR.o froll.o frollR.o frolladaptive.o fsort.o fwrite.o fwriteR.o gsumm.o idatetime.o ijoin.o init.o inrange.o nafill.o negate.o nqrecreateindices.o openmp-utils.o programming.o quickselect.o rbindlist.o reorder.o shift.o snprintf.o subset.o transpose.o types.o uniqlist.o utils.o vecseq.o wrappers.o -L/opt/homebrew/opt/libomp/lib -lomp -lz -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
   ld: warning: -undefined dynamic_lookup may not work with chained fixups
   PKG_CFLAGS = -I/opt/homebrew/opt/libomp/include -Xclang -fopenmp
   PKG_LIBS = -L/opt/homebrew/opt/libomp/lib -lomp -lz
   if [ "data.table.so" != "data_table.so" ]; then mv data.table.so data_table.so; fi
   if [ "" != "Windows_NT" ] && [ `uname -s` = 'Darwin' ]; then install_name_tool -id data_table.so data_table.so; fi
   installing to /private/var/folders/9_/qc989n050_d2sbtw92scjshr0000gn/T/RtmpUCKjHj/devtools_install_11d2035c30403/00LOCK-data.table/00new/data.table/libs
   ** checking absolute paths in shared objects and dynamic libraries
─  DONE (data.table)
> getDTthreads()
[1] 5

I truncated the console output above since there are a handful of C standard-oriented warnings in the compilation process that come as visible from the use of the -Wpedantic flag (all of them are to do with enumerator values in omp.h that exceed the integer range).

> sessionInfo()
R version 4.2.1 (2022-06-23)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Ventura 13.2.1

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.15.99 testthat_3.1.4    

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.9        compiler_4.2.1    later_1.3.0       urlchecker_1.0.1  profvis_0.3.7    
 [6] remotes_2.4.2     tools_4.2.1       digest_0.6.29     pkgbuild_1.4.3    pkgload_1.3.4    
[11] memoise_2.0.1     lifecycle_1.0.4   rlang_1.1.3       shiny_1.7.2       cli_3.6.2        
[16] rstudioapi_0.14   fastmap_1.1.0     withr_3.0.0       stringr_1.5.1     desc_1.4.3       
[21] fs_1.6.3          htmlwidgets_1.5.4 vctrs_0.6.5       devtools_2.4.4    rprojroot_2.0.4  
[26] glue_1.7.0        R6_2.5.1          processx_3.8.3    sessioninfo_1.2.2 purrr_1.0.2      
[31] callr_3.7.5       magrittr_2.0.3    promises_1.2.0.1  ps_1.7.6          ellipsis_0.3.2   
[36] htmltools_0.5.3   usethis_2.1.6     mime_0.12         xtable_1.8-4      httpuv_1.6.5     
[41] stringi_1.8.3     miniUI_0.1.1.1    cachem_1.0.6      brio_1.1.3

Definitely an improvement in terms of recognizing OpenMP support (wondering if the warnings above can be suppressed as well though, although it's tied to the OpenMP installation), as this was the case before:

ℹ Loading data.table
ℹ Re-compiling data.table (debug build)
── R CMD INSTALL ────────────────────────────────────────────────────────────────────────────────
─  installing *source* package ‘data.table’ ...
   ** using staged installation
   zlib 1.2.11 is available ok
─  checking if R installation supports OpenMP with "-Xclang -fopenmp" ... no
   no
   *** OpenMP not supported! data.table uses OpenMP to automatically
   ***   parallelize operations like sorting, grouping, file reading, etc.
   *** For details on how to install the necessary toolchains on your OS see:
   ***   https://github.com/Rdatatable/data.table/wiki/Installation
   *** Continuing installation without OpenMP support...
   ** libs
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c assign.c -o assign.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c between.c -o between.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c bmerge.c -o bmerge.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c chmatch.c -o chmatch.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c cj.c -o cj.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c coalesce.c -o coalesce.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c dogroups.c -o dogroups.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c fastmean.c -o fastmean.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c fcast.c -o fcast.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c fifelse.c -o fifelse.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c fmelt.c -o fmelt.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c forder.c -o forder.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c frank.c -o frank.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c fread.c -o fread.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c freadR.c -o freadR.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c froll.c -o froll.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c frollR.c -o frollR.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c frolladaptive.c -o frolladaptive.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c fsort.c -o fsort.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c fwrite.c -o fwrite.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c fwriteR.c -o fwriteR.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c gsumm.c -o gsumm.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c idatetime.c -o idatetime.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c ijoin.c -o ijoin.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c init.c -o init.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c inrange.c -o inrange.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c nafill.c -o nafill.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c negate.c -o negate.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c nqrecreateindices.c -o nqrecreateindices.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c openmp-utils.c -o openmp-utils.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c programming.c -o programming.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c quickselect.c -o quickselect.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c rbindlist.c -o rbindlist.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c reorder.c -o reorder.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c shift.c -o shift.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c snprintf.c -o snprintf.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c subset.c -o subset.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c transpose.c -o transpose.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c types.c -o types.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c uniqlist.c -o uniqlist.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c utils.c -o utils.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c vecseq.c -o vecseq.o
   clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c wrappers.c -o wrappers.o
   clang -arch arm64 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/opt/R/arm64/lib -o data.table.so assign.o between.o bmerge.o chmatch.o cj.o coalesce.o dogroups.o fastmean.o fcast.o fifelse.o fmelt.o forder.o frank.o fread.o freadR.o froll.o frollR.o frolladaptive.o fsort.o fwrite.o fwriteR.o gsumm.o idatetime.o ijoin.o init.o inrange.o nafill.o negate.o nqrecreateindices.o openmp-utils.o programming.o quickselect.o rbindlist.o reorder.o shift.o snprintf.o subset.o transpose.o types.o uniqlist.o utils.o vecseq.o wrappers.o -lz -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
   ld: warning: -undefined dynamic_lookup may not work with chained fixups
   PKG_CFLAGS =
   PKG_LIBS = -lz
   if [ "data.table.so" != "data_table.so" ]; then mv data.table.so data_table.so; fi
   if [ "" != "Windows_NT" ] && [ `uname -s` = 'Darwin' ]; then install_name_tool -id data_table.so data_table.so; fi
   installing to /private/var/folders/9_/qc989n050_d2sbtw92scjshr0000gn/T/RtmpIIhoRq/devtools_install_10e99696786f0/00LOCK-data.table/00new/data.table/libs
   ** checking absolute paths in shared objects and dynamic libraries
─  DONE (data.table)
> getDTthreads()
[1] 1

clang -fopenmp -v  
Homebrew clang version 16.0.4
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin

kevinushey · 2024-03-30T00:03:51Z

To add, here's what I see after installing the OpenMP tools from https://mac.r-project.org/openmp/:

kevin@MBP-P2MQ:~/r/pkg/data.table [macos-openmp-support]
$ R CMD INSTALL --preclean .
* installing to library ‘/Users/kevin/Library/R/arm64/4.3/library’
* installing *source* package ‘data.table’ ...
** using staged installation
zlib 1.2.11 is available ok
* checking if R installation supports OpenMP with "-Xclang -fopenmp" ... yes
** libs
using C compiler: ‘Apple clang version 15.0.0 (clang-1500.3.9.4)’
using SDK: ‘’
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c assign.c -o assign.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c between.c -o between.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c bmerge.c -o bmerge.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c chmatch.c -o chmatch.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c cj.c -o cj.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c coalesce.c -o coalesce.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c dogroups.c -o dogroups.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c fastmean.c -o fastmean.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c fcast.c -o fcast.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c fifelse.c -o fifelse.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c fmelt.c -o fmelt.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c forder.c -o forder.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c frank.c -o frank.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c fread.c -o fread.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c freadR.c -o freadR.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c froll.c -o froll.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c frollR.c -o frollR.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c frolladaptive.c -o frolladaptive.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c fsort.c -o fsort.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c fwrite.c -o fwrite.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c fwriteR.c -o fwriteR.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c gsumm.c -o gsumm.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c idatetime.c -o idatetime.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c ijoin.c -o ijoin.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c init.c -o init.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c inrange.c -o inrange.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c nafill.c -o nafill.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c negate.c -o negate.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c nqrecreateindices.c -o nqrecreateindices.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c openmp-utils.c -o openmp-utils.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c programming.c -o programming.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c quickselect.c -o quickselect.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c rbindlist.c -o rbindlist.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c reorder.c -o reorder.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c shift.c -o shift.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c snprintf.c -o snprintf.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c subset.c -o subset.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c transpose.c -o transpose.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c types.c -o types.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c uniqlist.c -o uniqlist.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c utils.c -o utils.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c vecseq.c -o vecseq.o
clang -arch arm64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/opt/R/arm64/include   -Xclang -fopenmp   -fPIC  -falign-functions=64 -Wall -g -O2  -c wrappers.c -o wrappers.o
clang -arch arm64 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -L/Library/Frameworks/R.framework/Resources/lib -L/opt/R/arm64/lib -o data.table.so assign.o between.o bmerge.o chmatch.o cj.o coalesce.o dogroups.o fastmean.o fcast.o fifelse.o fmelt.o forder.o frank.o fread.o freadR.o froll.o frollR.o frolladaptive.o fsort.o fwrite.o fwriteR.o gsumm.o idatetime.o ijoin.o init.o inrange.o nafill.o negate.o nqrecreateindices.o openmp-utils.o programming.o quickselect.o rbindlist.o reorder.o shift.o snprintf.o subset.o transpose.o types.o uniqlist.o utils.o vecseq.o wrappers.o -lomp -lz -L/opt/homebrew/opt/libomp/lib -lomp -L/opt/homebrew/lib -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: ignoring duplicate libraries: '-lomp'
PKG_CFLAGS = -Xclang -fopenmp
PKG_LIBS = -lomp -lz
if [ "data.table.so" != "data_table.so" ]; then mv data.table.so data_table.so; fi
if [ "" != "Windows_NT" ] && [ `uname -s` = 'Darwin' ]; then install_name_tool -id data_table.so data_table.so; fi
installing to /Users/kevin/Library/R/arm64/4.3/library/00LOCK-data.table/00new/data.table/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (data.table)

configure

kevinushey · 2024-03-30T00:08:31Z

configure

-# Clean up.
-rm -f test-omp.* a.out
+    printf "%s" "* checking if R installation supports openmp with \"-fopenmp\" flag... "
+    if ${CC} ${CFLAGS} -fopenmp test-omp.c > /dev/null 2>&1; then


As a side note, note that we intentionally do not quote variable expansions here, since ${CC} could encode both a compiler name and its flags, e.g. clang -arch arm64 on macOS.

tdhock · 2024-03-30T13:33:43Z

this looks like a step in the right direction thanks Kevin.
I wonder if this can be tested by adding a Github actions mac CI job? or is this already covered by gitlab?

MichaelChirico · 2024-03-30T15:08:37Z

we have Mac jobs in CI (yes I think GLCI), my concern is getting the tests to imitate the CRAN setup. I think once we have 3-5 people confirm success on different Mac setups we can just merge & see what happens on CRAN.

jangorecki · 2024-03-30T15:16:34Z

We do not have macos job in glci yet, most likely it could be added

MichaelChirico · 2024-03-30T17:10:50Z

We do not have macos job in glci yet, most likely it could be added

Oh, I'd seen mac-related stuff in the config before but just noticed this:

data.table/.gitlab-ci.yml

Line 277 in 898dce3

# no macosx runner set yet

OTOH we do use Mac for codecov CI:

data.table/.github/workflows/test-coverage.yaml

Line 11 in 898dce3

runs-on: macOS-latest

MichaelChirico · 2024-04-02T18:29:40Z

@kevinushey great idea to use the MacOS builder service! Looks like no luck, however:

https://mac.r-project.org/macbuilder/results/1712082011-cdc942f298135d5b/

* checking if R installation supports OpenMP with "-Xclang -fopenmp" ... no
***
*** OpenMP not supported! data.table uses OpenMP to automatically
***   parallelize operations like sorting, grouping, file reading, etc.
*** For details on how to install the necessary toolchains on your OS see:
***   https://github.com/Rdatatable/data.table/wiki/Installation
*** Continuing installation without OpenMP support...

kevinushey · 2024-04-02T19:33:47Z

I guess this makes sense ... the CRAN binaries can't assume that the requisite OpenMP libraries will be available on user systems, so OpenMP isn't made available on CRAN. I imagine this would only change if R on macOS were to begin shipping the requisite libomp libraries?

I can't think of a great solution for macOS binaries on CRAN here. 😞

MichaelChirico · 2024-04-02T19:45:05Z

maybe @s-u can weigh in on the feasibility?

s-u · 2024-04-02T22:59:16Z

It's the right idea - the problem here turns out to be omp.h which is not shipped with R, only the libomp.dylib is. But I think I can work around that on the build machines by including that header (since it is a public API and not really implementation specific it would be also possible to include it in the package) - please try again.

BTW: while at it, can you please not send the test output to /dev/null but instead append to config.log so it can be checked by the user? Currently there is no way to debug failures in the tests since they are silently swallowed.

MichaelChirico · 2024-04-03T00:17:41Z

That's great Simon! Indeed we're up & running:

https://mac.r-project.org/macbuilder/results/1712103175-69beabcd29db214b/

* checking if R installation supports OpenMP with "-Xclang -fopenmp" ... yes

With that I think we can merge here, should be plenty of time to iterate before next release if needed & we have ample evidence in-thread that things are WAI.

Thanks @kevinushey 🎉

s-u · 2024-04-03T00:21:39Z

Great, thanks!

PS: there is a NOTE from the check:

Found the following apparent object files/libraries:
  test-omp.o test-omp.so

so you may want to clean up after the tests.

kevinushey added 2 commits March 29, 2024 10:31

improve OpenMP detection on macOS

8d0c327

detect LLVM OpenMP builds

f8ed338

kevinushey requested review from jangorecki and MichaelChirico as code owners March 29, 2024 17:42

remove some comments

bf2ad12

define _OPENMP when using LLVM libomp

c16b12c

kevinushey added 2 commits March 29, 2024 17:04

fix nesting for 'echo no'

0907296

let openmp warning stand out a bit more

605bfab

kevinushey commented Mar 30, 2024

View reviewed changes

configure Show resolved Hide resolved

kevinushey commented Mar 30, 2024

View reviewed changes

MichaelChirico mentioned this pull request Mar 30, 2024

Instructions for building from source for OpenMP support on Apple Silicon / M1 / arm64 under Monterey 12.4 #5419

Closed

kevinushey mentioned this pull request Apr 2, 2024

support OpenMP with system macOS toolchain #4348

Closed

Always redirect to config.log, not /dev/null

8d46cda

MichaelChirico approved these changes Apr 3, 2024

View reviewed changes

MichaelChirico added 2 commits April 2, 2024 17:22

NEWS

ff43c1d

Restore deletion of all test-omp* artefacts

3e970e0

MichaelChirico added 4 commits April 2, 2024 17:27

try and reduce diff by restoring comment

66f64af

reduce diff: delete in same order

274d587

annotate 'fi' for readability

603e778

Merge branch 'master' into macos-openmp-support

d3ab1c9

MichaelChirico merged commit 096b20f into master Apr 3, 2024
2 checks passed

MichaelChirico deleted the macos-openmp-support branch April 3, 2024 00:42

MichaelChirico mentioned this pull request Apr 3, 2024

ignore config.log #6045

Merged

jeroen mentioned this pull request Jul 16, 2024

MacOS: fix linker flags for SHLIB feature test #6283

Merged

MichaelChirico mentioned this pull request Aug 29, 2024

[ci] [R-package] macOS clang CMake jobs failing with segfaults microsoft/LightGBM#6628

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve OpenMP detection on macOS #6034

improve OpenMP detection on macOS #6034

kevinushey commented Mar 29, 2024

codecov bot commented Mar 29, 2024 •

edited

Loading

MichaelChirico commented Mar 29, 2024

Anirban166 commented Mar 29, 2024

kevinushey commented Mar 30, 2024

kevinushey Mar 30, 2024

tdhock commented Mar 30, 2024

MichaelChirico commented Mar 30, 2024

jangorecki commented Mar 30, 2024

MichaelChirico commented Mar 30, 2024

MichaelChirico commented Apr 2, 2024

kevinushey commented Apr 2, 2024

MichaelChirico commented Apr 2, 2024

s-u commented Apr 2, 2024

MichaelChirico commented Apr 3, 2024 •

edited

Loading

s-u commented Apr 3, 2024 •

edited

Loading

improve OpenMP detection on macOS #6034

improve OpenMP detection on macOS #6034

Conversation

kevinushey commented Mar 29, 2024

codecov bot commented Mar 29, 2024 • edited Loading

Codecov Report

MichaelChirico commented Mar 29, 2024

Anirban166 commented Mar 29, 2024

kevinushey commented Mar 30, 2024

kevinushey Mar 30, 2024

Choose a reason for hiding this comment

tdhock commented Mar 30, 2024

MichaelChirico commented Mar 30, 2024

jangorecki commented Mar 30, 2024

MichaelChirico commented Mar 30, 2024

MichaelChirico commented Apr 2, 2024

kevinushey commented Apr 2, 2024

MichaelChirico commented Apr 2, 2024

s-u commented Apr 2, 2024

MichaelChirico commented Apr 3, 2024 • edited Loading

s-u commented Apr 3, 2024 • edited Loading

codecov bot commented Mar 29, 2024 •

edited

Loading

MichaelChirico commented Apr 3, 2024 •

edited

Loading

s-u commented Apr 3, 2024 •

edited

Loading