juliaCall - MendelIHT initialization warning

[DanielEWeeks]

I am using juliaCall to call MendelIHT from within R on an Apple MacBook Air M1.

When I first issue the command

using MendelIHT

in an R Studio julia chunk, I get this warning/error message:

WARNING: both BGEN and SnpArrays export "maf"; uses of it in module MendelIHT must be qualified
Error: Error happens in Julia.
InitError: TaskFailedException

I guess it is just a 'warning' and so can be ignored, but of course the subsequent lines make it sound like it is an error that perhaps I should be concerned about.

But then if I do

using MendelIHT

a second time, it seems that MendelIHT is loaded OK.

This is with version 1.4.7:

> julia_installed_package("MendelIHT")
[1] "1.4.7"

But strangely I don't get this warning when starting up in the Terminal at the command line:

$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.5 (2023-01-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using MendelIHT

julia> 

Here's the full error message obtained when starting up MendelIHT via JuliaCall:

WARNING: both BGEN and SnpArrays export "maf"; uses of it in module MendelIHT must be qualified
Error: Error happens in Julia.
InitError: TaskFailedException
Stacktrace:
  [1] wait
    @ ./task.jl:345 [inlined]
  [2] fetch
    @ ./task.jl:360 [inlined]
  [3] take!(pool::ThreadPools.QueuePool)
    @ ThreadPools ~/.julia/packages/ThreadPools/ANo2I/src/qpool.jl:96
  [4] tmap(fn::MendelIHT.var"#71#76"{Float64, Vector{Int64}, Bool, Bool, Int64, Int64, Bool, Vector{Float64}, Vector{Tuple{Int64, Int64}}, Nothing, Vector{MendelIHT.mIHTVariable{Float64, Transpose{Float64, SnpArrays.SnpLinAlg{Float64}}}}, Vector{BitVector}, Vector{BitVector}}, pool::ThreadPools.QueuePool, itr::UnitRange{Int64})
    @ ThreadPools ~/.julia/packages/ThreadPools/ANo2I/src/qpool.jl:202
  [5] tforeach(fn::Function, pool::ThreadPools.QueuePool, itr::UnitRange{Int64})
    @ ThreadPools ~/.julia/packages/ThreadPools/ANo2I/src/interface.jl:32
  [6] #70
    @ ~/.julia/packages/ThreadPools/ANo2I/src/macros.jl:18 [inlined]
  [7] twith
    @ ~/.julia/packages/ThreadPools/ANo2I/src/interface.jl:91 [inlined]
  [8] macro e

> julia_command("using InteractiveUtils; versioninfo()")
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.5.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 8 on 8 virtual cores
Environment:
  JULIA_NUM_THREADS = 8

> sessionInfo()
R version 4.2.2 (2022-10-31)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Big Sur 11.7.2

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2-arm64/Versions/4.2-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
 [1] JuliaCall_0.17.5  pander_0.6.5      glmnet_4.1-6     
 [4] Matrix_1.5-3      parameters_0.20.2 tidylog_1.0.2    
 [7] lubridate_1.9.2   forcats_1.0.0     stringr_1.5.0    
[10] dplyr_1.1.0       purrr_1.0.1       readr_2.1.4      
[13] tidyr_1.3.0       tibble_3.1.8      ggplot2_3.4.1    
[16] tidyverse_2.0.0   knitr_1.42       

loaded via a namespace (and not attached):
 [1] shape_1.4.6        zoo_1.8-11         clisymbols_1.2.0  
 [4] tidyselect_1.2.0   xfun_0.37          splines_4.2.2     
 [7] lattice_0.20-45    colorspace_2.1-0   vctrs_0.5.2       
[10] generics_0.1.3     htmltools_0.5.4    yaml_2.3.7        
[13] utf8_1.2.3         survival_3.5-3     rlang_1.0.6       
[16] pillar_1.8.1       glue_1.6.2         withr_2.5.0       
[19] emmeans_1.8.4-1    multcomp_1.4-22    foreach_1.5.2     
[22] lifecycle_1.0.3    munsell_0.5.0      gtable_0.3.1      
[25] bayestestR_0.13.0  mvtnorm_1.1-3      evaluate_0.20     
[28] codetools_0.2-19   coda_0.19-4        fastmap_1.1.1     
[31] tzdb_0.3.0         datawizard_0.6.5   fansi_1.0.4       
[34] Rcpp_1.0.10        TH.data_1.1-1      xtable_1.8-4      
[37] scales_1.2.1       digest_0.6.31      hms_1.1.2         
[40] stringi_1.7.12     insight_0.19.0     grid_4.2.2        
[43] cli_3.6.0          tools_4.2.2        sandwich_3.0-2    
[46] magrittr_2.0.3     pkgconfig_2.0.3    ellipsis_0.3.2    
[49] MASS_7.3-58.2      estimability_1.4.1 timechange_0.2.0  
[52] rmarkdown_2.20     rstudioapi_0.14    iterators_1.0.14  
[55] R6_2.5.1           compiler_4.2.2   

Minimal Working Example:

> Sys.setenv(JULIA_NUM_THREADS = 8)
> library(JuliaCall)
> julia <- julia_setup(JULIA_HOME = "/Applications/Julia-1.8.app/Contents/Resources/julia/bin")
> julia_command("using MendelIHT")
WARNING: both BGEN and SnpArrays export "maf"; uses of it in module MendelIHT must be qualified
Error: Error happens in Julia.
InitError: TaskFailedException
Stacktrace:
  [1] wait
    @ ./task.jl:345 [inlined]
  [2] fetch
    @ ./task.jl:360 [inlined]
  [3] take!(pool::ThreadPools.QueuePool)
    @ ThreadPools ~/.julia/packages/ThreadPools/ANo2I/src/qpool.jl:96
  [4] tmap(fn::MendelIHT.var"#71#76"{Float64, Vector{Int64}, Bool, Bool, Int64, Int64, Bool, Vector{Float64}, Vector{Tuple{Int64, Int64}}, Nothing, Vector{MendelIHT.mIHTVariable{Float64, Transpose{Float64, SnpArrays.SnpLinAlg{Float64}}}}, Vector{BitVector}, Vector{BitVector}}, pool::ThreadPools.QueuePool, itr::UnitRange{Int64})
    @ ThreadPools ~/.julia/packages/ThreadPools/ANo2I/src/qpool.jl:202
  [5] tforeach(fn::Function, pool::ThreadPools.QueuePool, itr::UnitRange{Int64})
    @ ThreadPools ~/.julia/packages/ThreadPools/ANo2I/src/interface.jl:32
  [6] #70
    @ ~/.julia/packages/ThreadPools/ANo2I/src/macros.jl:18 [inlined]
  [7] twith
    @ ~/.julia/packages/ThreadPools/ANo2I/src/interface.jl:91 [inlined]
  [8] macro e

[biona001]

I'm a bit confused since none of the errors seems to be caused by code in MendelIHT.jl. I will try to reproduce this error later next week. For now, I can only suggest a few possibilities:

1. Do you get the same error if you run in Julia REPL? (I want to know if this is an issue with interfacing between Julia and R, since apparently R is single threaded?)
2. I wonder if this is an M1 issue? Could you try starting Julia and the terminal with Rosetta and see if the issue persists? I think you can start Julia with Rosetta by right-clicking /Applications/Julia-1.8.app -> get info -> open using Rosetta
3. If this is actually an issue with ThreadsPools.jl, we no longer have to depend on it in Julia 1.8+, but I doubt this is the case...

[kose-y]

Could we try to resolve it by just qualifying the maf() call as suggested in the warning message?

[biona001]

Sorry for the lack of updates. It took me some time to figure out what the issue was.

This is an issue with multithreaded cross-validation (nothing to do with R or M1 CPUs). I'm not 100% sure yet, but it seems like a dynamic scheduler causes a race condition in my cross-validation code. I'll fix this in a separate issue.

To resolve this issue, I'll revert cross-validation back to a static scheduler in the next minor release, which seems to work. This shouldn't really cause any computational slowdowns, but it'd still be interesting to know why the difference causes a race condition.

> Sys.setenv(JULIA_NUM_THREADS = 8)
> library(JuliaCall)
> julia <- julia_setup(JULIA_HOME = "/Applications/Julia-1.8.app/Contents/Resources/julia/bin")
Julia version 1.8.5 at location /Applications/Julia-1.8.app/Contents/Resources/julia/bin will be used.
Loading setup script for JuliaCall...
Finish loading setup script for JuliaCall.
> julia_command("using MendelIHT")
>