Skip to content
This repository has been archived by the owner on Mar 12, 2021. It is now read-only.

Test failures on Tesla K20c/CUDA 9.0 #594

Closed
timholy opened this issue Feb 17, 2020 · 1 comment
Closed

Test failures on Tesla K20c/CUDA 9.0 #594

timholy opened this issue Feb 17, 2020 · 1 comment
Labels
bug

Comments

@timholy
Copy link
Member

timholy commented Feb 17, 2020

I get test failures when I run the tests locally:

julia> ENV["JULIA_DEBUG"]="CUDAapi"
"CUDAapi"

julia> include("runtests.jl")
┌ Debug: CUDA toolkit identified as 9.0.176
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:297
┌ Debug: Request to look for library cublas
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcublas.so, libcublas.so.9, libcublas.so.9.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Found library libcublas.so at /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:90
┌ Debug: Request to look for library cusparse
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcusparse.so, libcusparse.so.9, libcusparse.so.9.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Found library libcusparse.so at /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:90
┌ Debug: Request to look for library cusolver
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcusolver.so, libcusolver.so.9, libcusolver.so.9.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Found library libcusolver.so at /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:90
┌ Debug: Request to look for library cufft
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcufft.so, libcufft.so.9, libcufft.so.9.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Found library libcufft.so at /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:90
┌ Debug: Request to look for library curand
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcurand.so, libcurand.so.9, libcurand.so.9.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Found library libcurand.so at /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:90
┌ Debug: Request to look for library cudnn
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcudnn.so, libcudnn.so.7, libcudnn.so.7.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Request to look for library cutensor
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcutensor.so, libcutensor.so.1, libcutensor.so.1.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
[ Info: Testing using device Tesla K20c (compute capability 3.5.0, 4.873 GiB available memory) on CUDA driver 9.0.0 and toolkit 9.0.176
┌ Warning: Performing scalar operations on GPU arrays: This is very slow, consider disallowing these operations with `allowscalar(false)`
└ @ GPUArrays ~/.julia/packages/GPUArrays/1wgPO/src/indexing.jl:16
Array: Test Failed at /home/tim/.julia/dev/CuArrays/test/base.jl:44
  Expression: cu([1]) isa AbstractArray{Int}
   Evaluated: Float32[1.0] isa AbstractArray{Int64,N} where N
Stacktrace:
 [1] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:44
 [2] top-level scope at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
 [3] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:34
SubArray: Error During Test at /home/tim/.julia/dev/CuArrays/test/base.jl:173
  Got exception outside of a @test
  ArgumentError: invalid index: 1.0f0 of type Float32
  Stacktrace:
   [1] to_index(::Float32) at ./indices.jl:273
   [2] to_index(::Array{Float32,1}, ::Float32) at ./indices.jl:250
   [3] to_indices at ./indices.jl:301 [inlined]
   [4] to_indices at ./indices.jl:298 [inlined]
   [5] getindex at ./abstractarray.jl:981 [inlined]
   [6] getindex at ./subarray.jl:230 [inlined]
   [7] isassigned(::SubArray{Float32,1,Array{Float32,1},Tuple{Array{Float32,1}},false}, ::Int64) at ./abstractarray.jl:405
   [8] show_delim_array(::IOContext{Base.DevNull}, ::SubArray{Float32,1,Array{Float32,1},Tuple{Array{Float32,1}},false}, ::Char, ::String, ::Char, ::Bool, ::Int64, ::Int64) at ./show.jl:661
   [9] show_delim_array at ./show.jl:654 [inlined]
   [10] show_vector(::Base.DevNull, ::SubArray{Float32,1,Array{Float32,1},Tuple{Array{Float32,1}},false}, ::Char, ::Char) at ./arrayshow.jl:443
   [11] show_vector(::Base.DevNull, ::SubArray{Float32,1,Array{Float32,1},Tuple{Array{Float32,1}},false}) at ./arrayshow.jl:432
   [12] show_vector(::Base.DevNull, ::SubArray{Float32,1,CuArray{Float32,1,Nothing},Tuple{CuArray{Float32,1,Nothing}},false}) at /home/tim/.julia/packages/GPUArrays/1wgPO/src/abstractarray.jl:71
   [13] show(::Base.DevNull, ::SubArray{Float32,1,CuArray{Float32,1,Nothing},Tuple{CuArray{Float32,1,Nothing}},false}) at ./arrayshow.jl:418
   [14] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:237
   [15] top-level scope at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [16] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:174
   [17] include at ./boot.jl:328 [inlined]
   [18] include_relative(::Module, ::String) at ./loading.jl:1105
   [19] include(::Module, ::String) at ./Base.jl:31
   [20] include(::String) at ./client.jl:424
   [21] top-level scope at /home/tim/.julia/dev/CuArrays/test/runtests.jl:46
   [22] top-level scope at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [23] top-level scope at /home/tim/.julia/dev/CuArrays/test/runtests.jl:38
   [24] include at ./boot.jl:328 [inlined]
   [25] include_relative(::Module, ::String) at ./loading.jl:1105
   [26] include(::Module, ::String) at ./Base.jl:31
   [27] include(::String) at ./client.jl:424
   [28] top-level scope at REPL[5]:1
   [29] eval(::Module, ::Any) at ./boot.jl:330
   [30] eval_user_input(::Any, ::REPL.REPLBackend) at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:86
   [31] run_backend(::REPL.REPLBackend) at /home/tim/.julia/packages/Revise/ZOWOa/src/Revise.jl:1057
   [32] (::Revise.var"#85#87"{REPL.REPLBackend})() at ./task.jl:333
  
aliasing: Error During Test at /home/tim/.julia/dev/CuArrays/test/base.jl:420
  Got exception outside of a @test
  ArgumentError: an array of type `CuArray` shares memory with another argument and must
  make a preventative copy of itself in order to maintain consistent semantics,
  but `copy(A)` returns a new array of type `CuArray{Int64,1,Nothing}`. To fix, implement:
      `Base.unaliascopy(A::CuArray)::typeof(A)`
  Stacktrace:
   [1] _unaliascopy(::CuArray{Int64,1,CuArray{Int64,1,Nothing}}, ::CuArray{Int64,1,Nothing}) at ./abstractarray.jl:1174
   [2] unaliascopy(::CuArray{Int64,1,CuArray{Int64,1,Nothing}}) at ./abstractarray.jl:1172
   [3] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:428
   [4] top-level scope at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [5] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:421
   [6] include at ./boot.jl:328 [inlined]
   [7] include_relative(::Module, ::String) at ./loading.jl:1105
   [8] include(::Module, ::String) at ./Base.jl:31
   [9] include(::String) at ./client.jl:424
   [10] top-level scope at /home/tim/.julia/dev/CuArrays/test/runtests.jl:46
   [11] top-level scope at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [12] top-level scope at /home/tim/.julia/dev/CuArrays/test/runtests.jl:38
   [13] include at ./boot.jl:328 [inlined]
   [14] include_relative(::Module, ::String) at ./loading.jl:1105
   [15] include(::Module, ::String) at ./Base.jl:31
   [16] include(::String) at ./client.jl:424
   [17] top-level scope at REPL[5]:1
   [18] eval(::Module, ::Any) at ./boot.jl:330
   [19] eval_user_input(::Any, ::REPL.REPLBackend) at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:86
   [20] run_backend(::REPL.REPLBackend) at /home/tim/.julia/packages/Revise/ZOWOa/src/Revise.jl:1057
   [21] (::Revise.var"#85#87"{REPL.REPLBackend})() at ./task.jl:333
  
julia: symbol lookup error: /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib/libcusolver.so: undefined symbol: omp_get_max_threads

Build log

tim@cannon:~/.julia/dev/CuArrays/deps$ cat build.log 
Could not find library 'cusolver'.
Could not find library 'cudnn'.

CuArrays.jl has been built successfully, but there were warnings.
Some functionality may be unavailable.

Environment details (please complete this section)
Details on Julia:

Julia Version 1.3.2-pre.0
Commit 2e6715c (2019-12-31 00:49 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, sandybridge)
Environment:
  JULIA_CUDA_VERBOSE = 1

Julia packages:

  • CuArrays.jl:
  • CUDAnative.jl:
  • ...
@timholy timholy added the bug label Feb 17, 2020
@maleadt
Copy link
Member

maleadt commented Feb 18, 2020

Bug in CUDA 9.0, as confirmed by NVIDIA: JuliaGPU/julia-ngc#10 (comment)

@maleadt maleadt closed this as completed Feb 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@maleadt @timholy