offload link error Undefined reference

[ye-luo]

Bugzilla Link	38703
Resolution	FIXED
Resolved on	Jul 17, 2020 15:05
Version	unspecified
OS	Linux
CC	@alexey-bataev,@hahnjo,@hfinkel

Extended Description

When compiling miniQMC offload branch with clang+cuda8
https://github.com/QMCPACK/miniqmc/tree/OMP_offload
mkdir build_clang_offload; cd build_clang_offload
cmake -DCMAKE_CXX_COMPILER="clang++" -DQMC_MPI=0 -DENABLE_OFFLOAD=1 ..
make

I got the following error. The strange thing is the other binary miniqmc is built without problem. It includes the same part of code as check_spo.

$ /sandbox/opt/llvm-clang/clang-offload/bin/clang++ -v --gcc-toolchain=/soft/apps/packages/gcc/gcc-6.2.0 -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda -fomit-frame-pointer -fstrict-aliasing -D__forceinline=inline -Wno-deprecated -Wno-unused-value -Wno-undefined-var-template -O3 -DNDEBUG -ffast-math CMakeFiles/check_spo.dir/check_spo.cpp.o -o ../../bin/check_spo ../../lib/libminiwfs.a ../../lib/libqmcbase.a ../../lib/libqmcutil.a /usr/lib/liblapack.so /usr/lib/libf77blas.so /usr/lib/libatlas.so
clang version 8.0.0 (https://github.com/llvm-mirror/clang.git 1bdec838c54117d9befe6dfcd3336a0ba5de9e0d) (https://github.com/llvm-mirror/llvm.git 7b5db98)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /sandbox/opt/llvm-clang/clang-offload/bin
Found candidate GCC installation: /soft/apps/packages/gcc/gcc-6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0
Selected GCC installation: /soft/apps/packages/gcc/gcc-6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda-8.0, version 8.0
"/sandbox/opt/llvm-clang/clang-offload/bin/clang-offload-bundler" -type=o -targets=host-x86_64-unknown-linux-gnu,openmp-nvptx64-nvidia-cuda -inputs=CMakeFiles/check_spo.dir/check_spo.cpp.o -outputs=/tmp/check_spo-f173d6.o,/tmp/check_spo-9d05af.cubin -unbundle
"/sandbox/opt/llvm-clang/clang-offload/bin/clang-offload-bundler" -type=o -targets=host-x86_64-unknown-linux-gnu,openmp-nvptx64-nvidia-cuda -inputs=../../lib/libminiwfs.a -outputs=/tmp/libminiwfs-91a43f.o,/tmp/libminiwfs-499503.cubin -unbundle
"/sandbox/opt/llvm-clang/clang-offload/bin/clang-offload-bundler" -type=o -targets=host-x86_64-unknown-linux-gnu,openmp-nvptx64-nvidia-cuda -inputs=../../lib/libqmcbase.a -outputs=/tmp/libqmcbase-2f9488.o,/tmp/libqmcbase-479ba6.cubin -unbundle
"/sandbox/opt/llvm-clang/clang-offload/bin/clang-offload-bundler" -type=o -targets=host-x86_64-unknown-linux-gnu,openmp-nvptx64-nvidia-cuda -inputs=../../lib/libqmcutil.a -outputs=/tmp/libqmcutil-23428e.o,/tmp/libqmcutil-11fb62.cubin -unbundle
"/sandbox/opt/llvm-clang/clang-offload/bin/clang-offload-bundler" -type=o -targets=host-x86_64-unknown-linux-gnu,openmp-nvptx64-nvidia-cuda -inputs=/usr/lib/liblapack.so -outputs=/tmp/liblapack-bac4fc.o,/tmp/liblapack-71b054.cubin -unbundle
"/sandbox/opt/llvm-clang/clang-offload/bin/clang-offload-bundler" -type=o -targets=host-x86_64-unknown-linux-gnu,openmp-nvptx64-nvidia-cuda -inputs=/usr/lib/libf77blas.so -outputs=/tmp/libf77blas-8ab156.o,/tmp/libf77blas-879c92.cubin -unbundle
"/sandbox/opt/llvm-clang/clang-offload/bin/clang-offload-bundler" -type=o -targets=host-x86_64-unknown-linux-gnu,openmp-nvptx64-nvidia-cuda -inputs=/usr/lib/libatlas.so -outputs=/tmp/libatlas-aa3969.o,/tmp/libatlas-c13852.cubin -unbundle
"/usr/local/cuda-8.0/bin/nvlink" -o /tmp/check_spo-4c4fc9.out -v -arch sm_61 -L/sandbox/opt/llvm-clang/clang-offload/lib -L/usr/lib/x86_64-linux-gnu -L/sandbox/opt/llvm-clang/clang-offload/lib -lomptarget-nvptx /tmp/check_spo-9d05af.cubin /tmp/libminiwfs-499503.cubin /tmp/libqmcbase-479ba6.cubin /tmp/libqmcutil-11fb62.cubin /tmp/liblapack-71b054.cubin /tmp/libf77blas-879c92.cubin /tmp/libatlas-c13852.cubin
nvlink error : Undefined reference to 'modf' in '/tmp/check_spo-9d05af.cubin'
nvlink error : Undefined reference to 'ZSt3minIiERKT_S2_S2' in '/tmp/check_spo-9d05af.cubin'
nvlink error : Undefined reference to 'ZSt3maxIiERKT_S2_S2' in '/tmp/check_spo-9d05af.cubin'
nvlink error : Undefined reference to '_ZN11qmcplusplus16MultiBsplineDataIdE3A44E' in '/tmp/check_spo-9d05af.cubin'
nvlink error : Undefined reference to '_ZN11qmcplusplus16MultiBsplineDataIdE4dA44E' in '/tmp/check_spo-9d05af.cubin'
nvlink error : Undefined reference to '_ZN11qmcplusplus16MultiBsplineDataIdE5d2A44E' in '/tmp/check_spo-9d05af.cubin'
nvlink info : 703726375 bytes gmem
nvlink info : Function properties for '__omp_offloading_805_19e0a96__ZN11qmcplusplus13einspline_spoIdEC1ERKS1_ii_l114':
nvlink info : used 254 registers, 544 stack, 1092 bytes smem, 344 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_805_19e0a96__ZN11qmcplusplus13einspline_spoIdE18multi_evaluate_vghERKSt6vectorIPNS_6SPOSetESaIS4_EERKS2_INS_10TinyVectorIdLj3EEESaISA_EE_l388':
nvlink info : used 254 registers, 568 stack, 1092 bytes smem, 384 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_805_19e0a96__ZN11qmcplusplus13einspline_spoIdE12evaluate_vghERKNS_10TinyVectorIdLj3EEE_l285':
nvlink info : used 254 registers, 560 stack, 1092 bytes smem, 376 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_805_19e0a96__ZN11qmcplusplus13einspline_spoIdE3setEiiiiib_l185':
nvlink info : used 254 registers, 544 stack, 1092 bytes smem, 352 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_805_19e0a96__ZN11qmcplusplus13einspline_spoIdE18multi_evaluate_vghERKSt6vectorIPNS_6SPOSetESaIS4_EERKS2_INS_10TinyVectorIdLj3EEESaISA_EE_l364':
nvlink info : used 254 registers, 544 stack, 1092 bytes smem, 376 bytes cmem[0], 0 bytes lmem
nvlink info : Function properties for '__omp_offloading_805_19e0a96__ZN11qmcplusplus13einspline_spoIdE12evaluate_vghERKNS_10TinyVectorIdLj3EEE_l267':
nvlink info : used 254 registers, 544 stack, 1092 bytes smem, 376 bytes cmem[0], 0 bytes lmem
"/usr/lib/binutils-2.26/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o ../../bin/check_spo /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o /soft/apps/packages/gcc/gcc-6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0/crtbegin.o /soft/apps/packages/gcc/gcc-6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0/crtfastmath.o -L/soft/apps/packages/gcc/gcc-6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0 -L/soft/apps/packages/gcc/gcc-6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/soft/apps/packages/gcc/gcc-6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0/../../.. -L/sandbox/opt/llvm-clang/clang-offload/bin/../lib -L/lib -L/usr/lib /tmp/check_spo-f173d6.o /tmp/libminiwfs-91a43f.o /tmp/libqmcbase-2f9488.o /tmp/libqmcutil-23428e.o /tmp/liblapack-bac4fc.o /tmp/libf77blas-8ab156.o /tmp/libatlas-aa3969.o -L/sandbox/opt/llvm-clang/clang-offload/lib -L/usr/lib/x86_64-linux-gnu -lstdc++ -lm -lomp -lomptarget -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /soft/apps/packages/gcc/gcc-6.2.0/lib/gcc/x86_64-pc-linux-gnu/6.2.0/crtend.o /usr/lib/x86_64-linux-gnu/crtn.o -T /tmp/check_spo-343f59.lk
clang-8: error: nvlink command failed with exit code 255 (use -v to see invocation)

[hahnjo]

Are these functions defined in the static libraries? That's not supported yet, see http://clang.llvm.org/docs/OpenMPSupport.html#features-not-supported-or-with-limited-support-for-cuda-devices.

[hfinkel]

see also: http://lists.llvm.org/pipermail/cfe-dev/2018-August/059168.html

[hahnjo]

and / or https://reviews.llvm.org/D47394

[ye-luo]

No more error at linking but the issue of linking library archives with device code remains. Tracked by llvm/llvm-bugzilla-archive#42395

[ye-luo]

mentioned in issue llvm/llvm-bugzilla-archive#42395

[alexey-bataev]

mentioned in issue llvm/llvm-bugzilla-archive#42395