Linking CUDA executable exception_helpers fails

[greole]

Hi,
when building ginkgo I get the following linker error:

[ 28%] Linking CUDA executable exception_helpers /usr/bin/ld: ../../../third_party/gtest/build/googlemock/gtest/./libgtest.a(gtest-all.cc.o): in function testing::internal::JsonUnitTestResultPrinter::PrintJsonTestCa
se(std::ostream*, testing::TestCase const&)':
gtest-all.cc:(.text+0x2008c): undefined reference to std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()' /usr/bin/ld: ../../../third_party/gtest/build/googlemock/gtest/./libgtest.a(gtest-all.cc.o): in function testing::internal::JsonUnitTestResultPrinter::PrintJsonUnitTe
st(std::ostream*, testing::UnitTest const&)':
gtest-all.cc:(.text+0x20f1b): undefined reference to std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()' /usr/bin/ld: ../../../third_party/gtest/build/googlemock/gtest/./libgtest.a(gtest-all.cc.o): in function testing::internal::edit_distance::CreateUnifiedDiff(std::vect
or<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::alloca
tor > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, st
d::char_traits, std::allocator > > > const&, unsigned long)':
gtest-all.cc:(.text+0x233e3): undefined reference to std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()' /usr/bin/ld: ../../../third_party/gtest/build/googlemock/gtest/./libgtest.a(gtest-all.cc.o): in function testing::AssertionResult testing::internal::FloatingPointLE(char const*, char const*, float, float)':
gtest-all.cc:(.text.ZN7testing8internal15FloatingPointLEIfEENS_15AssertionResultEPKcS4_T_S5[ZN7testing8internal15FloatingPointLEIfEENS_15AssertionResultEPKcS4_T_S5
]+0xfc): undefined reference to std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()' /usr/bin/ld: gtest-all.cc:(.text._ZN7testing8internal15FloatingPointLEIfEENS_15AssertionResultEPKcS4_T_S5_[_ZN7testing8internal15FloatingPointLEIfEENS_15AssertionResul tEPKcS4_T_S5_]+0x13a): undefined reference to std::__cxx11::basic_stringstream<char, std::char_traits, std::allocator >::basic_stringstream()'
/usr/bin/ld: ../../../third_party/gtest/build/googlemock/gtest/./libgtest.a(gtest-all.cc.o):gtest-all.cc:(.text.ZN7testing8internal15FloatingPointLEIdEENS_15Assertion
ResultEPKcS4_T_S5[ZN7testing8internal15FloatingPointLEIdEENS_15AssertionResultEPKcS4_T_S5]+0x124): more undefined references to std::__cxx11::basic_stringstream<ch ar, std::char_traits<char>, std::allocator<char> >::basic_stringstream()' follow /usr/bin/ld: ../../../core/libginkgo.so.1.1.1: undefined reference to std::__cxx11::basic_ostringstream<char, std::char_traits, std::allocator >::basic_os
tringstream()@GLIBCXX_3.4.26'
collect2: error: ld returned 1 exit status
make[2]: *** [cuda/test/base/CMakeFiles/cuda_test_base_exception_helpers.dir/build.make:111: cuda/test/base/exception_helpers] Error 1
make[1]: *** [CMakeFiles/Makefile2:2908: cuda/test/base/CMakeFiles/cuda_test_base_exception_helpers.dir/all] Error 2
`
I use the following tools with given version

• CUDA Version 10.2.89
• cmake version 3.17.3
• gcc (GCC) 10.1.0

[upsj]

Thanks for letting us know! I don't think we ever used a version of GCC that recent, that might contribute to the issue (I didn't even know CUDA 10.2 supports such recent GCC versions). For completeness, can you post your CMakeCache.txt file and the Commit ID you are working on as well?

[greole]

Yeah, for CUDA 10.2 only gcc <= 8 is supported. I'll try clang or an older version of gcc.

Here is my CMakeCache.txt and i am working on 74f47ac.

[tcojean]

From afar, these look like the code isn't compiled with C++11 in this case. Either gtest or the CUDA example itself.

What if you do something like:

nm build/third_party/gtest/build/googlemock/gtest/./libgtest.a | grep "basic_stringstream"

I guess you will have a reference to std::__cxx11::...?

And what is the command used for compiling this cuda example, by using make VERBOSE=1? Both the actual compilation and the linking which throws the error.

[greole]

Weird, shouldn't gcc default to c++ 14? To me it looks like a problem propagating the -std=c++1? flag. And other parts are linking well.

Here are the relevant outputs:

~/data/code/ginkgo/build develop nm third_party/gtest/build/googlemock/gtest/./libgtest.a | grep "basic_stringstream"
0000000000005d20 t _ZN7testing8internal10scoped_ptrINSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEEE5resetEPS7_.part.0.constprop.0
0000000000002c70 T _ZN7testing8internal20StringStreamToStringEPNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEE
0000000000000153 t _ZN7testing8internal20StringStreamToStringEPNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEE.cold
                 U _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev
                 U _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEED1Ev
                 U _ZTTNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEE
                 U _ZTVNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEE

[ 28%] Linking CUDA executable exception_helpers
cd /home/go/data/code/ginkgo/build/cuda/test/base && /usr/bin/cmake -E cmake_link_script CMakeFiles/cuda_test_base_exception_helpers.dir/link.txt --verbose=1
/opt/cuda/bin/g++  CMakeFiles/cuda_test_base_exception_helpers.dir/exception_helpers.cu.o -o exception_helpers  -Wl,-rpath,/home/go/data/code/ginkgo/build/core:/home/go/data/code/ginkgo/build/omp:/home/go/data/code/ginkgo/build/cuda:/home/go/data/code/ginkgo/build/reference:/home/go/data/code/ginkgo/build/core/device_hooks ../../../core/libginkgo.so.1.1.1 ../../../third_party/gtest/build/googlemock/gtest/./libgtest_main.a ../../../third_party/gtest/build/googlemock/gtest/./libgtest.a ../../../omp/libginkgo_omp.so.1.1.1 ../../libginkgo_cuda.so.1.1.1 ../../../reference/libginkgo_reference.so.1.1.1 ../../../core/device_hooks/libginkgo_hip.so.1.1.1 -lcudadevrt -lcudart_static  -L"/opt/cuda/targets/x86_64-linux/lib/stubs" -L"/opt/cuda/targets/x86_64-linux/lib" -lrt -lpthread -ldl
/usr/bin/ld: ../../../third_party/gtest/build/googlemock/gtest/./libgtest.a(gtest-all.cc.o): in function `testing::internal::JsonUnitTestResultPrinter::PrintJsonTestCase(std::ostream*, testing::TestCase const&)':
gtest-all.cc:(.text+0x2008c): undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/usr/bin/ld: ../../../third_party/gtest/build/googlemock/gtest/./libgtest.a(gtest-all.cc.o): in function `testing::internal::JsonUnitTestResultPrinter::PrintJsonUnitTest(std::ostream*, testing::UnitTest const&)':
gtest-all.cc:(.text+0x20f1b): undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/usr/bin/ld: ../../../third_party/gtest/build/googlemock/gtest/./libgtest.a(gtest-all.cc.o): in function `testing::internal::edit_distance::CreateUnifiedDiff(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned long)':
gtest-all.cc:(.text+0x233e3): undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/usr/bin/ld: ../../../third_party/gtest/build/googlemock/gtest/./libgtest.a(gtest-all.cc.o): in function `testing::AssertionResult testing::internal::FloatingPointLE<float>(char const*, char const*, float, float)':
gtest-all.cc:(.text._ZN7testing8internal15FloatingPointLEIfEENS_15AssertionResultEPKcS4_T_S5_[_ZN7testing8internal15FloatingPointLEIfEENS_15AssertionResultEPKcS4_T_S5_]+0xfc): undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/usr/bin/ld: gtest-all.cc:(.text._ZN7testing8internal15FloatingPointLEIfEENS_15AssertionResultEPKcS4_T_S5_[_ZN7testing8internal15FloatingPointLEIfEENS_15AssertionResultEPKcS4_T_S5_]+0x13a): undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/usr/bin/ld: ../../../third_party/gtest/build/googlemock/gtest/./libgtest.a(gtest-all.cc.o):gtest-all.cc:(.text._ZN7testing8internal15FloatingPointLEIdEENS_15AssertionResultEPKcS4_T_S5_[_ZN7testing8internal15FloatingPointLEIdEENS_15AssertionResultEPKcS4_T_S5_]+0x124): more undefined references to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()' follow
/usr/bin/ld: ../../../core/libginkgo.so.1.1.1: undefined reference to `std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream()@GLIBCXX_3.4.26'
collect2: error: ld returned 1 exit status
make[2]: *** [cuda/test/base/CMakeFiles/cuda_test_base_exception_helpers.dir/build.make:111: cuda/test/base/exception_helpers] Error 1
make[2]: Leaving directory '/home/go/data/code/ginkgo/build'
make[1]: *** [CMakeFiles/Makefile2:2908: cuda/test/base/CMakeFiles/cuda_test_base_exception_helpers.dir/all] Error 2
make[1]: Leaving directory '/home/go/data/code/ginkgo/build'
make: *** [Makefile:183: all] Error 2

[tcojean]

The last output is for the linking command, what about the compiling one? You would need to do rm /home/go/data/code/ginkgo/build/cuda/test/base/CMakeFiles/cuda_test_base_exception_helpers.dir/exception_helpers.cu.o, or something like that.

Also, my opinion is not that gcc is the issue but maybe nvcc is the one which doesn't properly use/interpret the C++11 flags. It also depends on the host compiler used for nvcc since I don't think GCC 10.1 is compatible with CUDA 10.2.
Can you also run /usr/bin/c++ --version, is it also GCC 10.1? What about /opt/cuda/bin/g++ --version?

[tcojean]

After recompiling, can you also try to run:
nm /home/go/data/code/ginkgo/build/cuda/test/base/CMakeFiles/cuda_test_base_exception_helpers.dir/exception_helpers.cu.o | c++filt | grep "basic_stringstream" ?

[tcojean]

As a side comment, I tried to reproduce the issue in a container based on Ubuntu but failed. Everything compiles successfully. I had to explicitly give as a host compiler g++-7 since on Ubuntu, the nvcc host compiler isn't integrated into the CUDA package, it is simply supposed to be the default compiler of that release. Of course, by default it would not compile since GCC 10.1 cannot be used as a host compiler for CUDA 10.2.

One can use the image localhost:5000/cuda102_gnu10_llvm9 to check this.

I think the root of the issue is not really in GCC 10.1 then but rather in the compiler that you use as CUDA host compiler. Either /usr/bin/c++ or /opt/cuda/bin/g++ as the CMakeCache.txt and make output you shared shows.

[greole]

@tcojean sorry, i missed the building step.

[ 28%] Building CUDA object cuda/test/base/CMakeFiles/cuda_test_base_exception_helpers.dir/exception_helpers.cu.o
cd /home/go/data/code/ginkgo/build/cuda/test/base && /opt/cuda/bin/nvcc -forward-unknown-to-host-compiler  -I/home/go/data/code/ginkgo/build -I/home/go/data/code/ginkgo/build/include -I/home/go/data/code/ginkgo/include -I/home/go/data/code/ginkgo -isystem=/home/go/`

As you can see there is no -std=c++11 flag, does nvcc default to pre c++ 11? I tried compiling it using clang 10.0.0 which had ended with the same linker error, which rules gcc out.

As for the nm output:

nm /home/go/data/code/ginkgo/build/cuda/test/base/CMakeFiles/cuda_test_base_exception_helpers.dir/exception_helpers.cu.o | c++filt | grep "basic_stringstream"
0000000000000000 t testing::internal::scoped_ptr<std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> > >::reset(std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*) [clone .isra.11] [clone .constprop.30]

I am running on arch linux and will try to reproduce it in a clean container, to rule out my machine setup as the root cause.

[tcojean]

Thanks for the new details. @thoasm also looked a bit into it on his own Arch Linux setup and seems to be having the same issue.

It is intriguing that no mention of c++11 is added to the compilation line. In that case, what if you run the command for compiling by hand but add --std=c++11 to the nvcc invocation? Afterwards when you manually run the linking, do you still have the same error?

[greole]

Ok, the build output line was wrong, it got truncated at the line break. So the full output was

[ 28%] Building CUDA object cuda/test/base/CMakeFiles/cuda_test_base_exception_helpers.dir/exception_helpers.cu.o cd /home/go/data/code/ginkgo/build/cuda/test/base && /opt/cuda/bin/nvcc -forward-unknown-to-host-compiler -I/home/go/data/code/ginkgo/build -I/home/go/data/code/ginkg o/build/include -I/home/go/data/code/ginkgo/include -I/home/go/data/code/ginkgo -isystem=/home/go/data/code/ginkgo/build/third_party/gtest/src/googletest/include -O3 -DNDEBUG -std=c++14 -x cu -c /home/go/data/code/ginkgo/cuda/test/base/exception_helpers.cu -o CMakeFiles/cuda_test_base_exception_helpers.dir/exception_helpers.cu.o
So in fact there is a -std=c++14 .

[greole]

So i found a similar issue on the arch linux bug tracker.

[thoasm]

Yes, it is the same for me. For some reason, an -std=c++14 appears when compiling. I tried to set this to c++11 by hand, but all I achieved was that both c++11 and c++14 appeared during the compilation, obviously failing right away.
Setting CXX=g++-8 (and, for completion, CC=gcc-8) works fine for all CUDA versions I have tried.

[greole]

Yes, setting CXX=g++-8 and CC=gcc-8 works. So i am going to close the issue. Thanks for your help!

[tcojean]

Thanks for checking and solving this!