Address "Error 127" when trying to build libs with ETI and COMPLEX enabled

[bartlettroscoe]

CC: @trilinos/framework, @trilinos/kokkos, @trilinos/kokkos-kernels

Next Action Status

Waiting to resolve all of the issues with CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON (e.g. #2115) before trying to automate turning this on ...

Description

This issue is to address the problem with the library creation command failing with "Error 127" with KokkosKernels with CUDA, ETI, and COMPLEX enabled described in kokkos/kokkos-kernels#141. The issue is that Kokkos and KokkosKernels create a bunch of *.cpp files, one for each function, instantiation set, and scalar type. This quickly explodes to hundreds of *.cpp files per library and therefore hundreds of *.o files when creating a library. It seems that some compilers choke on that many object files and just return "Error 127". Christian found the CMake option CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS that when set to TRUE will put the list of all of the object files into a *.rsp file that is then passed to the command to create the library. This was reported to solve the problem described in kokkos/kokkos-kernels#141.

It turns out that the option CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS is not documented in the official CMake documentation. There are just a few references to it when you Google. Apparently, this option is only meant to be set in a compiler toolchain file for a given system. However, Google searches mention this options working with Linux GCC compilers, Windows Visual Studio compilers, and MAC OS-X. Therefore, we might hope that it is somewhat portable.

The proposed solution that we will attempt to implement in this Issue is to put in automatic logic to set CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS when ETI is enabled, COMPLEX is enabled, and KokkosKernels is enabled. I think this case works right now and only fails when CUDA is also enabled which doubles the number of *.o files (one for the host CPU, and one for the GPU). But talking with @crtrott, it seems likely that as KokkosKernels is continued to be developed that we will hit this "Error 127" soon even without CUDA enabled. Therefore, it seems like a good idea to go ahead and turn this on by default when just ETI, COMPLEX and KokkosKernels are enabled.

I will add a NOTE: to the CMake output remarking that this var is being set and then just cross our fingers that the target system will support this feature. Since there are fairly few clients that will enable ETI, COMPLEX and KokkosKernels, hopefully this will not cause a lot of trouble.

I guess to be safe, we could add a configure-time test to make sure that CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON works for the selected compilers and other options but that would be a good bit of work (since we would have to create a separate dummy CMake project to try this out and verify that a *.rsp file is getting used). In fact, if we do that, we should just always set CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON if we determine that response files are supported for the selected compilers and system. Since this will take a bit more work we can avoid doing this for now. But If we discover that this people are getting broken builds due to this logic, we can go ahead and implement the more sophisticated configure-time check and be done with this once and for all.

Related Issues

• Blocks
• Is blocked by
• Follows
• Precedes
• Related to: CMake Trilinos CUDA build fails with link error 127 kokkos/kokkos-kernels#141, KokkosKernels: build failure when enabling double, complex<double>, and shared libraries #2115
• Part of
• Composed of

[bartlettroscoe]

FYI:

I also discovered another undocumented option -DCMAKE_CXX_USE_RESPONSE_FILE_FOR_INCLUDES=ON works to move all of the -I <dir> arguments to a *.rsp file for each compile line as well (at least as of CMake 3.5.2). And few newer versions of CMake, you can use the option -DCMAKE_CXX_USE_RESPONSE_FILE_FOR_LIBRARIES=ON to do something similar for the list of libraries passed to the linker.

These three options should completely eliminate every "compile-line too long" error that even have seen in huge projects like CASL VERA that use CMake. This means that using these options that the TriBITS approach should work with the largest projects we can imagine.

[bartlettroscoe]

I did a more detailed Google search on "compiler response file" and it seems that nearly every major compiler (GCC, Intel, Clang, MS Visual C++) supports them as well as Ninja itself. But it looks like the IBM XL C++ compiler does not support *.rsp files that I can find. Therefore, this may not be a 100% portable solution. Just another reason not to use the IBM XL compiler.

[bartlettroscoe]

I am not sure we should pull the trigger on adding an automated setting of -DCMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS:BOOL=ON just yet. The experience in #2115 (comment) suggests that this feature does not work with CUDA + CMake (likely a lack of support for *.rsp files with nvcc_wrapper?).

I think we need to research resolve all of the issues -DCMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS:BOOL=ON before we try to automate anything yet.

[srajama1]

@bartlettroscoe : Is there a way to specify wildcards (*.o) instead of this response files ? I assume that will work in portable way.

[bartlettroscoe]

Is there a way to specify wildcards (*.o) instead of this response files ? I assume that will work in portable way.

@srajama1 not that I know of with CMake.

I will see if I can reproduce this failure and see what is going on.

[bartlettroscoe]

It looks like it is premature to be thinking about an automated way to set CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON given that this does not allow for a passing build with CUDA, ETI, and COMPLEX enabled (see #2115). We need to put this story on hold until we figure out how to fix that case and how urgent this really is.

[trudeaun]

@bartlettroscoe Will IBM XL compiler's option @file (-qoptfile) do what you are looking for?

[bartlettroscoe]

Will IBM XL compiler's option @file (-qoptfile) do what you are looking for?

@trudeaun,

Good to know that the XL compiler supports that ability. Unfortunately, it does not look like CMake knows about that option from grepping the CMake sources:

[rabartl@crf450 cmake ((v3.10.1))]$ find . -type f -exec grep -nH qoptfile {} \;
[empty]

We have a contract with Kitware and I will ask them to add support for that for the IBM XL compiler when CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON is set. We have an active contract with Kitware so I will create a backlog story for them to look into updating CMake to support this option.

[bartlettroscoe]

have an active contract with Kitware so I will create a backlog story for them to look into updating CMake to support this option.

I created:

• https://gitlab.kitware.com/snl/project-1/issues/48

[bartlettroscoe]

FYI: CMake 3.11.0 will support response files for the IBM XL compiler (see https://gitlab.kitware.com/cmake/cmake/merge_requests/1691). Also, newer versions of CMake should automatically set CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON when it is detected that the commandline is too long.

[bartlettroscoe]

Given that #2115 is resolved, can we close this or should we try to automatically set CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON in Trilinos or TriBITS? But future versions of CMake should automatically turn this on when it needs to so do we need to put in our own fix?

[bartlettroscoe]

But I guess we should at least document CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS=ON and the other response file CMake cache vars so that people know about them and know how to set them to solve this problem.

[mhoemmen]

@bartlettroscoe I would say, let's open a new issue that says "either set that CMake option automatically, or document that it's needed and fail the configure if it's not set."

[bartlettroscoe]

@bartlettroscoe I would say, let's open a new issue that says "either set that CMake option automatically, or document that it's needed and fail the configure if it's not set."

@mhoemmen ,

We could. But it might be just as fast to just add a section to the document:

• http://trilinos.org/docs/files/TrilinosBuildReference.html

Then we can leave this issue here in the backlog to perhaps automated this if justified.

But I suspect that very few people are doing a CUDA+ETI+COMPLEX builds and we can just point them to the documentation.

@mhoemmen,

If I write up some updated documentation on a branch could you review that real fast?

[mhoemmen]

@bartlettroscoe wrote:

If I write up some updated documentation on a branch could you review that real fast?

Yes, gladly :-) . If you need a faster response just call my office phone.

[mhoemmen]

@bartlettroscoe wrote:

But I suspect that very few people are doing a CUDA+ETI+COMPLEX builds and we can just point them to the documentation.

We want Sierra to use ETI, and Sierra always enables complex.

[bartlettroscoe]

We want Sierra to use ETI, and Sierra always enables complex.

Is Sierra doing CUDA builds? In any case, Sierra is just one customer so if we tell them about this option then they will add it. Who else would use CUDA+ETI+COMPLEX?

[mhoemmen]

@bartlettroscoe Any of various ATDM projects. External users about which I don't know.

[bartlettroscoe]

If I write up some updated documentation on a branch could you review that real fast?

Yes, gladly :-) . If you need a faster response just call my office phone.

@mhoemmen,

So much for "real fast" but can you please review the new documentation for these CMake vars at:

• https://tribits.org/doc/TribitsBuildReference.html#enabling-the-usage-of-resource-files-to-reduce-length-of-build-lines

If you find any typos or other problems, please let me know and I will fix them.

Otherwise, this will automatically update the Trilinos build reference guild:

• http://trilinos.org/docs/files/TrilinosBuildReference.html

the next time I update the snapshot of TriBITS.

[mhoemmen]

@bartlettroscoe Thanks! I just have one question about the documentation: Should users always set all three of those options the same, or might it make sense only to enable some of them?

[bartlettroscoe]

I just have one question about the documentation: Should users always set all three of those options the same, or might it make sense only to enable some of them?

@mhoemmen, please take a look at the new "NOTE" in the updated section:

• https://tribits.org/doc/TribitsBuildReference.html#enabling-the-usage-of-resource-files-to-reduce-length-of-build-lines

Does that answer your question?

[mhoemmen]

@bartlettroscoe Yes it does -- thanks Ross! This helps a lot :D

[bartlettroscoe]

Given the recent CASL experience with trying use CMAKE_CXX_USE_RESPONSE_FILE_FOR_INCLUDES=ON with gfortran, I would say that it would not be a good idea for TriBITS to try to figure out when it is safe to enable these options. We should leave this up to future versions of CMake or let the user set these to try them out.

With that, should we just reduce the scope of the Issue to what has been completed (i.e. the documentation added to TriBITS) and then close this issue?

[mhoemmen]

@bartlettroscoe I think it makes sense to close this issue. I will do so. I also opened a kokkos-kernels issue kokkos/kokkos-kernels#176 to add some CMake logic in that package, for the specific issue that the package itself creates. Thanks!