-
Notifications
You must be signed in to change notification settings - Fork 577
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide Trilinos-global CMake options to make it easier for apps to disable float & complex Scalar types #362
Comments
In Tpetra, Imagine that a supercomputer software vendor wants to support Trilinos. Do you think they will ask us how to enable complex Scalar types? Of course not. They will configure with the minimal settings to make it build, then install the thing. Users then get Applications can and do disable types that they don't need. Also, we could always make Tpetra's ETI work like the "ETI" in KokkosKernels, where any Scalar type works, but some types get pre-compiled kernels by default. The new I like your second feature request. A while back, I was trying to add |
I am not sure that we want to change what types are enabled by default based on a debug build or not. I think that would confuse users (I would be surprised and confused by this).
I disagree. We make it clear in the Trilinos/INSTALL document that by default you only get support for If super computing centers are installing Trilinos in a vacuum (i.e. not talking to Trilinos developers or Trilinos users), then people end up ignoring the default installed version of Trilnos anyway. How many people do you know who use the default installed version of Trilinos on these machines? Currently, if they don't enable enough TPLs, then people ignore the default installed version of Trilinos. Hopefully with Spack that will change (and we can make sure that Trilinos is configured appropriately with Spack which will include if float and complex types are enabled or not).
Close application customers are not a problem either way. We just tell them how to configure and they do it. The issue is new potential customers that are getting ridiculously long build times by default. That is the issue.
Not sure what
I don't have any major objections to this but Tpetra is just an optional dependency for Thyra so it might take some work to get this to behave correctly. I think we would really rather have the selection of Scalar (and Ordinal) types specified by default at the global level and then the various Trilinos packages should use those, as described above for float and complex types. We should discuss this at the Trilinos Leaders Meeting today. |
I'm renaming this, because not everyone agrees about default behavior. We agree that such options should be shared among Trilinos packages (in particular between Thyra, Tpetra, and their downstream packages). We also agree that it should be easy for installers and applications to change the settings. |
Everyone does not have to agree, but we still need to define default behavior. We will discuss this at the next Trilinos leaders meeting but I will argue that the massive build times for Trilinos should make the default to exclude float and complex types in the default configure of Trilinos. |
I added blocking relationship with #158 since I don't think we can test such a change without building almost all of Trilinos (which is very hard to do right now). |
Should we let users specify a list of enabled e.g., Scalar types, rather than saying Trilinos_ENABLE_FLOAT etc.? This could get confusing, e.g., if users want to enable Scalar=int (does Trilinos_ENABLE_INT mean Scalar, LocalOrdinal, or GlobalOrdinal?). |
At the Trilinos Leaders meeting today, we discussed this if we should enabled float and complex by default or not. The consensus decision was to disable float and complex by default but make it easy and obvious for how to enable them. Now the question is how to enable them?
That is not a bad idea. I might suggest:
(the type So above, if the user specified Then we could do something similar to for GlobalOrdinal andLocalOrdinal:
These types of CMake arguments would require more CMake work to parse this (i.e.. Comments, opinions? |
I think the lists approach is the right way to go. I have a couple questions:
|
I think @bartlettroscoe was just abbreviating -- it would make sense for the list to be parsed as if it occurred in the global namespace with no |
I think we could allow the CMake parsing to be smart enough to allow The bigger question is if this will also support addon user-defined types? That is a another type of requirement that I would rather address in another Story ticket. I would guess that given the current scope, this should be about a days work (i.e. 8 dev hours) including automated tests, full manual testing on Trilinos, documentation, etc. But this is the easiest thing that we can do that will result in a dramatic reduction in default build times (which we should time). But before we start this, we should do #158 to make this easy to test. |
I agree that we don't have to go there for now. Any type that requires including a header file would not work without further modifications. Built-in but nonstandard types that don't require including a header file (or already have it included) should work fine. This includes GCC's |
Let's make users spell this out for us. How do we know the difference between std::complex and Kokkos::complex otherwise? |
A lot of the build time stuff is not ETI-related, it's that we instantiate too much code. To me, this issue is about simplifying Tpetra - Thyra - Stratimikos integration, and improving the user experience. |
I think for the purposes of a global option to enable complex support, it should be just I am actually about to implement this soon since this will be important to create a more efficient pre-push testing process. |
Regardless what we use as the final user interface, we are going to want simple CMake variables like
That should be fast and easy to implement for now. |
Just a comment to Kokkos vs std complex: std:: complex doesn't work in parallel environments well, since it a) doesn't exist on GPUs, and b) misses volatile overloads of its operators which prevents things like Kokkos reductions to work. What we did with Tpetra is that you request std::complex, but internally will get Kokkos::complex. The two types are interoperable (i.e. assignable and promotable to each other). |
Responding to @crtrott in above comment:
I think that means that it makes no sens to say that you are supporting "Kokkos::complex" and "std::complex". It will just be "complex" and the Kokkos::complex will be used where it needs to be and std::complex can be used else where. I that a fair assessment? |
Yeah that is pretty much what I think about this. Kokkos complex is not meant as an alternative to std::complex, it is meant to supplement it since std::complex has deficiencies which would prevent its usage throughout the threaded Trilinos stack. |
Right, so that argues that my recommendation in the above comment to use:
It makes no sense to differentiate "std::complex" from "complex" because Trilinos will pick the right complex impl for the given situation. The user would interact with Kokkos::complex or std::complex or a C-struct or Fortran equivalent depending on their situation. It is just "complex". |
Agreed. |
Now that complex is disbaled by default (trilinos#362), we need to enable it in some nightly testing. It looks like there are several Trilinos builds on this machine so this is a good place to start.
Now there are top-level options for float and complex and these are off by default as per the conversation and decision documented in trilinos#362. This makes it easy to enable and disable support for these types. Some work needs to be done to get some of the packages to allow enabling and disabling float, std:complexfloat> and std::complex<double> somewhat independently but that can be done later. Really only Tpetra allows this correctly. Also, these are disabled by default to speed up the default build of Trilinos and very few customers use or need float and std::complex types. As a matter of fact, float support in Tpetra and downstream is currently broken. This is likely due to the fact that Tpetra unconditionally disabled float by default and it is doubtful that anyone has tested this in a long time (and there are no automated tests that test float support). I fixed the logic for Tpetra complex<double> support in Anasazi so that Tpetra (take a look at the updated logic). I also fixed some typos in documentation.
…os#362) This is a bit of a hack since Teuchos is not properly respond to Trilinos_ENABLE_COMPLEX_FLOAT. I noticed this while working on trilinos#362.
…os#482, trilinos#362) Until we get get the CI server changed over to the new PT server and get a bunch of other Nighty tests to turn on complex, we want this CI build to at least test complex.
Now that complex is disbaled by default (trilinos#362), we need to enable it in some nightly testing. It looks like there are several Trilinos builds on this machine so this is a good place to start.
This has been merged to Trilinos 'develop' branch for a long time. In am not going to bother with the task
Closing as complete. |
…trilinos#10636, trilinos#362) This now keys off of Trilinos_ENABLE_COMPLEX_DOUBLE and Trilinos_ENABLE_COMPLEX_FLOAT to make it so that: -D Trilinos_ENABLE_FLOAT=ON \ -D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \ causes the Teuchos ETI macros in Teuchos_ExplicitInstantiationHelpers.hpp to do instatiations for complex<double> but not complex<float>. More details: * Added new cache vars Teuchos_INST_FLOAT, Teuchos_INST_COMPLEX_FLOAT, and Teuchos_INST_COMPLEX_DOUBLE with identical defaults to Tpetra versions of these vars and then used these vars to determine instantations in Teuchos_ExplicitInstantiationHelpers.hpp and teuchos/numerics/src/Teuchos_ExpandScalarTypeMacros.hpp. To make setting Teuchos_INST_COMPLEX_FLOAT consistent, it must take its default value from Trilinos_ENABLE_COMPLEX_FLOAT if that var is non-empty. * Removed duplicate warnings coming from macros in Teuchos_ExplicitInstantiationHelpers.hpp being redefined in Teuchos_ExpandScalarTypeMacros.hpp. (I made the latter include the former.) * Removed assert_defined() calls for Trilinos_ENABLE_COMPLEX_FLOAT and Trilinos_ENABLE_COMPLEX_DOUBLE so at least there is a chance that we can still build Teuchos as its own CMake project.
…plex<float> not supported (trilinos#10636, trilinos#362) That test fails if you try to run it with complex<double> ETIs enabled but not complex<float>. (There is no existing nightly or PR build of Trilinos that matches that configuration yet it is the configuration for the most important Trilinos customer, Sierra.)
…trilinos#10635, trilinos#362) This now keys off of Trilinos_ENABLE_COMPLEX_DOUBLE and Trilinos_ENABLE_COMPLEX_FLOAT to make it so that: -D Trilinos_ENABLE_FLOAT=ON \ -D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \ causes the Teuchos ETI macros in Teuchos_ExplicitInstantiationHelpers.hpp to do instatiations for complex<double> but not complex<float>. More details: * Added new cache vars Teuchos_INST_FLOAT, Teuchos_INST_COMPLEX_FLOAT, and Teuchos_INST_COMPLEX_DOUBLE with identical defaults to Tpetra versions of these vars and then used these vars to determine instantations in Teuchos_ExplicitInstantiationHelpers.hpp and teuchos/numerics/src/Teuchos_ExpandScalarTypeMacros.hpp. To make setting Teuchos_INST_COMPLEX_FLOAT consistent, it must take its default value from Trilinos_ENABLE_COMPLEX_FLOAT if that var is non-empty. * Removed duplicate warnings coming from macros in Teuchos_ExplicitInstantiationHelpers.hpp being redefined in Teuchos_ExpandScalarTypeMacros.hpp. (I made the latter include the former.) * Removed assert_defined() calls for Trilinos_ENABLE_COMPLEX_FLOAT and Trilinos_ENABLE_COMPLEX_DOUBLE so at least there is a chance that we can still build Teuchos as its own CMake project.
…plex<float> (trilinos#10635, trilinos#362) This is to make the instantiations consistent with (updated) Teuchos and Tpetra so that if you configure with: -D Trilinos_ENABLE_FLOAT=ON \ -D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \ you will get consistent instantations and tests. Specific changes: * Removed HAVE_THYRA_TEUCHOS_BLASFLOAT, HAVE_THYRA_FLOAT, and HAVE_THYRA_COMPLEX and go with Teuchos versions of these. * Fix build errors for some float tests (Shows that float testing has not been enabled in a long time?) * Loosen tolerance for some float tests so they pass. With this set of changes, the full Thyra test suite passes for the above configure arguments (with GCC 7.2.0).
…plex<float> not supported (trilinos#10635, trilinos#362) That test fails if you try to run it with complex<double> ETIs enabled but not complex<float>. (There is no existing nightly or PR build of Trilinos that matches that configuration yet it is the configuration for the most important Trilinos customer, Sierra.)
…plex<float> (trilinos#10635, trilinos#362) This is to make the instantiations consistent with (updated) Teuchos and Tpetra so that if you configure with: -D Trilinos_ENABLE_FLOAT=ON \ -D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \ you will get consistent instantations and tests. Specific changes: * Removed HAVE_THYRA_TEUCHOS_BLASFLOAT, HAVE_THYRA_FLOAT, and HAVE_THYRA_COMPLEX and go with Teuchos versions of these. * Fix build errors for some float tests (Shows that float testing has not been enabled in a long time?) * Loosen tolerance for some float tests so they pass. With this set of changes, the full Thyra test suite passes for the above configure arguments (with GCC 7.2.0).
…plex<float> not supported (trilinos#10635, trilinos#362) That test fails if you try to run it with complex<double> ETIs enabled but not complex<float>. (There is no existing nightly or PR build of Trilinos that matches that configuration yet it is the configuration for the most important Trilinos customer, Sierra.)
…plex<float> (trilinos#10635, trilinos#362) This is to make the instantiations consistent with (updated) Teuchos and Tpetra so that if you configure with: -D Trilinos_ENABLE_FLOAT=ON \ -D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \ you will get consistent instantations and tests. Specific changes: * Removed HAVE_THYRA_TEUCHOS_BLASFLOAT, HAVE_THYRA_FLOAT, and HAVE_THYRA_COMPLEX and go with Teuchos versions of these. * Fix build errors for some float tests (Shows that float testing has not been enabled in a long time?) * Loosen tolerance for some float tests so they pass. With this set of changes, the full Thyra test suite passes for the above configure arguments (with GCC 7.2.0).
…plex<float> not supported (trilinos#10635, trilinos#362) That test fails if you try to run it with complex<double> ETIs enabled but not complex<float>. (There is no existing nightly or PR build of Trilinos that matches that configuration yet it is the configuration for the most important Trilinos customer, Sierra.)
…trilinos#10635, trilinos#362) This now keys off of Trilinos_ENABLE_COMPLEX_DOUBLE and Trilinos_ENABLE_COMPLEX_FLOAT to make it so that: -D Trilinos_ENABLE_FLOAT=ON \ -D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \ causes the Teuchos ETI macros in Teuchos_ExplicitInstantiationHelpers.hpp to do instatiations for complex<double> but not complex<float>. More details: * Added new cache vars Teuchos_INST_FLOAT, Teuchos_INST_COMPLEX_FLOAT, and Teuchos_INST_COMPLEX_DOUBLE with identical defaults to Tpetra versions of these vars and then used these vars to determine instantations in Teuchos_ExplicitInstantiationHelpers.hpp and teuchos/numerics/src/Teuchos_ExpandScalarTypeMacros.hpp. To make setting Teuchos_INST_COMPLEX_FLOAT consistent, it must take its default value from Trilinos_ENABLE_COMPLEX_FLOAT if that var is non-empty. * Removed duplicate warnings coming from macros in Teuchos_ExplicitInstantiationHelpers.hpp being redefined in Teuchos_ExpandScalarTypeMacros.hpp. (I made the latter include the former.) * Removed assert_defined() calls for Trilinos_ENABLE_COMPLEX_FLOAT and Trilinos_ENABLE_COMPLEX_DOUBLE so at least there is a chance that we can still build Teuchos as its own CMake project.
…plex<float> (trilinos#10635, trilinos#362) This is to make the instantiations consistent with (updated) Teuchos and Tpetra so that if you configure with: -D Trilinos_ENABLE_FLOAT=ON \ -D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \ you will get consistent instantations and tests. Specific changes: * Removed HAVE_THYRA_TEUCHOS_BLASFLOAT, HAVE_THYRA_FLOAT, and HAVE_THYRA_COMPLEX and go with Teuchos versions of these. * Fix build errors for some float tests (Shows that float testing has not been enabled in a long time?) * Loosen tolerance for some float tests so they pass. With this set of changes, the full Thyra test suite passes for the above configure arguments (with GCC 7.2.0).
…plex<float> not supported (trilinos#10635, trilinos#362) That test fails if you try to run it with complex<double> ETIs enabled but not complex<float>. (There is no existing nightly or PR build of Trilinos that matches that configuration yet it is the configuration for the most important Trilinos customer, Sierra.)
…s:develop' (816e7eb). * trilinos-develop: Xpetra: Use as<Scalar>() to fix complex<float> tests (trilinos#10635) Stratimikos: Exclude test Galeri_double_Jacobi_HalfPrecision when complex<float> not supported (trilinos#10635, trilinos#362) Thyra: Use Teuchos vars to consistent ETI including for float and complex<float> (trilinos#10635, trilinos#362) Teuchos: Support ETI macros do complex<double> without complex<float> (trilinos#10635, trilinos#362) name default mpi tag constants Remove ifdefs for SUN_CXX (trilinos#10636) Tpetra: Get rid of unnecessary function on DistributorActor Tpetra: Strip out remainder of deprecated useDistinctTags usage Tpetra: remove the no-longer-necessary DistributorPlan::getTag Tpetra: use DistributorActor's tag for all communication in the actor Tpetra: change pathTags so Plan's is 0 Tpetra: add missing include in DistributorActor Tpetra: Add a variable to store the MPI tag for DistributorActor
…s:develop' (816e7eb). * trilinos-develop: Xpetra: Use as<Scalar>() to fix complex<float> tests (trilinos#10635) Stratimikos: Exclude test Galeri_double_Jacobi_HalfPrecision when complex<float> not supported (trilinos#10635, trilinos#362) Thyra: Use Teuchos vars to consistent ETI including for float and complex<float> (trilinos#10635, trilinos#362) Teuchos: Support ETI macros do complex<double> without complex<float> (trilinos#10635, trilinos#362) name default mpi tag constants Remove ifdefs for SUN_CXX (trilinos#10636) Tpetra: Get rid of unnecessary function on DistributorActor Tpetra: Strip out remainder of deprecated useDistinctTags usage Tpetra: remove the no-longer-necessary DistributorPlan::getTag Tpetra: use DistributorActor's tag for all communication in the actor Tpetra: change pathTags so Plan's is 0 Tpetra: add missing include in DistributorActor Tpetra: Add a variable to store the MPI tag for DistributorActor
Next Action Status:
Pushed to 'develop' branch as part of #482. Complete!
CC: @trilinos/framework, @trilinos/tpetra
Blocked By: #158
Blocking: #482
Description:
By default, in release and development mode, Trilinos provides support for float and complex (
std::complex<float>
andstd::complex<double>
) scalar types. When ETI is used, this massively increases the compile and link times for Trilinos. Most Trilihnos users don't even need float or complex scalar types.Currently, to disable float and complex , the user has to set a bunch of separate Teuchos and Tpetra options. Very few users know what options these are so they get long build times by default and no idea how to disable them.
This story is to change the default for Trilinos to not build float or complex types by default. This story will also provide top level build options
Trilinos_ENABLE_FLOAT
andTrilinos_ENABLE_COMPLEX
(andTrilinos_ENABLE_COMPLEX_FLOAT
andTrilinos_ENABLE_COMPLEX_DOUBLE
with obvious defaults based onTrilinos_ENABLE_FLOAT
andTrilinos_ENABLE_COMPLEX
) that all Trilinos packages that currently have their own variables will respond to.Definition of Done:
-DTrilinos_ENABLE_FLOAT=ON
and/or-DTrilinos_ENABLE_COMPLEX=ON
, respectively (or something similar).Tasks:
Trilinos_ENABLE_FLOAT
,Trilinos_ENABLE_COMPLEX
,Trilinos_ENABLE_COMPLEX_FLOAT
andTrilinos_ENABLE_COMPLEX_DOUBLE
and make Teuchos, Tpetra, Thyra, and other packages set their defaults based on this. (These should be Trilinos repository level options so that they can be set in meta projects like CASL VERA, MPACT, SCALE, as well) [Done]Trilinos_ENABLE_FLOAT
,Trilinos_ENABLE_COMPLEX
,Trilinos_ENABLE_COMPLEX_FLOAT
andTrilinos_ENABLE_COMPLEX_DOUBLE
in a few automated Trilinos builds ... Not going to do this in this storyThe text was updated successfully, but these errors were encountered: