BLAS underscore mangling

[theguruat12]

With XL, there is no underscore added for Fortran calls from C. Kokkos-Kernels appears to hard-code this with one underscore (which works for most compilers).

Attached is the patch that I had to implement to get it to build on the miniature version of the Sierra platform (Essentially just changing the hard-coding to not have underscores; this only changed the symbols that were referenced in my project, it will be necessary to change them for all of the externs). I would like a configure-time option to choose the mangling, just like Trilinos does (I think it's in Teuchos).

Patch:
0001-KokkosKernels-Change-BLAS-to-no-underscores.patch.txt

[srajama1]

@kyungjoo-kim @ndellingwood Can you make this a PR ?

[ndellingwood]

@srajama1 based on the comment it looks this is a good start but more work will be necessary for a complete fix. I can help, but would like to talk with you and @kyungjoo-kim more about what is involved to support XL and not break compatibility with other compilers. Adding reference to #307 and trilinos/Trilinos#2850 (closed issues with follow-ons) that have discussion that may be relevant.

[kyungjoo-kim]

This may be okay for xl but it is not okay with other compilrs like intel and gcc. To do this more correctly, we need to detect fortran mangling from the compilation level (like trilinos does). CMake can do this but I dont know how to. Anyone CMake grue might know how to. We also need help from @crtrott for the stand-alone makefile.

[mhoemmen]

@kyungjoo-kim Teuchos uses some CMake logic for this. Tpetra used to have an improved version of that CMake logic, until y'all deleted the GEMV and GEMM wrappers. If you want me to help, I'll be happy to, but please send me an e-mail first asking specifically for what you want.

[kyungjoo-kim]

@mhoemmen I appreciate if you can help for CMake. I am not sure if kokkoskernels stand alone cmake logic completes or not. @srajama1 do we have a stand alone cmake to test now ? If the work is already merged, we can start to moify the cmake logic.

[mhoemmen]

@kyungjoo-kim Please send me an e-mail requesting this feature. I will begin at that time. Thanks!

[theguruat12]

@kyungjoo-kim @mhoemmen I apologize, I should have been more clear. What you have suggested IS what I would like kokkos to do (properly select the underscore mangling).

[mhoemmen]

@theguruat12 reminded me that kokkos-kernels has a customer that needs this to work for the Makefile-based build system as well. This shouldn't be hard -- just specify the mangling macro definition on the command line as a -D. It's also a lower priority.

[mhoemmen]

@kyungjoo-kim @ndellingwood I have a fix in Trilinos, in the Fix-KokkosKernels-BLAS-mangling branch (here, not in my fork). It works fine with the CMake-based build system. I made an attempt to fix it for the Makefile-based build system too, but I don't know how to test that. Please take a look and see if it's OK. If you think it's OK, I'll make this a kokkos-kernels PR and simultaneously a Trilinos PR.

Here is a link to the branch: https://github.com/trilinos/Trilinos/compare/Fix-KokkosKernels-BLAS-mangling

[ndellingwood]

@mhoemmen I'd like to get @kyungjoo-kim 's feedback on the PR, I can help work on testing the Makefile-based changes etc.

[mhoemmen]

Thanks @ndellingwood ! It really helps that we require a recent CMake; that means I can use entirely native CMake functions.

[mhoemmen]

Thanks @ndellingwood for your investigations!

[kyungjoo-kim]

Thank you for working on this. It seems to be quite easy to just include FortranCInterface. I am okay to merge your branch to both trilinos and kokkoskernels. We will make sure the Makefile part once your submit the PR to the kokkoskernels repo. I don't really have a good idea how to detect fortran mangling from a stand alone makefile (I usually see to find fortran mangling from autoconf or cmake only). At this moment, I would just take user input regarding to the fortran mangling for the stand alone make file to generate the the mangling header file. Any better way ?

[mhoemmen]

@kyungjoo-kim wrote:

I don't really have a good idea how to detect fortran mangling from a stand alone makefile

My plan was to have sane defaults and let users override by defining Makefile variables themselves. That's what I did in the branch. I'm just not sure how to test it.

[kyungjoo-kim]

@mhoemmen Thanks. If you don't mind, would you please submit PR for both KK and Trilinos ? From there I can take care and test it.

[mhoemmen]

I'll do Trilinos first today since Sierra needs it for the next Trilinos promotion.

[mhoemmen]

@kyungjoo-kim Here is the Trilinos patch: trilinos/Trilinos#4228

[kyungjoo-kim]

@theguruat12 Do you have this problem when you compile kokkoskernels with Trilinos ? Or kokkoskernels is used as stand alone ?

[kyungjoo-kim]

@theguruat12 Another question. According to XL programming guide, page 90, XL provides blas library with C interface. This is different from IBM ESSL library (this is fortran library).

So, I am wondering if you mean that the C-interface provided from the XL compiler (-lxopt) when you say XL uses no underscore blas. If so, trilinos/Trilinos#4228 won't fix your problem as it is not a problem of fortran mangling.

If XL indeed uses no-underscore fortran mangling, the current Trilinos won't identify such fortran mangling, either. Thus, Trilinos will not work either in that case.

[nmhamster]

@kyungjoo-kim - on Page 90 it says that they only ship dgemm, sgemm, sgemv and dgemv in libxlopt. I imagine you want more functions than that? XL usually does not add underscores Fortran name mangling.

[kyungjoo-kim]

@nmhamster I don't really mind about the functionalities but I just want to double check what is real problem in this issue.

When I googling examples of xlc with blas, it uses a single underbar interface, which conflicts this issue.

So, I want to assure that XL indeed uses no-underbar fortran mangling.

[nmhamster]

Example:

SUBROUTINE myFunction

   PRINT *, "Hello"

END SUBROUTINE

When compiled with XL (lower case, no underscore):

0000000000000000 T myfunction

When compiled with gfortran (lower case, with underscore):

0000000000000000 T myfunction_

[kyungjoo-kim]

@nmhamster Ah................................ sounds like the problem is quite complicated. So the example I googled uses blas library complied by another compiler as it uses a single underscore.

So, our IBM machines are equipped with OpenBLAS, LAPACK or ESSL compiled by XL, right ? Due to the different fortran interface, it might be also problematic with other third party libraries (if the TPLs uses BLAS internally and the code does not correctly deduce fortran interface).

Ah.............. have you compiled trilinos with XL compiled BLAS ? I am wondering this because Trilinos fortran mangling choose between single and double underscores if the fortran compiler is not enabled in the configuration and this is how trilinos is built for ATDM apps.

[sebrowne]

@kyungjoo-kim The one I'm attempting to use was compiled with XL, thus, no underscores.

For this use case, only the CMake (as part of Trilinos) build is needed.

[kyungjoo-kim]

@sebrowne Thank you.

[trudeaun]

Sorry that this is coming in late, but I wanted to make sure you saw that IBM XL Fortran offers a compiler option called -qextname, which adds an underscore to the end of global names, making it behave like gfortran. This option is not on by default. You can learn more about it on Knowledge Center.

[sebrowne]

@trudeaun , Thank you for the information. In this case, since I'm compiling against an existing version of BLAS, I do still need the functionality of dynamically determining the mangling, but yes, if I were to compile all TPLs (including BLAS) myself, I could specify that option.