Cray + NVIDIA GPU on NCSA Delta

[sbryngelson]

This is an example of how to use Cray compilers (17.0) on NCSA Delta with A100 devices for a minimal OpenACC program test.f90. It still needs to be integrated with MFC and ensured it works.

module load nvhpc
module load PrgEnv-cray

export CUDA_HOME=/sw/spack/deltas11-2023-03/apps/linux-rhel8-x86_64/gcc-8.5.0/cuda-12.3.0-7oylpzc
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}

ftn -haccel=nvidia80 -hacc -craype-verbose -M 296,878,1391,1069,5025 -h dynamic -h acc_model=auto_async_none -h acc_model=no_fast_addr -o test test.f90 -L/usr/lib64 -lcudart -L${CUDA_HOME}/lib64

[Malmahrouqi3]

Will look into this tonight

[Malmahrouqi3]

I have partitions on Delta for A40, A100, H200. I am not yet sure what is the end goal. I would appreciate more info on this issue.
Do you wanna make these commands part of standard ./mfc.sh load d??

Something like these changes: Malmahrouqi3@2573c90

[Malmahrouqi3]

Kinda odd warning while building mfc

CMake Warning at CMakeLists.txt:482 (find_package):
  By not providing "FindcuTENSOR.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "cuTENSOR",
  but CMake did not find one.

  Could not find a package configuration file provided by "cuTENSOR" with any
  of the following names:

    cuTENSORConfig.cmake
    cutensor-config.cmake

  Add the installation prefix of "cuTENSOR" to CMAKE_PREFIX_PATH or set
  "cuTENSOR_DIR" to a directory containing one of the above files.  If
  "cuTENSOR" provides a separate development package or SDK, be sure it has
  been installed.
Call Stack (most recent call first):
  CMakeLists.txt:562 (MFC_SETUP_TARGET)


CMake Warning at CMakeLists.txt:484 (message):
  Failed to locate the NVIDIA cuTENSOR library.  MFC will be built without
  support for it, disallowing the use of cu_tensor=T.  This can result in
  degraded performance.
Call Stack (most recent call first):
  CMakeLists.txt:562 (MFC_SETUP_TARGET)

[sbryngelson]

I found a better way to do this but it's more complicated. I need to put the description of how to do it in this GH issue. The above way isn't really reasonable, it would need many changes to the cmake file and it isn't worth it. I have a workaround that does the trick without many changes. Has to be on DeltaAI though.

[sbryngelson]

But, at the moment on DeltaAI we cannot have runners. So, we might make it an option (so that users can do it) by modifying the necessary templates and such.

The goal of the issue is to make it so people can use cray compilers to offload to NVIDIA GPU devices and ensure their code works. Many folks don't have access to Frontier for Cray compilers with AMD GPU offloading, but DeltaAI is broadly available and had Cray compilers + NV GPUs which is a helpful debugging point.