Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update trilinos integration scripts #335

Merged
merged 1 commit into from
Oct 29, 2018
Merged

update trilinos integration scripts #335

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 ...integration/white_run_jenkins_script_cuda → ...tion/white_run_jenkins_script_cuda-kepler
View file
Open in desktop
File renamed without changes.
89 changes: 89 additions & 0 deletions scripts/trilinos-integration/white_run_jenkins_script_cuda-pascal
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
#!/bin/bash -el
ulimit -c 0

KOKKOSKERNELS_BRANCH=$1
TRILINOS_UPDATE_BRANCH=$2
TRILINOS_PRISTINE_BRANCH=$3
KOKKOS_UPDATE_BRANCH=$4

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this needs a separate file ? I thought this would an i/p parameter like the four parameters here. These are few line changes, right ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make it a single file with extra parameters, but this seemed the easier path to me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, go ahead

if [ -z $KOKKOSKERNELS_BRANCH ]
then
KOKKOSKERNELS_BRANCH=develop
fi

if [ -z $TRILINOS_UPDATE_BRANCH ]
then
TRILINOS_UPDATE_BRANCH=develop
fi

if [ -z $TRILINOS_PRISTINE_BRANCH ]
then
TRILINOS_PRISTINE_BRANCH=develop
fi

module load devpack/20180521/openmpi/2.1.2/gcc/7.2.0/cuda/9.2.88
module swap openblas/0.2.20/gcc/7.2.0 netlib/3.8.0/gcc/7.2.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason not to use openblas ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a Trilinos issue where use of OpenBLAS resulted in lots of problems on White/Ride and the solution they decided upon was to use Netlib instead. I just followed the same course as the ATDM testing.

# Trilinos now requires cmake version >= 3.10.0
module swap cmake/3.9.6 cmake/3.12.3
export OMP_NUM_THREADS=8
export JENKINS_DO_CUDA=ON
export JENKINS_DO_OPENMP=OFF
export JENKINS_DO_PTHREAD=OFF
export JENKINS_DO_SERIAL=ON
export JENKINS_DO_COMPLEX=OFF

export JENKINS_ARCH="Power8,Pascal60"
export JENKINS_ARCH_CXX_FLAG="-mcpu=power8 -arch=sm_60 --expt-extended-lambda --std=c++11"
export JENKINS_ARCH_C_FLAG="-mcpu=power8"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this manual setup for cflags necessary ? it should be set from cmake script ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how/if the KOKKOS_ARCH info propogates to the cflags, this may be a remnant that can be removed but was in the scripts when Christian updated them after introducing KOKKOS_ARCH so I left it in.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guessed so. However, as this is trilinos integration test script, those environment variable should be caught from trilinos cmake script. If we provide correct environment, then we cannot test trilinos configure is right or not. While integrating trilinos, this problem is not important but it would be good to be addressed in my opinion.

export BLAS_LIBRARIES="${BLAS_ROOT}/lib/libblas.a;gfortran;gomp"
export LAPACK_LIBRARIES="${LAPACK_ROOT}/lib/liblapack.a;gfortran;gomp;m"

export JENKINS_DO_TESTS=ON
export JENKINS_DO_EXAMPLES=ON

export QUEUE=rhel7G
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it used to be 7F and now it is 7G. I like it.


module load python

export KOKKOSKERNELS_PATH=${PWD}/kokkos-kernels

#Already done:
if [ ! -d "${KOKKOSKERNELS_PATH}" ]; then
git clone https://github.com/kokkos/kokkos-kernels ${KOKKOSKERNELS_PATH}
fi


cd ${KOKKOSKERNELS_PATH}
git checkout $KOKKOSKERNELS_BRANCH
git pull
cd ..

if [ -n $KOKKOS_UPDATE_BRANCH ]
then
export KOKKOS_PATH=${PWD}/kokkos

if [ -z $KOKKOS_UPDATE_BRANCH ]
then
KOKKOS_UPDATE_BRANCH=develop
fi

#Already done:
if [ ! -d "${KOKKOS_PATH}" ]; then
git clone https://github.com/kokkos/kokkos ${KOKKOS_PATH}
fi

cd ${KOKKOS_PATH}
git checkout $KOKKOS_UPDATE_BRANCH
git pull
cd ..
fi

export CUDA_LAUNCH_BLOCKING=1
export CUDA_MANAGED_FORCE_DEVICE_ALLOC=1

source ${KOKKOSKERNELS_PATH}/scripts/trilinos-integration/prepare_trilinos_repos.sh $TRILINOS_UPDATE_BRANCH $TRILINOS_PRISTINE_BRANCH $KOKKOS_UPDATE_BRANCH

export OMPI_CXX=${TRILINOS_UPDATED_PATH}/packages/kokkos/bin/nvcc_wrapper

${TRILINOS_UPDATED_PATH}/sampleScripts/Sandia-SEMS/run_repo_comparison_lsf ${TRILINOS_UPDATED_PATH} ${TRILINOS_PRISTINE_PATH} ${TRILINOS_UPDATED_PATH}/sampleScripts/Sandia-SEMS/configure-testbeds-jenkins-all TestCompare ${QUEUE}