# Copyright (c) 1993-2015, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#  * Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#  * Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#  * Neither the name of NVIDIA CORPORATION nor the names of its
#    contributors may be used to endorse or promote products derived
#    from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

====================================================
Description document for the GPU-based Jacobi solver
====================================================

Contents:
---------
1)	Application overview
2)	Build instructions
3)	Run instructions
4)	Documentation


=======================
1) Application overview
=======================

This is a distributed Jacobi solver, using GPUs to perform the computation and MPI for halo exchanges.
It uses a 2D domain decomposition scheme to allow for a better computation-to-communication ratio than just 1D domain decomposition.
All sources for the Jacobi solver can be found in the "src" folder.
They have the following roles:

a) 	Jacobi.h 			- the main header, containing configuration parameters and prototypes of the most important functions
b)	Jacobi.c 			- the application entry point
c)	Input.c				- the command-line argument parser
d)	Host.c	 			- the functions covering host-side processing, including the main Jacobi loop and data exchanges
e)	Device.cu			- the device (GPU) kernels and the host wrappers for these kernels
f)	CUDA_Normal_MPI.c	- the functions managing data exchange through normal MPI (i.e. using intermediate host buffers)
g)	CUDA_Aware_MPI.c	- the functions managing data exchange through CUDA-aware MPI (i.e. without intermediate host buffers)

The flow of the application is as follows:

a)	The MPI environment is initialized (and the desired CUDA device is selected, when using CUDA-aware MPI)
b)	The command-line arguments are parsed
c)	Resources (including host and device memory blocks, streams etc.) are initialized
d)	The Jacobi loop is executed; in every iteration, the local block is updated and then the halo values are exchanged; the algorithm
	converges when the global residue for an iteration falls below a threshold, but it is also limited by a maximum number of
	iterations (irrespective if convergence has been achieved or not)
e)	Run measurements are displayed and resources are disposed

The application uses the following command-line arguments:
a)	-t x y		-	mandatory argument for the process topology, "x" denotes the number of processes on the X direction (i.e. per row) and
					"y" denotes the number of processes on the Y direction (i.e. per column); the topology size must always match the number of
					available processes (i.e. the number of launched MPI processes must be equal to x * y)
b)	-d dx dy 	-	optional argument indicating the size of the local (per-process) domain size; if it is omitted, the size will default to
					DEFAULT_DOMAIN_SIZE as defined in "Jacobi.h"
c)	-fs 		-	optional argument indicating that the replacement of the old block with the new one after an update should be performed using
					a fast pointer swap rather than a full block copy (which is the default behavior)
d)	-h | --help	-	optional argument for printing help information; this overrides all other arguments

=====================
2) Build instructions
=====================

To build the application, please ensure that the following are available on the platform:

a) an MPI implementation (with optional support for CUDA-aware MPI if this version of the application is to be built)
b) a CUDA toolkit (preferably, the latest available)

You can build the CUDA-aware MPI an the normal MPI version by calling make in src

cd src
make

To find mpi.h and the CUDA runtime library the provied Makefile relies on CUDA_INSTALL_PATH and MPI_HOME beeing set correctly. If you are using CUDA 5 or newer and running on a device with compute capability 3.0 or 3.5 you should also add GENCODE_SM30 and GENCODE_SM35 to the GENCODE_FLAGS in the Makefile. Also the macro ENV_LOCAL_RANK might need to
be changed in Jacobi.h to handle the GPU affinit properly. It defaults to MV2_COMM_WORLD_LOCAL_RANK which works with MVAPICH2.
The generated binaries can then be found found the bin directory.

===================
3) Run instructions
===================

To run the normal MPI version use: 
mpiexec -np 2 ./jacobi_cuda_normal_mpi -t 2 1

To run the CUDA-aware MPI version depending on the MPI implementation you are using you need to activate the CUDA-aware feature, e.g. for MVAPICH2 use
MV2_USE_CUDA=1 mpiexec -np 2 --exports=MV2_USE_CUDA ./jacobi_cuda_aware_mpi -t 2 1
 
================
4) Documentation
================

Documentation for this project may be generated automatically using Doxygen by calling make doc. A configuration file for this may be found in the "src" folder. If 
Doxygen is not available, the doc folder also contains pregenerated documentation.