Skip to content

Commit

Permalink
Remove 2D and packing version from gcl (#1233)
Browse files Browse the repository at this point in the history 
This PR removes untested and unused code from GCL, namely the 2D cases (they can be implemented by reducing the third dimension of a 3D case to 1), the packing version (only the manual is tested and used) and the process grid type.

API changes:
- `halo_exchange_dynamic_ut` does not take the packing version and the process grid type as template arguments
- `halo_exchange_generic` does not take the number of dimensions and the packing versions as template argument
  • Loading branch information
@mbianco @havogt
mbianco authored and havogt committed Mar 28, 2019
1 parent 61c415e commit 412eba6
Show file tree
Hide file tree
Showing 28 changed files with 111 additions and 3,647 deletions.
20 changes: 11 additions & 9 deletions docs_src/manuals/user_manual/halo_exchanges.hrst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ distribution like in :numref:`fig_dist1`.
:scale: 40%

Example data distribution among two processes.

In this case the map between data and the processor grid is:

.. code-block:: gridtools
Expand Down Expand Up @@ -162,20 +162,21 @@ the name of arguments should be self-explanatory:
GCL Communication Module
-------------------------

Now we are ready to describe the :term:`Halo Exchange` patterns objects. The first one is ``halo_exchange_dynamic_ut``. The ``ut`` suffix stands for ``uniform types``, meaning that the data fields that this object will manage must all store the same value types, that are declared at instantiation time. The type of the object is defined as in this example:
Now we are ready to describe the :term:`Halo Exchange` patterns objects. The first one is ``halo_exchange_dynamic_ut``. The ``ut`` suffix stands for ``uniform types``, meaning that the data fields that this object will manage must all store the same value types, that are declared at instantiation time. The domain decomposition goes up to three dimensions and the data to be exchanged contained in 3 dimensional arrays (lower dimensions can be handled by setting the missing dimensions to 1). Being designed for three dimensional data, the layout maps have three elements (refer to :numref:storage-info for more information).

The type of the object is defined as in this example:

.. code-block:: gridtools

using pattern_type = halo_exchange_dynamic_ut<layout_map<0, 1, 2>,
ayout_map<0, 1, 2>, value_type, 3, gcl_cpu>;
layout_map<0, 1, 2>, value_type, gcl_cpu>;

The template arguments are:

1. the layout if the data;
2. the mapping between the data dimensions and processing :term:`Grid`, as described above (leave it as ``layout_map<0, 1, 2>`` if in doubt);
3. the type of the values to be exchanged;
4. the number of dimensions of the data/processing grid, and it has to be set to 3 (the GCL was designed to work with other dimensionalities, but the version in |GT| is a stripped down version waiting for the next more general interfaces);
5. the place where the data lives and for which the code is optimized. The options for this arguments are ``gcl_gpu``, ``gcl_cpu`` and ``gcl_mc`` (not supported yet).
4. the place where the data lives and for which the code is optimized. The options for this arguments are ``gcl_gpu``, ``gcl_cpu`` and ``gcl_mc`` (not supported yet).

The :term:`Halo Exchange` object can be instantiated as:

Expand Down Expand Up @@ -235,11 +236,12 @@ An alternative pattern supporting different element types is:

.. code-block:: gridtools

using pattern_type = halo_exchange_generic<layout_map<0, 1, 2>, 3, arch_type>;
using pattern_type = halo_exchange_generic<layout_map<0, 1, 2>, arch_type>;

Now the :term:`Layout Map` in the type is the mapping of dimensions to the
computing grid, 3 is the number of dimensions, and arch_type is either
``gcl_gpu``, ``gcl_cpu`` or ``gcl_mc`` (not supported yet).
Now the :term:`Layout Map` in the type is the mapping of dimensions to
the computing grid (the number of dimensions is 3, so the layout map
has three elements), and arch_type is either ``gcl_gpu``, ``gcl_cpu``
or ``gcl_mc`` (not supported yet).

The construction of the object is identical to the previous one, but
the set-up somewhat more complex now, since we have to indicate the
Expand Down
9 changes: 0 additions & 9 deletions include/gridtools/communication/GCL.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,6 @@

#include "low_level/gcl_arch.hpp"

#ifdef GCL_GPU

// workaround that uses host buffering to avoid bad sends for messages larger than 512 kB on Cray systems
//#define GCL_HOSTWORKAROUND

#endif

#ifdef GCL_GPU
#ifdef GCL_MULTI_STREAMS
#ifdef GCL_USE_3
Expand Down Expand Up @@ -61,8 +54,6 @@ extern cudaStream_t XU_stream;

namespace gridtools {

enum packing_version { version_mpi_pack = 0, version_datatype, version_manual };

#ifdef GCL_MPI
extern MPI_Comm GCL_WORLD;
#else
Expand Down
128 changes: 30 additions & 98 deletions include/gridtools/communication/halo_exchange.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,10 @@

#include "../common/boollist.hpp"
#include "low_level/Halo_Exchange_3D.hpp"
#include "low_level/Halo_Exchange_3D_DT.hpp"
#include "low_level/proc_grids_3D.hpp"

#include "high_level/descriptor_generic_manual.hpp"
#include "high_level/descriptors.hpp"
#include "high_level/descriptors_dt.hpp"
#include "high_level/descriptors_dt_whole.hpp"
#include "high_level/descriptors_fwd.hpp"
#include "high_level/descriptors_manual_gpu.hpp"

Expand Down Expand Up @@ -46,36 +43,6 @@ namespace gridtools {
return CartComm;
}

template <int D, typename GT, int version>
struct get_pattern;

template <typename GT>
struct get_pattern<3, GT, 0> {
typedef Halo_Exchange_3D<GT> type;
};

template <typename GT>
struct get_pattern<3, GT, 1> {
typedef Halo_Exchange_3D_DT<GT> type;
};

template <typename GT>
struct get_pattern<3, GT, 2> {
typedef Halo_Exchange_3D<GT> type;
};

template <int D>
struct get_grid;

template <>
struct get_grid<2> {
typedef MPI_3D_process_grid_t<2> type;
};

template <>
struct get_grid<3> {
typedef MPI_3D_process_grid_t<3> type;
};
} // namespace _impl

/**
Expand Down Expand Up @@ -214,7 +181,6 @@ namespace gridtools {
template <typename T_layout_map,
typename layout2proc_map_abs,
typename DataType,
typename GridType,
typename Gcl_Arch = gcl_cpu,
int version = 0>
class halo_exchange_dynamic_ut {
Expand All @@ -229,14 +195,14 @@ namespace gridtools {
/**
Type of the computin grid associated to the pattern
*/
/*typedef typename _impl::get_grid<DIMS>::type grid_type;*/
typedef GridType grid_type;
static const uint_t DIMS = GridType::ndims;
typedef MPI_3D_process_grid_t<3> grid_type;

static constexpr int DIMS = 3;

/**
Type of the Level 3 pattern used. This is available only if the pattern uses a Level 3 pattern.
In the case the implementation is not using L3, the type is not available.
Type of the Level 3 pattern used.
*/
typedef typename _impl::get_pattern<DIMS, grid_type, version>::type pattern_type;
typedef Halo_Exchange_3D<grid_type> pattern_type;

private:
template <typename Array>
Expand All @@ -250,7 +216,7 @@ namespace gridtools {
return CartComm;
}

typedef hndlr_dynamic_ut<DataType, GridType, pattern_type, layout2proc_map, Gcl_Arch, version> hd_t;
typedef hndlr_dynamic_ut<DataType, grid_type, pattern_type, layout2proc_map, Gcl_Arch> hd_t;

hd_t hd;

Expand Down Expand Up @@ -289,15 +255,15 @@ namespace gridtools {

/** constructor that takes the periodicity (mathich the \link
boollist_concept \endlink concept, and the MPI CART
communicator in DIMS (specified as template argument to the
pattern) dimensions of the processing grid. the periodicity is
specified in the order chosen by the programmer for the data,
as in the rest of the application. It is up tp the
construnctor implementation to translate it into the right
order depending on the gridtools::layout_map passed to the class.
communicator in 3 dimensions of the processing grid. the
periodicity is specified in the order chosen by the
programmer for the data, as in the rest of the
application. It is up to the constructor implementation
to translate it into the right order depending on the
gridtools::layout_map passed to the class.
\param[in] c Periodicity specification as in \link boollist_concept \endlink
\param[in] comm MPI CART communicator with dimension DIMS (specified as template argument to the pattern).
\param[in] comm MPI CART communicator with dimension 3
*/
explicit halo_exchange_dynamic_ut(typename grid_type::period_type const &c, MPI_Comm const &comm)
: hd(c.template permute<layout2proc_map_abs>(), comm) {}
Expand Down Expand Up @@ -496,19 +462,6 @@ namespace gridtools {
grid_type const &comm() const { return hd.comm(); }
};

template <int I>
struct pick_version;

template <>
struct pick_version<2> {
static const int value = gridtools::version_mpi_pack;
};

template <>
struct pick_version<3> {
static const int value = gridtools::version_manual;
};

/**
This is the main class for the halo exchange pattern in the case
in which the data pointers, data types, and shapes are not known
Expand All @@ -534,25 +487,25 @@ namespace gridtools {
\tparam GCL_ARCH Specification of the "architecture", that is the place where the data to be exchanged is.
Possible coiches are defined in low_level/gcl_arch.h .
*/
template <typename layout2proc_map, int DIMS, typename Gcl_Arch = gcl_cpu, int version = pick_version<DIMS>::value>
template <typename layout2proc_map, typename Gcl_Arch>
class halo_exchange_generic_base {

public:
// typedef typename reverse_map<t_layout2proc_map>::type layout2proc_map;
static constexpr int DIMS = 3;

/**
Type of the computin grid associated to the pattern
*/
typedef typename _impl::get_grid<DIMS>::type grid_type;
typedef MPI_3D_process_grid_t<3> grid_type;

/**
Type of the Level 3 pattern used. This is available only if the pattern uses a Level 3 pattern.
In the case the implementation is not using L3, the type is not available.
*/
typedef typename _impl::get_pattern<DIMS, grid_type, version>::type pattern_type;
typedef Halo_Exchange_3D<grid_type> pattern_type;

private:
hndlr_generic<DIMS, pattern_type, layout2proc_map, Gcl_Arch, version> hd;
hndlr_generic<pattern_type, layout2proc_map, Gcl_Arch> hd;

public:
/** constructor that takes the periodicity (matching the \link
Expand Down Expand Up @@ -668,37 +621,17 @@ namespace gridtools {
void wait() { hd.wait(); }
};

template <typename layout2proc_map, int DIMS, typename Gcl_Arch = gcl_cpu, int version = version_manual>
class halo_exchange_generic : public halo_exchange_generic_base<layout2proc_map, DIMS, Gcl_Arch, version> {

typedef halo_exchange_generic_base<layout2proc_map, DIMS, Gcl_Arch, version> base_type;
// typedef typename layout_transform<layout_map, layout2proc_map_abs>::type layout2proc_map;

public:
typedef typename base_type::grid_type grid_type;

typedef typename base_type::pattern_type pattern_type;

template <typename DT>
struct traits {
static const int I = DIMS;
typedef empty_field<DT, I> base_field;
};

explicit halo_exchange_generic(typename grid_type::period_type const &c, MPI_Comm comm) : base_type(c, comm) {}

explicit halo_exchange_generic(grid_type const &g) : base_type(g) {}
};
template <typename layout2proc_map, typename Gcl_Arch = gcl_cpu>
class halo_exchange_generic;

// different traits are needed
template <typename layout2proc_map, int DIMS>
class halo_exchange_generic<layout2proc_map, DIMS, gcl_cpu, version_manual>
: public halo_exchange_generic_base<layout2proc_map, DIMS, gcl_cpu, version_manual> {
template <typename layout2proc_map>
class halo_exchange_generic<layout2proc_map, gcl_cpu>
: public halo_exchange_generic_base<layout2proc_map, gcl_cpu> {

static const int version = version_manual;
typedef gcl_cpu Gcl_Arch;

typedef halo_exchange_generic_base<layout2proc_map, DIMS, gcl_cpu, version_manual> base_type;
typedef halo_exchange_generic_base<layout2proc_map, gcl_cpu> base_type;

public:
typedef typename base_type::grid_type grid_type;
Expand All @@ -707,8 +640,8 @@ namespace gridtools {

template <typename DT>
struct traits {
static const int I = DIMS;
typedef empty_field_no_dt<I> base_field;
static const int I = 3;
typedef empty_field_no_dt base_field;
};

explicit halo_exchange_generic(typename grid_type::period_type const &c, MPI_Comm comm) : base_type(c, comm) {}
Expand All @@ -718,13 +651,12 @@ namespace gridtools {

// different traits are needed
template <typename layout2proc_map>
class halo_exchange_generic<layout2proc_map, 3, gcl_gpu, version_manual>
: public halo_exchange_generic_base<layout2proc_map, 3, gcl_gpu, version_manual> {
class halo_exchange_generic<layout2proc_map, gcl_gpu>
: public halo_exchange_generic_base<layout2proc_map, gcl_gpu> {
static const int DIMS = 3;

static const int version = version_manual;
typedef gcl_gpu Gcl_Arch;
typedef halo_exchange_generic_base<layout2proc_map, DIMS, gcl_gpu, version_manual> base_type;
typedef halo_exchange_generic_base<layout2proc_map, gcl_gpu> base_type;

public:
typedef typename base_type::grid_type grid_type;
Expand Down
4 changes: 2 additions & 2 deletions include/gridtools/communication/high_level/descriptor_generic_manual.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
namespace gridtools {

template <typename HaloExch, typename proc_layout_abs>
class hndlr_generic<3, HaloExch, proc_layout_abs, gcl_cpu, version_manual> : public descriptor_base<HaloExch> {
class hndlr_generic<HaloExch, proc_layout_abs, gcl_cpu> : public descriptor_base<HaloExch> {
static const int DIMS = 3;
gridtools::array<char *, _impl::static_pow3<DIMS>::value> send_buffer; // One entry will not be used...
gridtools::array<char *, _impl::static_pow3<DIMS>::value> recv_buffer;
Expand Down Expand Up @@ -381,7 +381,7 @@ namespace gridtools {

#ifdef __CUDACC__
template <typename HaloExch, typename proc_layout_abs>
class hndlr_generic<3, HaloExch, proc_layout_abs, gcl_gpu, version_manual> : public descriptor_base<HaloExch> {
class hndlr_generic<HaloExch, proc_layout_abs, gcl_gpu> : public descriptor_base<HaloExch> {
typedef gcl_gpu arch_type;

static const int DIMS = 3;
Expand Down
Loading

0 comments on commit 412eba6

Please sign in to comment.