Improvements to the communications flow

[ahoenselaar]

There are areas where the current communications code could be improved:

• The sequence of send and receive operations is inferred multiple times per timestep, even though it only changes in connect_the_chunks.
• Asynchronous receives are initiated too late (after all data has been sent)
• Send operations are sequenced without consideration for a particular prioritization criterion, e.g. data transfer size
• The use of MPI_WaitAll delays the processing of incoming data excessively
• The data structures used for communication buffers are complex (3D arrays), error-prone to use, and are sized under the assumption of O(n^2) communication links. In practice, each chunk exchanges data with a small number of other chunks.
• Sequential boundary updates of {PH, H}, {PE, E} are treated as separate operations with full synchronization barriers.
• The flattening of data into comms buffers and its reverse operation are implemented using lists of pointers (one pointer per transferred float). As the regions that need to be transferred can be expected to be cuboids, the flattening could be implemented using strided loops.

The first three issues are addressed in #1656.

[ahoenselaar]

The logic in connect_the_chunks is so convoluted and lacking coverage by unit tests, that it is very unlikely for the proposed refactoring to succeed without causing a subtle form of breakage.