Jmm/one buffer

[Yurlungur]

PR Summary

Built on the discussions in the hackathon and on @pgrete 's unification of communication buffers in #493 . This combines the flux and coarse cell buffers into a single kokkos view. A couple of things to note:

• This reduces the number of allocations per variable per meshblock by 3.
• This relies on LayoutRight in our kokkos views.
• This is wasteful of some memory because the coarse buffer is now the same size as the dense buffer.
• This is built on Use subviews for buffers #493 and should go in after it.

I haven't done any profiling yet.

PR Checklist

• Code passes cpplint
• New features are documented.
• Adds a test for any bugs fixed. Adds tests for new features.
• Code is formatted
• Changes are summarized in CHANGELOG.md
• (@lanl.gov employees) Update copyright on changed files

[forrestglines]

I'm curious to see the fix to the unit test, but once that's in we should merge

[forrestglines]

Getting around the wasted memory would probably mean allocating from fluxes and this coarse data from one long array, right? However, I don't think Kokkos supports changing layouts like that with a subview.

[Yurlungur]

Yeah, I tried that actually. Kokkos yelled at me because types.

[forrestglines]

Is comm_data_ only the collection of 4 arrays - the 3 fluxes and the coarse array? Does the benefit of combining these allocations outweight the cost of wasted memory for coarse_s? Could coarse_s remain it's own array?

[forrestglines]

This is less relevant for this PR, but is allocating space for all the fluxes necessary? Could the flux divergence be computed in each dimension one by one so that only one flux array is needed in memory? If possible, this would allow larger problem sizes but might be too big a code change.

[Yurlungur]

Is comm_data_ only the collection of 4 arrays - the 3 fluxes and the coarse array? Does the benefit of combining these allocations outweight the cost of wasted memory for coarse_s? Could coarse_s remain it's own array?

That could be done. I'm not sure what the right trade-off is. I would say we accept the memory overhead for now, and we can split out the coarse buffer later if we desire.

This is less relevant for this PR, but is allocating space for all the fluxes necessary? Could the flux divergence be computed in each dimension one by one so that only one flux array is needed in memory? If possible, this would allow larger problem sizes but might be too big a code change.

This is an interesting idea. I agree not for this PR, but something to think about.