Machinery for packing sparse variables

[lroberts36]

PR Summary

This pull request is for a new class SparsePack that packs sparse data across blocks in a way that eliminates the need to check the allocation status of each variable. This duplicates some of the functionality of the other types of packs already in Parthenon and in one of @jdolence branches, but has the following nice features:

• Allows for regexes to generically select groupings of variables.
• Only includes allocated variables in the pack, so that allocation status does not need to be checked in kernels.
• Stores the range of allocated variables of each group on each block on device, so that the range of indices for a particular variable group can easily be accessed in an outer loop.
• Provides both an unordered_map<string, SparseVarIdx> and a type based method for accessing variables within a kernel (see the test_sparse_pack.cpp unit test for examples).
• Templated so that SparsePacks can be created from MeshData or MeshBlockData.
• Implements a SparsePackCache type to internally handle cacheing.
• Aside from including a SparsePackCache member variable in MeshData and MeshBlockData (and an associated GetSparsePackCache() method), methods relating to SparsePack are not included in MeshData and MeshBlockData.
• For better or worse, the View of Views for the pack is currently implemented as a ParArray3D<ParArray3D<Real>> with the first index distinguishing between the variable and its fluxes, the second the block index, and the third the variable index on the block. Since the number of variables differs between blocks when things are sparsely allocated, the size of the last dimension is the maximum number of variables allocated on any block. This means there is sometimes space in the array that is unused, but I doubt this will be a real major issue unless it creates too large of a deep_copy.

Currently, I have changed FluxDivergence(MeshData<Real>*, MeshData<Real>*) to use SparsePacks just to test if this breaks anything. Not surprisingly, the exact choice of loop pattern substantially impacts performance. The chosen loop pattern seems to do as well as the old one on the regression_test:advection_performance test, but maybe there is a better way. Aside from this, this PR should not impact any aspect of the code and the change to FluxDivergence could be reverted.

I also removed the <regex> header from the list of unapproved headers in cpplint.py. As far as I am aware, there isn't a really good reason for having it there (see google/styleguide#194). That being said, a cursory glance around the internet suggested that boost::regex might be superior performance wise.

PR Checklist

• Code passes cpplint
• Adds tests for new features.
• Code is formatted
• Changes are summarized in CHANGELOG.md
• CI has been triggered on Darwin for performance regression tests.
• (@lanl.gov employees) Update copyright on changed files

[lroberts36]

@forrestglines: I think this is ready for your re-review at this point. If you are ok with it, go ahead and merge.

Also added an issue #697 to add documentation for sparse packs

[forrestglines]

Is this ready to merge? I think it is. Do we need to do something to get that "ci/gitlab/gitlab.com" check to go through?

[lroberts36]

Yeah, I think it should automatically merge once the gitlab CI runs, but that CI is broken right now for everything. There is a way to force a merge I think, but I am not sure how that interacts github.

[forrestglines]

As I've been testing out using this pack for face and edge centered fields, I've realized that the index order for SparsePack:operator() is a bit unintuitive for me. Currently we have

v( dir, meshblock, var, k, j, i)

In my head, the following organization seems more intuitive

v( var, dir, meshblock, k, j, i)

So that the first two indicies are to index the variable and the last four are to index the location in space.

[lroberts36]

I am not sure what the best way to order this is. I chose the current ordering for regular variables to be the same as MeshBlockPacks. I don't think there is a way to directly access fluxes in MeshBlockPacks, so there it appears there are no flux calls that include direction, block, variable, and kji currently (@Yurlungur correct me if I am wrong). I think your ordering makes logical sense, but I would be concerned that it requires a lot of index reordering if someone where to update their code from the old packing machinery.

[forrestglines]

I don't think there is a way to directly access fluxes in MeshBlockPacks, so there it appears there are no flux calls that include direction, block, variable, and kji currently

I agree, that sounds correct. However, that will change with face and edge centered fields. I considered adding another comp for "component" to complement flux but I found it a bit redundant

but I would be concerned that it requires a lot of index reordering if someone where to update their code from the old packing machinery.

I suspect that too. I think the v( meshblock, var, k, j, i) is likewise awkward although it reflects the order of things in memory. However, it's probably too much bug-prone work to change now. If we could automate that refactor in downstream codes with a script, I would advocate for that change later on.