Subphase API Proposal

Now that we have the PR #1255 for subphase labels, here's the proposal for the user-facing API for subphases.

There are two elements to the design. First, is the interaction with vt::CollectionChainSet; second, the pure epoch variant:

Potential Problems

• How do we deal with nesting? It's definitely possible to end up with nested subphases because epochs can be easily nested. Do we need to keep a stack of live subphases? That starts to sound complicated.
• Potential conflicts in names between rooted and collective subphase labels. Right now, they are handled individually so they will be distinct IDs in the output file.
• How do we represent work that doesn't end up in a subphase when other work is in a subphase. Should be have a default "null" subphase that means it wasn't assigned to one?

CollectionChainSet

nextStep

I propose we add another call for nextStep that also creates a subphase:

  void CollectionChainSet::nextStep(
    std::string const& label, std::function<PendingSend(Index)> step_action
  );

This new call will create a subphase with the label that already exists and will group actions in step_action from that rooted step into a rooted subphase:

  void CollectionChainSet::nextStepSubphase(
    std::string const& label, std::function<PendingSend(Index)> step_action
  );

nextStepCollective

I propose we add a similar call for nextStepCollective that creates a subphase:

  void CollectionChainSet::nextStepCollective(
    std::string const& label, std::function<PendingSend(Index)> step_action
  );

This new call with also create a subphase with the label similar to the previous call:

  void CollectionChainSet::nextStepCollectiveSubphase(
    std::string const& label, std::function<PendingSend(Index)> step_action
  );

Higher-level calls

The calls will be stateful calls on the chain set that will start/stop a new subphase across the whole collection. They will end up creating an epoch under the hood to group all the work enqueued by them and associating it with the subphase. This brings up potential nesting issues wrt to subphases...

  void CollectionChainSet::startCollectiveSubphase(std::string const& label);
  void CollectionChainSet::endCollectiveSubphase(std::string const& label);

Regular epochs and subphases

I propose we add two calls:

  vt::runInEpochCollectiveSubphase(std::string const& label, Callable&& action);
  vt::runInEpochRootedSubphase(std::string const& label, Callable&& action);