In-graph synchronization with Epochs

[fknorr]

So far, queue::slow_full_sync() and the shutdown of worker executors are implemented by sending SYNC and SHUTDOWN commands through broadcast_control_command() and then busy-waiting on the main thread. Since we already introduced synchronization logic to task and command graphs before as part of horizons, we now have the opportunity to integrate SYNC and SHUTDOWN into the graphs as well and reduce specialized code paths. This will also be a stepping stone towards using explicit synchronization to exchange data in celerity buffers with the main thread.

This PR represents synchronization points in the graph with epochs, which themselves are a generalization of INIT/NOP tasks and commands. Each graph node (except the first epoch) has exactly one preceding epoch, and no node can ever depend on a node before its epoch. In that sense they are similar to horizons, except that they fully serialize execution and dependency tracking:

To ensure correct temporal ordering, a new epoch node receives a true-dependency on the entire execution front (orange edges); and all nodes without other true-dependencies (pure producers, tasks a b d e) receive a dependency on their epoch (pink edges). A new epoch will immediately become the last-writer and host-buffer initializer for all successor tasks. Like any other node, epochs can be subsumed by horizons, after which the subsuming horizon becomes the epoch for all following nodes.

Inserting a new epoch task with task_manager::finish_epoch() will create one epoch command per node, with dependencies on the current execution front. Each epoch_command will issue one epoch_job, which awaits the completion of said dependencies, optionally issues an MPI_Barrier and then notifies the task manager via notify_epoch_completed(). Other threads can await this notification to block the main thread in queue::slow_full_sync() and ~runtime().

The nodes preceding a task epoch are pruned once the first task is created after notify_epoch_completed(). In the command graph generator, nodes can be deleted as soon as they lie behind the last epoch, as they can't be depended on by any successor task. In that regard, epochs act like immediately-applied horizons.

This PR also recognizes that the task manager can prune all nodes preceding a horizon A as soon as it has been notified that horizon B with A → B has been reached. This notification signifies that all commands of tasks preceding A have finished executing, so even though they might still be referenced from already-executed commands in the command graph, the task data never has to be accessed again.

To aid debugging, Celerity will now also track the origin of a dependency in addition to its kind. This is currently visualized in generated dot graphs.

The old synchronization infrastructure (broadcast_control_command(), sync_id, ...) has been removed.

[fknorr]

From live discussion with @PeterTh: We need to re-visit an earlier concern about race conditions on tasks returned form task_manager::get_task(): The assumption was that this method can return a const task* that can be accessed without a lock since tasks don't change after creation. However, their dependencies are modified from ~intrusive_graph_node(), so that is not actually true. This is relevant for the PR in question since reducing the pruning delay from 2 horizons to 1 might cause conflicting accesses on the current epoch task.

[psalz]

I'm only about a third through the changes so far; I've added some minor notes. However I did a quick test and it seems like there is a ~10% performance drop when running wave_sim -N 4096 -T 200 with 4 workers on gpuc1: Current master takes 522ms on average (over 5 runs), where this PR takes 585ms on average.

[fknorr]

I have collected longer-running benchmark results on the wave_sim behavior, since previous graph-generation timings turned out to be spurious. It appears that the runtime is usually pretty stable but produces rare and pronounced outliers. This happens on both master and this PR, so I believe we cannot attribute their existence to the changes in this PR.

The following numbers were collected by running wave_sim on 4 nodes, repeated 1000 times, while measuring the run time surrounding all kernel submissions after slow_full_sync():

The outliers skew the mean, but by eliminating them (or using the median), we see that the "normal" run-time behavior has not changed noticeably.

[psalz]

Thanks again, I've added a few more notes.

The only thing that I still find somewhat confusing is that epochs are both an abstract concept and a concrete type of task/command; in that horizons can also act as epochs, and there are some fields that are named *epoch*, but can also point to a horizon task/command. Can you outline to me again briefly why you decided against treating horizons as a type of epoch?

[BlackMark29A]

Looks good to me. The only thing I found was a typo that wasn't even introduced by this PR.

[fknorr]

Thanks to all of you for the continued endurance on this!