Declarative side effects

[fknorr]

Currently, state that is not managed by Celerity must be modified through reference-captures in host tasks. This is opaque to the runtime and does not allow us to detect dependencies on global state between tasks and commands. SYCL proposes handler::depends_on for this purpose, mostly motivated by USM. In Celerity however, we want to avoid control-flow dependencies if possible and focus on data flow.

New Public APIs

This PR introduces new declarative APIs for host-task side-effects.

• Creating a celerity::experimental::host_object<T> wraps and manages an arbitrary value which should be tracked by the runtime. This has roughly the semantics of a buffer. Specializations exists for host_object<T&> which tracks a user-owned value and host_object<void> which tracks access to global state.
• Capturing a celerity::experimental::side_effect in a host-task kernel declares that the task will affect the host object, and that a dependency exists to preceding and following tasks affecting the same host object. A side_effect has roughly the semantics of an accessor.

This eliminates some use cases for reference-captures and properly solves the ordering problem in the wave_sim example that is being worked around in #66.

So far, side effects always generate true-dependencies to the last task that affected the same host object. We are aware that this might be overly restrictive for uses that do not care about access ordering or are fully thread-safe. A backwards-compatible API for specifying constraints (similar to celerity::access_mode or std::memory_order) will be investigated in a future RFC.

Internal additions

• The runtime now holds a host_object_manager that manages the host_object_ids and knows when the last host-object goes out of scope to (potentially) initiate runtime shutdown.
• celerity::handler tracks a side_effect_map similar to the buffer_access_map and passes it to the task on creation.
• task_manager and graph_generator track the last task / command affecting each host object to properly generate dependencies.
• Tests are added covering dependency generation, proper ordering, and correct behavior around horizons.

For an example usage, see the wave_sim example.

Resolved Questions

• Do we need to handle host-object dependencies explicitly on horizon generation? (=> Yes, now explicitly handled in the spirit of Correctly apply horizons to collective-group dependencies #69)
• Do we need init-task / init-command dependencies? (=> This might become a concern if we generate anti-dependencies after introducing more than one side_effect_order).
• Should the buffer-equivalent hold the object by-value or by-reference? By-value is lifetime-safe and can restrict access, but by-reference might be convenient when porting to Celerity (=> we want both, as well as a void specialization for side effects on globals)
• Can we consider the final design (after the RFC period) future-proof, or should we stage it through the experimental:: namespace? (=> experimental namespace for now)
• Would a shared vs exclusive dichotomy be better than the current classification into read and write side effects (we were re-using access_mode in the first design iteration)? We use these to control concurrency, but if a mutating kernel does so atomically, we do not need to serialize it. Classifying it as a read would be counter-intuitive. Also, will exclusive vs. shared also change dependency generation in any way? (=> Will be solved in a future RFC)
• The name host_object is not very descriptive, although it highlights the connection to host_task. The name needs to clarify that, unlike a buffer, this type is neither replicated nor distributed. (=> Good for now, though we might change the name before stabilization).
• Currently, we've implemented a default-sequential order on side effects. Is that a good idea? (=> Yes, similar to memory_order::seq_cst being the default for atomics, this should only cause inefficiency, never incorrectness).

[fknorr]

From today's Celerity meeting:

• The operator() access protection on side_effect breaks with what the API pattern of accessors, so we will remove it
• We need to think about access modes further. exclusive vs shared might be better than read vs write to communicate that we want to constrain concurrency rather than declare data movement. Some operations are not easily classified into read/write patterns, and manual synchronization and atomic operations may allow concurrency between writing tasks.
• The final name of host_object should express better that this type, unlike buffers, is neither distributed nor replicated
• We want host_object<T&> to naturally provide side effects on user-managed variables and host_object<void> to represent side effects on globals
• For now, the new features should reside in the experimental:: namespace

[psalz]

One more thing from the discussion: If we do end up using the existing deduction tags, we should probably opt for the *_host_task variants for consistency.

[fknorr]

From yesterday's Celerity meeting:

• This PR will proceed without a public API for different side effect access modes / concurrency and reordering constraints. We want to have this at some point, however it's not clear yet what the precise model should be. Side effects will however always default to the strictest mode, i.e. full serialization in submission order relative to other tasks affecting the same host object. This makes the API forward-compatible when we add "side effect order" deduction tags (think of std::atomic and std::memory_order).
• Although we feel host_object is not optimally named, we will go forward with it for the lack of a better alternative. host_object might in the future have an equivalence in host-local, non-distributed buffers (i.e. thinly wrapped SYCL buffers for passing data between kernels locally per-node). If that ever becomes reality, both types should share a naming scheme.
• host_object<void> will be necessary to serialize global side effects (like calls to printf), however host_object<T&> is more controversial. It might be irreplaceable while porting existing applications to Celerity, but it will be another footgun like celerity::allow_by_ref. We will keep it for now, but it might not ever be stabilized.

[fknorr]

This has now left the RFC phase and is ready for review.

[PeterTh]

Looks pretty good to me. The introduction of task_geometry also cleans that part up a bit.

[facuMH]

LGTM

[BlackMark29A]

Apart from some minor bikeshedding, this looks good to me.

[psalz]

LGTM!