[FEA] External libcudf APIs should expose CUDA streams

[jrhemstad]

Is your feature request related to a problem? Please describe.
In order to maximize efficiency and utilization of the GPU, you need to be able to execute kernels and memory copies on independent streams. However, no libcudf API currently exposes streams to the end user.

Describe the solution you'd like
Any API that executes a GPU kernel or allocates/copies memory should provide the end user with the option to specify a stream.

Open questions:

• How should the stream be exposed? E.g., accept a cudaStream_t*? Or bundled inside of an options struct?
• Who is responsible for creating/destroying the stream?
• In a C++ API its easy to provide a default argument for the stream, but in the C API the user will always be required to specify a stream. Is this what we want?
• Should streams be exposed in all of the Python APIs?

I suspect we'll all agree this does need to be done, the question is rather one of how and when.

[felipeblazing]

I woudl argue that we should take it one step further and say that every exposed api should provide the capacity at some point to allow stream to be defined with the most priority ones you mention which do allocation which can block execution , super gross. What other options would you want to pass it? An allocator? (besides stream)

Creating / Destroying
A pattern libraries like thrust follow is that the user creates the stream and if the user doesn't supply one it uses thee default stream. I like this pattern in general.

C vs C++ api
Who wants a c only api? The user can always send stream = 0 which is the default stream.

Exposing in python
I woudl imagine so right? Why wouldn't we?

If we are compiling with --default-stream per-thread then this is slightly less of a concern in the more immediate future.

[jrhemstad]

I woudl argue that we should take it one step further and say that every exposed api should provide the capacity at some point to allow stream to be defined with the most priority ones you mention which do allocation which can block execution , super gross.

Sure, what I really meant is that every API that it makes sense to have execute on a stream should accept a stream argument. Some functions obviously never need a stream, e.g., https://github.com/rapidsai/cudf/blob/branch-0.6/cpp/include/cudf/functions.h#L72

What other options would you want to pass it? An allocator? (besides stream)

Maybe a device ID? Device properties? I was thinking of the ModernGPU context_t when I suggested the options struct. https://github.com/moderngpu/moderngpu/blob/master/src/moderngpu/context.hxx#L47

[jrhemstad]

I like the approach that was taken in cuML here using a cuml::handle object that is passed into every API.

rapidsai/cuml#247

You can bundle lots of library specific resources in there such as:

• stream
• device ID
• memory allocator (this would allow us to better abstract the memory allocator instead of using RMM_ALLOC directly).

[mrocklin]

Drawing inspiration from various other Python libraries it seems that some include a stream= keyword in most operations, such as in these numba examples

While othres use context managers, which are nice at managing global state sensibly. Here is an example from CuPy

n = 10
zs = []
map_streams = []
stop_events = []
reduce_stream = cupy.cuda.stream.Stream()
for i in range(n):
    map_streams.append(cupy.cuda.stream.Stream())

start_time = time.time()

# Map
for stream in map_streams:
    with stream:
        x = rand.normal(size=(1, 1024 * 256))
        y = rand.normal(size=(1024 * 256, 1))
        z = cupy.matmul(x, y)
        zs.append(z)
    stop_event = stream.record()
    stop_events.append(stop_event)

Personally I like the cupy solution, just because it doesn't require touching all of the API

(I acknowledge that this response may be outside of the original scope of libcudf. My apologies if so).

[EvenOldridge]

Hey @jrhemstad checking in about the status of this. This is an issue for the PyTorch and TF dataloaders that we're building. I know there are potentially other workarounds being explored.

[jrhemstad]

Hey @jrhemstad checking in about the status of this. This is an issue for the PyTorch and TF dataloaders that we're building. I know there are potentially other workarounds being explored.

It's still an active point of conversation, but we don't have any current plans to add streams to public APIs in the near future.

[GregoryKimball]

Closing in favor of #13744