Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SYCL Graphs and empty barriers #13066

Closed
al42and opened this issue Mar 19, 2024 · 3 comments
Closed

SYCL Graphs and empty barriers #13066

al42and opened this issue Mar 19, 2024 · 3 comments
Assignees
Labels
bug Something isn't working sycl-graph

Comments

@al42and
Copy link
Contributor

al42and commented Mar 19, 2024

Describe the bug

Trying to record an empty barrier while recording a graph throws an exception. The same pattern works fine when not recording a graph.

By itself, this operation is meaningless. However, it might appear in more complex code, like the following:

initializeData(q1, options); // May or may not submit anything, depending on `options`
sycl::event ev = q1.ext_oneapi_submit_barrier();
computeData(q2, ev); // Compute stuff (some operations depend on `ev`)

I don't see anything in either https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/supported/sycl_ext_oneapi_enqueue_barrier.asciidoc or https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/experimental/sycl_ext_oneapi_graph.asciidoc prohibiting such use, so I believe it should be allowed.

To reproduce

#include <sycl/sycl.hpp>
#include <iostream>

namespace syclex = sycl::ext::oneapi::experimental;

int main() {
  for (const auto &dev : sycl::device::get_devices()) {
    if (dev.has(sycl::aspect::ext_oneapi_graph)) {
      std::cout << "Running on " << dev.get_info<sycl::info::device::name>()
                << std::endl;
      sycl::context ctx{dev};
      sycl::queue q1{ctx, dev, {sycl::property::queue::in_order()}};
      const std::vector<sycl::queue> queuesToRecord{q1};

      const sycl::property_list propList{
          syclex::property::graph::no_cycle_check()};
      syclex::command_graph<syclex::graph_state::modifiable> graph(ctx, dev,
                                                                   propList);

      for (bool useGraph : {false, true}) {
        std::cout << "Running " << (useGraph ? "with" : "without")
                  << " graphs..." << std::endl;
        if (useGraph) {
          bool result = graph.begin_recording(queuesToRecord);
          if (!result) {
            std::cout << "  Could not start the recording" << std::endl;
          }
        }
        sycl::event ev = q1.ext_oneapi_submit_barrier();
        if (useGraph) {
          graph.end_recording();
          auto instance = graph.finalize();
          q1.ext_oneapi_graph(instance).wait_and_throw();
        }
        std::cout << "  Ok!" << std::endl;
      }
    }
  }
}
$ clang++ -fsycl -O1 -g test_graph_event.cpp && ONEAPI_DEVICE_SELECTOR=level_zero:0 ./a.out
Running on Intel(R) Arc(TM) A770 Graphics
Running without graphs...
  Ok!
Running with graphs...
terminate called after throwing an instance of 'sycl::_V1::exception'
  what():  No event has been recorded for the specified graph node
Aborted (core dumped)

Environment

  • Ubuntu 22.04 HWE
  • Arc A770 (via L0), RTX 3060, RX6400
$ clang++ --version
clang version 19.0.0git (https://github.com/intel/llvm 76167854fb3edba3a575302ccde14392c671529b)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/aland/intel-sycl/llvm/build/install/bin

$ sycl-ls 
[opencl:gpu][opencl:0] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) A770 Graphics OpenCL 3.0 NEO  [23.35.27191.9]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) UHD Graphics 770 OpenCL 3.0 NEO  [23.35.27191.9]
[opencl:cpu][opencl:2] Intel(R) OpenCL, 12th Gen Intel(R) Core(TM) i9-12900K OpenCL 3.0 (Build 0) [2023.16.12.0.12_195853.xmain-hotfix]
[level_zero:gpu][level_zero:0] Intel(R) Level-Zero, Intel(R) Arc(TM) A770 Graphics 1.3 [1.3.27191]
[level_zero:gpu][level_zero:1] Intel(R) Level-Zero, Intel(R) UHD Graphics 770 1.3 [1.3.27191]
[cuda:gpu][cuda:0] NVIDIA CUDA BACKEND, NVIDIA GeForce RTX 3060 8.6 [CUDA 12.4]
[hip:gpu][hip:0] AMD HIP BACKEND, AMD Radeon RX 6400 gfx1034 [HIP 60032.83]

Additional context

No response

@al42and al42and added the bug Something isn't working label Mar 19, 2024
@al42and
Copy link
Contributor Author

al42and commented Mar 19, 2024

The problem appears even in less pathological cases:

sycl::queue q1{ctx, dev, {sycl::property::queue::in_order()}};
sycl::queue q2{ctx, dev, {sycl::property::queue::in_order()}};
const std::vector<sycl::queue> queuesToRecord{q1, q2};
// TODO: Allocate buf_h and buf_d...
// TODO: Start recording the graph  
sycl::event ev = q1.copy(buf_h, buf_d, n);
// Now, we want all future operations in q2 to wait for the copy to complete...
q2.ext_oneapi_submit_barrier({ev}); // Ooops, throws!

@reble reble assigned reble and EwanC Mar 20, 2024
EwanC added a commit to reble/llvm that referenced this issue Mar 28, 2024
Fix for intel#13066

The special case for using barriers on an in-order queue
is that the last event/node submitted to the queue is used
as an event for the barrier to depend on.

Looking at the last command submitted to the queue isn't
correct for a graph, because previous commands
submitted to a graph could have been added explicitly or
from recording another queue. Therefore, there is not
guaranteed that the last command submitted by the in-order
queue is correct dependency for the barrier node in the graph.
@EwanC
Copy link
Contributor

EwanC commented Mar 29, 2024

Thanks for the bug report, proposed fix - #13193

martygrant pushed a commit that referenced this issue Apr 1, 2024
Fix for #13066

The special case for using barriers on an in-order queue is that the
last event/node submitted to the queue is used as an event for the
barrier to depend on.

Looking at the last command submitted to the queue isn't correct for a
graph, because previous commands
submitted to a graph could have been added explicitly or from recording
another queue. Therefore, there is not guaranteed that the last command
submitted by the in-order queue is correct dependency for the barrier
node in the graph.

---------

Co-authored-by: Ori Sky <4142775+ori-sky@users.noreply.github.com>
@al42and
Copy link
Contributor Author

al42and commented Apr 1, 2024

Thanks for the fix, @EwanC!

@al42and al42and closed this as completed Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working sycl-graph
Projects
None yet
Development

No branches or pull requests

4 participants