Support hipGraph usage in PyTorch #40

xinyazhang · 2024-08-14T17:41:28Z

This simplifies hipGraph support for AOTriton.

This is a draft and shall be validated with PyTorch integration.

The tensor version is now added with suffix _tensor.

SQLite does not have a dedicated boolean type and thus boolean valued performance options will be stored as 0/1 in the datbase. Therefore -DAOTRITON_BUILD_FOR_TUNING=OFF will use slightly different options than -DAOTRITON_BUILD_FOR_TUNING=ON, and it is observed this slight different has impact on the build time and may cause Triton kernel compiling timeout on certain cases.

…et1. This is to match the API used by PyTorch. TODO: make philox_offset1 optional, i.e., passing nullptr means it's zero.

xinyazhang · 2024-08-15T20:53:12Z

Thankfully the PyTorch validation is done before merging the PR. For hipGraph support PyTorch asks much more than it really needs.

…ences

…licts with global cache

1. Profile the kernel unconditionally 2. Use PyTorch 2.5's UT scheme, i.e., only use atol and ignore rtol 3. Record the Lmax error 4. Record the estimated fudge factor that can pass the UT The output json can then be used to justify the new fudge factors for UTs.

… for MI200/300X Trying to get a balance between precision and performance with --fudge_factor_tolerance. See table_tool --help for more detailed document.

xinyazhang · 2024-08-21T02:32:56Z

Testing result of 82e19a8 on MI300X.

Mostly FP32 problems (click to expand).

FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype1-0.0-False-64-8-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype1-0.0-False-64-8-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-256-512-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-256-512-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-256-1024-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-256-1024-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-256-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-256-2048-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-587-1024-256-1-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-587-1024-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-587-1024-256-4-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-587-2048-256-1-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-587-2048-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-1024-143-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-1024-143-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-1024-512-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-1024-512-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-1024-512-192-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-1024-2048-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-1024-2048-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-1024-2048-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-4-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-4-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-4-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-8-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-8-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-8-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-64-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-64-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-64-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-64-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-64-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-64-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-64-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-64-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-143-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-143-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-143-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-143-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-143-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-143-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-143-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-143-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-256-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-256-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-256-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-256-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-256-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-256-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-256-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-256-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-512-192-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-512-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-512-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-512-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-512-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-512-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-512-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-512-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-512-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-512-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-512-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-1024-192-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-1024-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-1024-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-1024-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-1024-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-1024-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-1024-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-1024-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-1024-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-1024-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-1024-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-2048-192-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-2048-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-2048-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-2048-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-2048-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-2048-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-2048-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-2048-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-2048-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-False-2048-2048-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-True-2048-2048-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-True-2048-2048-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-True-2048-2048-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-True-2048-2048-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-True-2048-2048-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-True-2048-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-True-2048-2048-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.0-True-2048-2048-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.5-False-587-4-192-4-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype2-0.5-False-587-8-256-4-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype1-0.0-False-64-4-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype1-0.0-False-64-8-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-256-512-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-256-1024-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-256-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-587-256-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-587-256-256-4-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-587-1024-256-1-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-587-1024-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-587-2048-256-1-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-587-2048-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-64-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-143-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-143-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-143-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-143-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-256-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-256-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-256-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-256-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-512-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-512-192-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-512-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-512-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-512-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-512-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-1024-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-1024-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-2048-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-2048-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-2048-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-1024-2048-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-4-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-4-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-4-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-8-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-8-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-8-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-64-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-64-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-64-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-64-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-64-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-64-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-64-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-64-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-143-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-143-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-143-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-143-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-143-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-143-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-143-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-143-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-143-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-256-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-256-192-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-256-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-256-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-256-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-256-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-256-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-256-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-256-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-256-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-192-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-192-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-512-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-1024-192-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-1024-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-1024-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-1024-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-1024-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-1024-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-1024-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-1024-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-1024-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-1024-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-1024-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-192-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-192-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-192-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-203-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-False-2048-2048-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-True-1024-1024-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-True-2048-2048-192-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-True-2048-2048-203-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-True-2048-2048-203-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-True-2048-2048-203-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-True-2048-2048-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-True-2048-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-True-2048-2048-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.0-True-2048-2048-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype2-0.5-False-587-8-192-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-256-512-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-256-512-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-256-1024-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-256-1024-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-256-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-256-2048-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-587-256-256-1-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-587-256-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-587-1024-256-4-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-587-2048-256-1-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-587-2048-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-1024-512-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-1024-512-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-1024-512-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-1024-1024-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-1024-1024-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-1024-1024-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-1024-2048-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-1024-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-1024-2048-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-4-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-8-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-8-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-8-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-64-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-64-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-64-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-64-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-143-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-143-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-143-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-143-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-256-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-256-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-256-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-256-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-512-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-512-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-512-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-512-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-1024-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-1024-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-1024-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-1024-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-2048-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-2048-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[False-1.2-dtype2-0.0-2048-2048-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-256-512-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-256-1024-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-256-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-587-143-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-587-256-256-1-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-587-256-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-587-256-256-4-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-587-512-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-587-512-256-4-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-587-1024-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-587-2048-256-1-4] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-587-2048-256-4-1] - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-64-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-64-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-143-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-143-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-256-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-256-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-512-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-512-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-512-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-512-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-1024-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-1024-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-1024-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-1024-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-2048-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-1024-2048-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-4-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-4-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-4-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=False
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-8-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-8-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-8-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-8-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-64-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-64-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-64-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-64-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-143-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-143-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-143-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-143-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-256-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-256-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-256-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-256-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-512-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-512-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-512-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-512-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-1024-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-1024-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-1024-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-1024-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-2048-256-1-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-2048-256-1-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-2048-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True
FAILED test/test_backward.py::test_op_bwd_with_matrix_bias[True-1.2-dtype2-0.0-2048-2048-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True

… found at least two devices, cuda:0 and cpu!"

xinyazhang · 2024-08-21T04:00:12Z

69aa5c6 on MI300X

FAILED test/test_backward.py::test_op_bwd[False-1.2-dtype1-0.0-False-64-8-256-4-4] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True tfts={'out': 0.05404883585318915, 'q': 0.22281531579474873, 'k': None, 'v': 0.49811981735596084, 'b': 1.0}
FAILED test/test_backward.py::test_op_bwd[True-1.2-dtype1-0.0-False-64-4-256-4-1] - AssertionError: dk_allclose=False dv_allclose=True dq_allclose=True db_allclose=True tfts={'out': 0.08804808472358727, 'q': 0.16534377859828883, 'k': None, 'v': 0.44964190881448785, 'b': 1.0}

User can turn off fp32 support

…g database.

groenenboomj

Function meets PyTorch reference and tests are valid.

xinyazhang added 6 commits August 14, 2024 17:34

Passing Philox parameters as tensors in forward kernel.

27a3f0d

Passing Philox parameters as tensors in backward kernel.

80fbb18

Add back the scalar version of debug_fill_dropout_rng

c5ee545

The tensor version is now added with suffix _tensor.

Efforts to debug NAN problems

85b015b

Fix debug_fill_dropout_rng and debug_fill_dropout_rng_tensor

806b738

Bug fix on mptune

03ac51e

xinyazhang requested a review from groenenboomj August 14, 2024 17:41

xinyazhang added 3 commits August 14, 2024 17:48

Update README.md

caca3b9

Add scalar philox_offset2, and rename the tensor input as philox_offs…

375b861

…et1. This is to match the API used by PyTorch. TODO: make philox_offset1 optional, i.e., passing nullptr means it's zero.

xinyazhang changed the title ~~Change Philox arguments from scalar to tensor~~ Support hipGraph usage in PyTorch Aug 15, 2024

xinyazhang force-pushed the xinyazhang/tensor_philox branch from 50146cc to 0033481 Compare August 15, 2024 21:22

Add philox_seed_output and philox_offset_output

1d04e6a

xinyazhang force-pushed the xinyazhang/tensor_philox branch from 0033481 to 1d04e6a Compare August 15, 2024 21:23

xinyazhang added 15 commits August 16, 2024 00:24

tune_flash.py: add --max_seqlen_q and --max_seqlen_k to SKIP log sequ…

cad7079

…ences

mptune: fix the counting of failed_kernels and show adiff in status.

ae6e6a2

build: Use build/v2src/triton-cache as Triton's cache dir, avoid conf…

1795c57

…licts with global cache

tiny fix

90f378a

Turn MI200 for futher hipGraph related philox changes.

a0aa853

Fix cannot import name 'KernelOutput' from 'mptune.core.cpp_autotune'

5fb8941

Add missing changes from _common_test.py

beec450

Various fixes to mptune

d255b0e

tune_flash: when resuming, skip result=skipped entries.

dc21ce3

Fix cache flush

e977b3c

Fix classify_monads

cc4aa91

Add property Monad.pid

bfa698a

tune_flash.py: add --debug_headless

01ac29d

Print all monads' PIDs.

336b1b8

xinyazhang added 2 commits August 21, 2024 00:02

Add --fudge_factor_tolerance to table_tool and update tuning database…

92aad9e

… for MI200/300X Trying to get a balance between precision and performance with --fudge_factor_tolerance. See table_tool --help for more detailed document.

Adjust fudge factor for MI200

82e19a8

xinyazhang added 2 commits August 20, 2024 21:36

Fix "RuntimeError: Expected all tensors to be on the same device, but…

ad3ce18

… found at least two devices, cuda:0 and cpu!"

Adjust fudge factors and fix UT bugs on MI300X to pass all tests.

69aa5c6

xinyazhang added 7 commits August 21, 2024 15:04

MI200 requires special treating for UT.

34ff858

Add build option AOTRITON_ENABLE_FP32_INPUTS

a69e05d

User can turn off fp32 support

Add sanity check to lut tensor to detect missing entries in the tunin…

4ac7584

…g database.

test/attn_torch_function: also check return value of forward kernel.

727ea10

Add MI200 missing entries

fa12fe9

Report all tuning LUT tensor problems

d215061

Code clean up.

dd30080

xinyazhang marked this pull request as ready for review August 22, 2024 04:28

groenenboomj approved these changes Aug 22, 2024

View reviewed changes

xinyazhang merged commit 9be0406 into main Aug 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support hipGraph usage in PyTorch #40

Support hipGraph usage in PyTorch #40

xinyazhang commented Aug 14, 2024

xinyazhang commented Aug 15, 2024

xinyazhang commented Aug 21, 2024 •

edited

Loading

xinyazhang commented Aug 21, 2024

groenenboomj left a comment

Support hipGraph usage in PyTorch #40

Support hipGraph usage in PyTorch #40

Conversation

xinyazhang commented Aug 14, 2024

xinyazhang commented Aug 15, 2024

xinyazhang commented Aug 21, 2024 • edited Loading

xinyazhang commented Aug 21, 2024

groenenboomj left a comment

Choose a reason for hiding this comment

xinyazhang commented Aug 21, 2024 •

edited

Loading