Skip to content

Commit

Permalink
Refactor matmul test suite. (#22)
Browse files Browse the repository at this point in the history
Progress on #2. See
also the long [Discord thread
here](https://discord.com/channels/689900678990135345/1270451599231156266).

## Summaries of changes

### Further decoupled test suites from the core CMake project

* Forked `iree_native_test.cmake` to
`iree_test_suites_native_test.cmake`
* Dropped support (temporarily?) for testing on Android, RISC-V, and ARM
with SME
* Forked `iree_e2e_generated_runner_test.cmake` to
`iree_test_suites_runner_test.cmake`
* Dropped support (temporarily?) for filtering within the build system
which tests are defined and compile .vmfb files
* Now we can set `-DIREE_BUILD_TESTS=OFF` and avoid pulling in IREE's
other tests
* Added a new hand-authored `linalg_ops/matmul/CMakeLists.txt` that runs
tests on each backend using default flags

### Simplified the test generator

* Dropped unused functions
* Folded GPU-specific shapes into generic "small" and "large" shape test
suites

### Ran the `generate_e2e_matmul_tests.py` script offline and checked in
the generated files

* Currently 56 files totaling 1.90MB on disk (~27000 lines of code
according to GitHub)
* Now we can inspect the test cases without needing to run the generator
locally, and I fixed a few formatting issues
* I think this makes test suite management easier, and having the
generated files in this test suites repository doesn't cost the main
repository much (just extra `git checkout` time), but I could see a case
for more tightly coupling the generator with the test runner

## What is left to do?

* I want to iterate some more on the `linalg_ops/matmul/CMakeLists.txt`
file or move to a different test runner somehow. I mainly want to
support XFAIL in some way for both compiling and running.
* We should add back tests using CPU features like AVX512, GPU features
like Vulkan float16 extensions, and other non-default flags somehow.
Either infer what the compiler can from the host / target, or add test
suites explicitly.
  • Loading branch information
ScottTodd authored Oct 15, 2024
1 parent d9eeb57 commit 3a6820f
Show file tree
Hide file tree
Showing 63 changed files with 28,985 additions and 2,607 deletions.
8 changes: 3 additions & 5 deletions linalg_ops/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,7 @@ set(IREE_PACKAGE_ROOT_DIR "${CMAKE_CURRENT_LIST_DIR}")
set(IREE_PACKAGE_ROOT_PREFIX "iree-test-suites")
set(IREE_BUILD_COMPILER OFF)
set(IREE_BUILD_SAMPLES OFF)
# We should also be able to set -DIREE_BUILD_TESTS=OFF, but this currently
# depends on the core project's CMake functions like iree_native_test and
# iree_bytecode_module.
set(IREE_BUILD_TESTS ON)
set(IREE_BUILD_TESTS OFF)

if(IREE_USE_LOCAL_REPO)
message(STATUS "Using IREE repo at path '${IREE_LOCAL_REPO_PATH}'")
Expand Down Expand Up @@ -122,6 +119,7 @@ iree_cc_binary(
#-------------------------------------------------------------------------------

list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR})
include(iree_e2e_generated_runner_test)
include(iree_test_suites_native_test)
include(iree_test_suites_runner_test)

add_subdirectory(matmul)
Loading

0 comments on commit 3a6820f

Please sign in to comment.