Rework how ASan and TSan workflows use Docker. (#18396)

Progress on #15332. I'm trying to get rid of the `docker_run.sh` scripts, replacing them with GitHub's `container:` feature. While local development flows _may_ want to use Docker like the CI workflows do, those scripts contained a lot of special handling and file mounting to be compatible with Bazel. Much of that is not needed for CMake and can be folded away, though the `--privileged` option needed here is one exception. This stops using the remote cache that is hosted on GCP. We can try adding back a cache using GitHub or our own hosted storage as part of #18238. Job | Cache? | Runner cluster | Time | Logs -- | -- | -- | -- | -- ASan | Cache | GCP runners | 14 minutes | [logs](https://github.com/iree-org/iree/actions/runs/10620030527/job/29438925064) ASan | No cache | GCP runners | 28 minutes | [logs](https://github.com/iree-org/iree/actions/runs/10605848397/job/29395467181) ASan | Cache | Azure runners | (not configured yet) ASan | No cache | Azure runners | 35 minutes | [logs](https://github.com/iree-org/iree/actions/runs/10621238709/job/29442788013?pr=18396) | | | TSan | Cache | GCP runners | 12 minutes | [logs](https://github.com/iree-org/iree/actions/runs/10612418711/job/29414025939) TSan | No cache | GCP runners | 21 minutes | [logs](https://github.com/iree-org/iree/actions/runs/10605848414/job/29395467002) TSan | Cache | Azure runners | (not configured yet) TSan | No cache | Azure runners | 32 minutes | [logs](https://github.com/iree-org/iree/actions/runs/10621238738/job/29442788341?pr=18396) ci-exactly: linux_x64_clang_asan
iree-org · Sep 12, 2024 · 8184258 · 8184258
1 parent ac50867
commit 8184258
Show file tree

Hide file tree

Showing 4 changed files with 68 additions and 26 deletions.
diff --git a/.github/workflows/ci_linux_x64_clang_asan.yml b/.github/workflows/ci_linux_x64_clang_asan.yml
@@ -29,21 +29,22 @@ jobs:
     needs: setup
     if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'linux_x64_clang_asan')
     runs-on: azure-linux
+    # TODO(scotttodd): use ghcr.io/iree-org/cpubuilder_ubuntu_jammy_x86_64 here
+    container: gcr.io/iree-oss/base-bleeding-edge@sha256:cf2e78194e64fd0166f4141317366261d7a62432b72e9a324cb8c2ff4e1a515a
+    defaults:
+      run:
+        shell: bash
     steps:
       - name: "Checking out repository"
         uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
         with:
           submodules: true
-      - name: "Building and testing with ASan"
+      - name: Install Python requirements
+        run: python3 -m pip install -r ./runtime/bindings/python/iree/runtime/build_requirements.txt
+      # TODO(#18238): add local or remote ccache
+      - name: Build and test with ASan
         env:
-          IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }}
-        run: |
-          # Note that this uses the latest version of the clang compiler, etc.
-          # This gives us access to the latest features and validates that IREE
-          # builds using the latest versions.
-          ./build_tools/github_actions/docker_run.sh \
-            --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \
-            --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \
-            --env "CCACHE_NAMESPACE=base-bleeding-edge@sha256:14200dacca3a0f3a66f8aa87c6f64729b83a2eeb403b689c24204074ad157418" \
-            gcr.io/iree-oss/base-bleeding-edge@sha256:cf2e78194e64fd0166f4141317366261d7a62432b72e9a324cb8c2ff4e1a515a \
-            ./build_tools/cmake/build_and_test_asan.sh
+          # Use a modern clang explicitly.
+          CC: clang-19
+          CXX: clang++-19
+        run: ./build_tools/cmake/build_and_test_asan.sh
diff --git a/.github/workflows/ci_linux_x64_clang_tsan.yml b/.github/workflows/ci_linux_x64_clang_tsan.yml
@@ -8,8 +8,6 @@ name: CI - Linux x64 clang TSan
 
 on:
   pull_request:
-    branches:
-      - main
     paths:
       - ".github/workflows/ci_linux_x64_clang_tsan.yml"
   schedule:
@@ -32,21 +30,29 @@ jobs:
   linux_x64_clang_tsan:
     needs: setup
     runs-on: azure-linux-scale
+    container:
+      # TODO(scotttodd): use ghcr.io/iree-org/cpubuilder_ubuntu_jammy_x86_64 here
+      image: gcr.io/iree-oss/base-bleeding-edge@sha256:cf2e78194e64fd0166f4141317366261d7a62432b72e9a324cb8c2ff4e1a515a
+      # TSan in particular needs some settings that this option includes:
+      #   * https://github.com/google/sanitizers/issues/1716
+      #   * https://security.stackexchange.com/q/214923
+      # Note that we are not using Docker for sandboxing. If we were, this
+      # would *not* be safe.
+      options: --privileged
+    defaults:
+      run:
+        shell: bash
     steps:
       - name: "Checking out repository"
         uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
         with:
           submodules: true
-      - name: "Building and testing with TSan"
+      - name: Install Python requirements
+        run: python3 -m pip install -r ./runtime/bindings/python/iree/runtime/build_requirements.txt
+      # TODO(#18238): add local or remote ccache
+      - name: Build and test with TSan
         env:
-          IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }}
-        run: |
-          # Note that this uses the latest version of the clang compiler, etc.
-          # This gives us access to the latest features and validates that IREE
-          # builds using the latest versions.
-          ./build_tools/github_actions/docker_run.sh \
-            --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \
-            --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \
-            --env "CCACHE_NAMESPACE=base-bleeding-edge@sha256:14200dacca3a0f3a66f8aa87c6f64729b83a2eeb403b689c24204074ad157418" \
-            gcr.io/iree-oss/base-bleeding-edge@sha256:cf2e78194e64fd0166f4141317366261d7a62432b72e9a324cb8c2ff4e1a515a \
-            ./build_tools/cmake/build_and_test_tsan.sh
+          # Use a modern clang explicitly.
+          CC: clang-19
+          CXX: clang++-19
+        run: ./build_tools/cmake/build_and_test_tsan.sh
diff --git a/build_tools/cmake/build_and_test_tsan.sh b/build_tools/cmake/build_and_test_tsan.sh
@@ -47,6 +47,12 @@ CMAKE_ARGS=(
   "-DIREE_HAL_DRIVER_HIP=${IREE_HAL_DRIVER_HIP}"
   "-DIREE_TARGET_BACKEND_CUDA=${IREE_TARGET_BACKEND_CUDA}"
   "-DIREE_TARGET_BACKEND_ROCM=${IREE_TARGET_BACKEND_ROCM}"
+
+  # Workaround for this weird issue:
+  # https://github.com/google/benchmark/issues/773#issuecomment-616067912
+  "-DRUN_HAVE_STD_REGEX=0"
+  "-DRUN_HAVE_POSIX_REGEX=0"
+  "-DCOMPILE_HAVE_GNU_POSIX_REGEX=0"
 )
 
 "${CMAKE_BIN}" -B "${BUILD_DIR}" "${CMAKE_ARGS[@]?}"

diff --git a/docs/website/docs/developers/debugging/sanitizers.md b/docs/website/docs/developers/debugging/sanitizers.md
@@ -155,6 +155,35 @@ ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-12/bin/llvm-symbolizer \
 
 ### TSan (ThreadSanitizer)
 
+To enable TSan:
+
+```shell
+cmake -DIREE_ENABLE_TSAN=ON ...
+```
+
+You may also need:
+
+* Depending on your system (see
+  <https://github.com/google/benchmark/issues/773#issuecomment-616067912>):
+
+    ```shell
+    -DRUN_HAVE_STD_REGEX=0 \
+    -DRUN_HAVE_POSIX_REGEX=0 \
+    -DCOMPILE_HAVE_GNU_POSIX_REGEX=0 \
+    ```
+
+* For clang < 18.1.0 on system with `vm.mmap_rnd_bits` > 28 (see
+  <https://stackoverflow.com/a/77856955>):
+
+    ```shell
+    sudo sysctl vm.mmap_rnd_bits=28
+    ```
+
+    TSan in LLVM >= 18.1.0 supports 30 bits of ASLR entropy. If the layout is
+    unsupported, TSan will automatically re-execute without ASLR.
+
+* If running under Docker, add `--privileged` to your `docker run` command
+
 #### C++ Standard Library with TSan support
 
 For best results to avoid false positives/negatives TSan needs all userspace