Remove build isolation in wheel builds

[jameslamb]

Description

Currently, RAPIDS projects build wheels with build isolation, like pip wheel ..

With build isolation, the build tool (in our case, pip wheel), creates a virtual environment and installs of the package's build-time dependencies into it. That virtual environment is created at a random location on each build, which means source files included from other wheels (like the rmm headers from the librmm wheel) are found at a different path on each build in CI.

Path changes lead to cache misses with sccache, and therefore longer-running and more memory-intensive builds.

As more of RAPIDS build-time dependencies are provided wheels (e.g. #33), this problem is going to get worse (see #33 (comment)).

This issue tracks the work of removing build isolation in RAPIDS wheel-building CI jobs.

Benefits of this work

Better sccache hit rate = faster, cheaper CI (contributes to #95).

Stronger guarantees that packages built in CI are actually installed at test time (i.e. would close #79).

Acceptance Criteria

• all RAPIDS C/C++ wheel builds in CI build with --no-build-isolation

Approach

Follow the approach from rapidsai/cuspatial#1473, and documented in #112.

• pure Python? build isolation
• Python + Cython? build isolation
• mostly C/C++? no build isolation

Notes

These can be done in any order.

Updates

Beta Give feedback

1. build-planning: document why we only use build isolation for non-Cython C++ builds ([DO NOT MERGE] document why and when --no-build-isolation is used in wheel-building CI #112)
2. cucim (updated 10/23 build wheels without build isolation, print sccache stats in builds cucim#792)
3. cudf (updated 10/23 WIP: [DO NOT MERGE] build wheels without build isolation cudf#17088)
4. cugraph (updated 10/23 WIP: [DO NOT MERGE] build wheels without build isolation cugraph#4719)
5. cuml (updated 10/23 WIP: [DO NOT MERGE] build wheels without build isolation cuml#6111)
6. cuopt
7. cuspatial (build wheels without build isolation cuspatial#1473)
8. cuvs (updated 10/23 WIP: [DO NOT MERGE] build wheels without build isolation cuvs#413)
9. cuxfilter (remove unnecessary sccache configuration, reduce verbosity of wheel-building scripts cuxfilter#641)
10. dask-cuda (remove unnecessary cmake and sccache configuration dask-cuda#1400)
11. kvikio
12. pynvjitlink
13. raft (WIP: [DO NOT MERGE] build wheels without build isolation raft#2470)
14. rmm (nothing to do ... librmm is header-only and rmm is only Python + Cython)
15. ucx-py (nothing to do ... only Python + Cython)
16. ucxx (WIP: [DO NOT MERGE] build wheels without build isolation ucxx#301)

Do the GNN repos last, as some of the cugraph repos are currently being shuffled around. See e.g. rapidsai/cugraph-gnn#58 (comment).

GNN repos

Beta Give feedback

1. cugraph-gnn
2. cugraph-ops (https://github.com/rapidsai/cugraph-ops/pull/700)
3. wholegraph

[jameslamb]

I've updated the description to reflect our decision from an offline conversation:

• pure Python? build isolation
• just Python + Cython? build isolation
• mostly C/C++? no build isolation

And proposed documenting that in the build-planning docs: #112

[vyasr]

We shouldn't need to do this for any of the packages that haven't switched to using C++ wheels from other libraries yet, i.e. anything that is still statically linking everything. cugraph, cuml, cuvs, and raft (that may be exhaustive, or I may have missed some) don't have relevant changes yet.