WIP: [DO NOT MERGE] introduce libraft wheels

[jameslamb]

Replaces #2360, contributes to rapidsai/build-planning#33.

Proposes packaging libraft as a wheel, which is then re-used by pylibraft-cu{11,12} and raft-cu{11,12} (and by libcugraph-cu{11,12} in rapidsai/cugraph#4804).

Notes for Reviewers

If you see this note, that means this is not ready for review.

Dependency Flows

libraft wheels contain all of:

• raft headers (and everything they require)
• libraft.a (static library)
• libraft.so (shared object)

libcugraph needs libraft at build time to statically link against libraft.a.

pylibcugraph and cugraph need libraft at build time just to get the RAFT headers (since those are used in libcugraph's public headers).

---
title: Build dependencies
---
flowchart TD
    A[libraft] -->B[pylibraft]
    A --> C[raft-dask]
    A --> D[libcugraph]
    D --> E[pylibcugraph]
    D --> F[cugraph]
    A --> E
    B --> E
    A --> F
    B --> F
    E --> F

Loading

Nothing in cugraph, would need libraft at runtime, because libcugraph is statically linked against libraft.

---
title: Runtime dependencies
---
flowchart LR
    A[libraft] --> B[pylibraft]
    A --> C[raft-dask]

Loading

Presumably cuml and wholegraph could follow the same pattern as cugraph.

Size changes (CUDA 12, Python 3.12, x86_64)

wheel	num files (before)	num files (after)	size (before)	size (this PR)
libraft.	---	---	---	---
pylibraft	---	---	---	---
raft-dask	---	---	---	---
libcugraph	---	---	---	---
pylibcugraph	190	---	900M	---
cugraph	314	---	901M	---
TOTAL	****	---	****	---

NOTES: size = compressed, "before" = 2024-12-17 nightlies (rapidsai/cugraph@5c8f850), cugraph libraries from rapidsai/cugraph#4804

how I calculated those (click me)

docker run \
    --rm \
    -v $(pwd):/opt/work:ro \
    -w /opt/work \
    --network host \
    --env RAPIDS_NIGHTLY_DATE=2024-12-06 \
    --env RAPIDS_NIGHTLY_SHA=5c8f850 \
    --env RAPIDS_PR_NUMBER=4804 \
    --env RAPIDS_PY_CUDA_SUFFIX=cu12 \
    --env RAPIDS_REPOSITORY=rapidsai/cugraph \
    --env WHEEL_DIR_BEFORE=/tmp/wheels-before \
    --env WHEEL_DIR_AFTER=/tmp/wheels-after \
    -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.12 \
    bash

mkdir -p "${WHEEL_DIR_BEFORE}"
mkdir -p "${WHEEL_DIR_AFTER}"

cpp_projects=(
    libcugraph
)
py_projects=(
    cugraph
    pylibcugraph
)

for project in "${py_projects[@]}"; do
    # before
    RAPIDS_BUILD_TYPE=nightly \
    RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \
    RAPIDS_REF_NAME="branch-25.02" \
    RAPIDS_SHA=${RAPIDS_NIGHTLY_SHA} \
        rapids-download-wheels-from-s3 python "${WHEEL_DIR_BEFORE}"

    # after
    #RAPIDS_BUILD_TYPE=pull-request \
    #RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \
    #RAPIDS_REF_NAME="pull-request/${RAPIDS_PR_NUMBER}" \
    #    rapids-download-wheels-from-s3 python "${WHEEL_DIR_AFTER}"
done

for project in "${cpp_projects[@]}"; do    
    # after
    RAPIDS_BUILD_TYPE=pull-request \
    RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \
    RAPIDS_REF_NAME="pull-request/${RAPIDS_PR_NUMBER}" \
        rapids-download-wheels-from-s3 cpp "${WHEEL_DIR_AFTER}"
done

pip install pydistcheck
pydistcheck \
    --inspect \
    --select 'distro-too-large-compressed' \
    ${WHEEL_DIR_BEFORE}/*.whl

pydistcheck \
    --inspect \
    --select 'distro-too-large-compressed' \
    ${WHEEL_DIR_AFTER}/*.whl

[jameslamb]

/ok to test

[jameslamb]

/ok to test

[jameslamb]

/ok to test

[jameslamb]

/ok to test