Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixing rapids test for presubmits #1218

Closed
wants to merge 2 commits into from
Closed

Fixing rapids test for presubmits #1218

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6c38d9a
Fixing rapids test for presubmits
prince-cs Aug 8, 2024
83e048d
Fixing rapids test for presubmits
prince-cs Aug 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions rapids/rapids.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,12 @@ function get_metadata_attribute() {
/usr/share/google/get_metadata_value "attributes/${attribute_name}" || echo -n "${default_value}"
}

readonly DEFAULT_DASK_RAPIDS_VERSION="23.12"
# Until mamba includes packages for cuda 12, this should stay at 22.04
readonly DEFAULT_DASK_RAPIDS_VERSION="22.04"
readonly RAPIDS_VERSION=$(get_metadata_attribute 'rapids-version' ${DEFAULT_DASK_RAPIDS_VERSION})

readonly SPARK_VERSION_ENV=$(spark-submit --version 2>&1 | sed -n 's/.*version[[:blank:]]\+\([0-9]\+\.[0-9]\).*/\1/p' | head -n1)
readonly DEFAULT_SPARK_RAPIDS_VERSION="22.10.0"
readonly DEFAULT_SPARK_RAPIDS_VERSION="23.12.0"

if [[ "${SPARK_VERSION_ENV%%.*}" == "3" ]]; then
readonly DEFAULT_CUDA_VERSION="11.8"
Expand All @@ -88,7 +89,7 @@ readonly RUN_WORKER_ON_MASTER=$(get_metadata_attribute 'dask-cuda-worker-on-mast

# RAPIDS config
CUDA_VERSION=$(get_metadata_attribute 'cuda-version' ${DEFAULT_CUDA_VERSION})
if [[ "${CUDA_VERSION%%.*}" == 12 ]]; then
if [[ "${CUDA_VERSION%%.*}" == "12" ]]; then
# at the time of writing 20240721 there is no support for the 12.x
# releases of cudatoolkit package in mamba. For the time being,
# we will use a maximum of 11.8
Expand Down Expand Up @@ -131,7 +132,7 @@ function install_dask_rapids() {
fi
# Install RAPIDS, cudatoolkit
mamba install -m -n 'dask-rapids' -y --no-channel-priority -c 'conda-forge' -c 'nvidia' -c 'rapidsai' \
"cudatoolkit=${CUDA_VERSION}" "pandas<1.5" "rapids=${RAPIDS_VERSION}" "python=${python_ver}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, we need to allow customer to select by rapids version. This needs to stay.

"cudatoolkit=${CUDA_VERSION%%.*}" "pandas<1.5" "rapids" "python=${python_ver}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

selecting by major version is right.

}

function install_spark_rapids() {
Expand Down
27 changes: 0 additions & 27 deletions rapids/test_rapids.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,33 +53,6 @@ def verify_spark_job(self):
self.remove_test_script(self.XGBOOST_SPARK_TEST_SCRIPT_FILE_NAME,
instance_name)

@parameterized.parameters(("STANDARD", ["m", "w-0"], GPU_T4, None),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what? no. We need to make the test pass, not delete the test. It's working for me. It takes 50+ minutes for cuda version 12. Have I not created a PR yet?
No, I have not. Let me push. We'll want to abandon this for #1219 I think.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It got pushed by mistake. This change was for my local testing.

("STANDARD", ["m", "w-0"], GPU_T4, "yarn"),
("STANDARD", ["m"], GPU_T4, "standalone"))
def test_rapids_dask(self, configuration, machine_suffixes, accelerator,
dask_runtime):

if self.getImageVersion() <= pkg_resources.parse_version("2.0"):
self.skipTest("Not supported in pre 2.0 images")

metadata = "gpu-driver-provider=NVIDIA,rapids-runtime=DASK"
if dask_runtime:
metadata += ",dask-runtime={}".format(dask_runtime)

self.createCluster(
configuration,
self.DASK_INIT_ACTIONS,
metadata=metadata,
machine_type="n1-standard-4",
master_accelerator=accelerator,
worker_accelerator=accelerator,
boot_disk_size="200GB",
timeout_in_minutes=30)

for machine_suffix in machine_suffixes:
self.verify_dask_instance("{}-{}".format(self.getClusterName(),
machine_suffix))

@parameterized.parameters(("SINGLE", ["m"], GPU_T4),
("STANDARD", ["w-0"], GPU_T4))
def test_rapids_spark(self, configuration, machine_suffixes, accelerator):
Expand Down