[rapids] removed spark tests, updated to a more recent rapids release #1219
+670
−335
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I prefer this to #1218 |
2 tasks
|
/gcbrun |
cjac
changed the title
|
/gcbrun |
2 similar comments
|
/gcbrun |
|
/gcbrun |
|
Should we increase the machine type from n1-standard-4 to n1-standard-16 |
|
cuda11 has been manually tested with all versions. |
|
/gcbrun |
1 similar comment
|
/gcbrun |
|
tests are failing for
|
cjac
changed the title
|
/gcbrun |
1 similar comment
|
/gcbrun |
|
[edit: this was a misconfiguration in the systemd unit] It looks like the dask infrastructure is out of date and I'll have to target 2023.12 instead. |
|
I also need to reduce the python abi to 3.10 |
cjac
changed the title
cjac
changed the title
|
/gcbrun |
6 similar comments
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
…and rapids init actions
|
/gcbrun |
|
/gcbrun |
jakirkham
reviewed
Oct 14, 2024
rapids/rapids.sh
Outdated
| function is_cuda12() { [[ "${CUDA_VERSION%%.*}" == "12" ]] ; } | ||
| function is_cuda11() { [[ "${CUDA_VERSION%%.*}" == "11" ]] ; } | ||
|
|
||
| readonly DEFAULT_DASK_RAPIDS_VERSION="24.08" |
There was a problem hiding this comment.
Suggested change
| readonly DEFAULT_DASK_RAPIDS_VERSION="24.08" | |
| readonly DEFAULT_DASK_RAPIDS_VERSION="24.10" |
rapids/rapids.sh
Outdated
| # SPARK config | ||
| readonly DEFAULT_SPARK_RAPIDS_VERSION="24.08.0" |
There was a problem hiding this comment.
Suggested change
| readonly DEFAULT_SPARK_RAPIDS_VERSION="24.08.0" | |
| readonly DEFAULT_SPARK_RAPIDS_VERSION="24.10.0" |
|
/gcbrun |
I'm having a hard time with 24.08 right now due to custreamz not matching: |
|
Is Dask installed before RAPIDS is installed? If so, could they be combined into the same install step? Think that would minimize conflicts Should add RAPIDS 24.10 was recently released so is using a newer version of Dask, which also may minimize conflicts |
cjac
changed the title
|
okay, I have a lot of changes to merge from some work I did while I was adding secure-boot support to the custom-images repo. It's kind of a lot. |
rapids/BUILD * removed dependence on verify_xgboost_spark.scala - this belongs in [spark-rapids] * removed dependence on dask rapids/rapids.sh * added utility functions * reverted dask_spec="dask>=2024.5" * using realpath to /opt/conda/miniconda3/bin/mamba instead of default symlink * remove conda environment [dask] if installed * asserting existence of directory depended on by the script when run as custom-images script * created exit_handler and prepare_to_install functions to set up and clean up rapids/test_rapids.py * refactored to make use of systemd unit defined in rapids.sh * added retry to ssh * removed condition to keep tests from running on 2.0 images
|
/gcbrun |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tested with CUDA=11 and CUDA=12