refactor CUDA versions in dependencies.yaml

[jameslamb]

Description

Follow-up to #14644.

Contributes to rapidsai/build-planning#7.

Similar to rapidsai/rmm#1422, this proposes splitting the cuda-version dependency in dependencies.yaml out to its own thing, separate from the bits of the CUDA Toolkit cudf needs.

Some other simplifications:

• removes the notebook-specific stuff added in Split cuda versions for notebook testing #14722 (which I think were added specifically because cuda-version and CTK stuff was coupled)
• consolidates two sections with selectors only based on CUDA {major}.{minor}

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[vyasr]

Can we drop 11.2 from this file? We still support 11.2 via CEC, but we don't need to be able to build envs with it any more do we?

[vyasr]

Perhaps not really in scope for this PR. Something we can come back to later.

[jakirkham]

If we are supporting it, wouldn't we want a way to construct an environment with this CUDA version for testing, debugging, etc.?

[bdice]

11.2 has been dropped from our CI matrix and Docker images. I think it’s OK to remove this but I agree it’s probably better to keep the scope narrow here. At some point we will drop CUDA 11 entirely and clean up a lot of stuff.

[jameslamb]

Something we can come back to later.

I think it should come later, yeah.

[jameslamb]

docs builds are failing like this:

/__w/cudf/cudf/cpp/doxygen/main_page.md:1: warning: multiple use of section label 'md_main_page' while adding anchor, (first occurrence: index.html)
...
reading sources... [  2%] user_guide/10min
/__w/cudf/cudf/docs/cudf/source/user_guide/10min.ipynb: Executing notebook using local CWD [mystnb]
../../thread/thread_load.cuh(36): warning: cuda.h: [jitify] File not found
../../thread/thread_store.cuh(36): warning: cuda.h: [jitify] File not found
2024-01-11 00:24:38,405 - distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/opt/conda/envs/docs/lib/python3.10/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/opt/conda/envs/docs/lib/python3.10/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
  File "/opt/conda/envs/docs/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 99, in loads
    return pickle.loads(x, buffers=buffers)
AttributeError: Can't get attribute 'func' on <module '__main__' (built-in)>
...
/__w/cudf/cudf/docs/cudf/source/user_guide/10min.ipynb: WARNING: Executing notebook failed: CellTimeoutError [mystnb.exec]
/__w/cudf/cudf/docs/cudf/source/user_guide/10min.ipynb: WARNING: Notebook exception traceback saved in: /__w/cudf/cudf/docs/cudf/build/dirhtml/reports/user_guide/10min.err.log [mystnb.exec]
...
build finished with problems, 2 warnings.
make: *** [Makefile:20: dirhtml] Error 1

(build link)

I re-checked the diff and don't see anything that's obviously related to these warnings and errors. I just tried pushing an empty commit to re-trigger CI. If that doesn't work I'll ask for some help.

[wence-]

/__w/cudf/cudf/docs/cudf/source/user_guide/10min.ipynb: WARNING: Executing notebook failed: CellTimeoutError [mystnb.exec]
/__w/cudf/cudf/docs/cudf/source/user_guide/10min.ipynb: WARNING: Notebook exception traceback saved in: > /__w/cudf/cudf/docs/cudf/build/dirhtml/reports/user_guide/10min.err.log [mystnb.exec]

These shouldn't timeout, I think. Do they run locally without problems?

[jakirkham]

As noted offline by Bradley,

• This happens outside of this change
  • https://github.com/rapidsai/cudf/actions/runs/7479141152/job/20356989102#step:9:619
• It looks related to a recent Distributed change
  • Fall back to pickle in protocol.dumps instead of failing hard dask/distributed#8447

[rjzamora]

Just a note that the CI failure is not RAPIDS related. Issue is here: dask/distributed#8454 (cause is dask/distributed#8443)

[jakirkham]

Thanks all! 🙏