Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bifurcate Dependency Lists [skip-gpuci] #1065

Merged
merged 8 commits into from
Dec 7, 2022

Conversation

ajschmidt8
Copy link
Member

This PR uses the rapids-dependency-file-generator to handle sourcing dependencies.

Similarly to rapidsai/rmm#1073 and rapidsai/cudf#11674, this PR introduces a GitHub Action that enforces consistency between the new dependencies.yaml file and the generated conda environment for developers.

@ajschmidt8 ajschmidt8 added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Dec 1, 2022
@github-actions github-actions bot added the gpuCI label Dec 5, 2022
@ajschmidt8 ajschmidt8 marked this pull request as ready for review December 5, 2022 21:49
@ajschmidt8 ajschmidt8 requested a review from a team as a code owner December 5, 2022 21:49
Copy link
Member

@cjnolet cjnolet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks AJ!

packages:
- c-compiler
- cxx-compiler
- rapids-build-env=23.02.*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ajschmidt8 Did you say this package was no longer needed on a different PR?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. For this PR, the intention is just to replace the manually managed conda/environment files with the dfg. I figured we can remove these packages in the follow-up GHA PR.

If we remove them before, it could break the environment. In the GHA PR, we can remove them and then identify which dependencies we need to re-add after they're removed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, that sounds fine to me. We'll clean up later.

common:
- output_types: [conda, requirements]
packages:
- clang=11.1.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can drop clang/clang-tools, which means the entire develop list could be deleted. I don't think these are used anywhere. clang-format (the original reason this was added, afaik) is now handled via pre-commit.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe @cjnolet said that they do/plan to run clang.

I notice this script among others https://github.com/rapidsai/raft/blob/branch-23.02/cpp/scripts/run-clang-tidy.py

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think only clang-tools is required for providing the clang-tidy binary. No scripts are calling clang as far as I can tell.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are in the process of updating our code so we can support clang. I'd prefer to leave in the dependency in the meantime (we do have an experimental clang compile script in there that we're working on making build successfully).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. We can resolve this.

- output_types: [conda]
packages:
- doxygen>=1.8.20
- rapids-doc-env=23.02.*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise, is this still needed?

common:
- output_types: [conda]
packages:
- rapids-notebook-env=23.02.*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still needed?

- ucx>=1.13.0
- ucx-py=0.30.*
- ucx-proc=*=gpu
- libfaiss>=1.7.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed a really annoying issue recently in rapids-compose. It merges the dependency lists / conda channels, and ends up placing the pytorch channel (which is used in cudf tests) above conda-forge. This in turn means that libfaiss is pulled from pytorch and it ignores the faiss-proc=*=cuda package below, which only affects libfaiss from conda-forge. 🙃 Just something to be aware of. We may need to specify libfaiss>=1.7.0=cuda* or something like that, in addition to faiss-proc=*=cuda. Another solution would be to fix the conda-forge packages of pytorch and never use the pytorch channel for any dependencies.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just ran into the same issue and fixed it in the old env files, but then I realized this PR was ready to go. I'll make a PR with the additional pinning, which is how I fixed it in my case and it worked fine.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like this is a valid port of the existing environment to use dependencies.yaml, but it leaves us with some cleanup tasks for later in the GitHub Actions migration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
gpuCI improvement Improvement / enhancement to an existing function non-breaking Non-breaking change
Projects
Development

Successfully merging this pull request may close these issues.

4 participants