Add support for affinity and tolerations, refactor unit tests #6232
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jason Parraga <sovietaced@gmail.com>
Code Review Agent Run Status
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #6232 +/- ##
=======================================
Coverage 36.87% 36.87%
=======================================
Files 1318 1318
Lines 134647 134653 +6
=======================================
+ Hits 49647 49657 +10
+ Misses 80679 80676 -3
+ Partials 4321 4320 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Code Review Agent Run #d27a94Actionable Suggestions - 4
Review Details
AI Code Review powered by |
Changelist by BitoThis pull request implements the following key changes.
|
flyte-bot
reviewed
Feb 8, 2025
flyte-bot
reviewed
Feb 8, 2025
flyte-bot
reviewed
Feb 8, 2025
flyte-bot
reviewed
Feb 8, 2025
Signed-off-by: Jason Parraga <sovietaced@gmail.com>
Code Review Agent Run #3fcfcfActionable Suggestions - 0Review Details
AI Code Review powered by |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tracking issue
Closes #6229
Why are the changes needed?
If you are using ray to do distributed training on GPUs you may want to use tolerations and node affinity to ensure GPU workloads land on GPU nodes.
What changes were proposed in this pull request?
Updates the ray plugin to cherry pick tolerations and affinity if they are non-null or non-empty in the customized head/worker pod specs.
I also changed how assertions are made in the unit tests since they were becoming cumbersome in the previous style.
How was this patch tested?
Unit tests
Check all the applicable boxes
Related PRs
Docs link
Summary by Bito
This PR enhances the Ray plugin by implementing pod tolerations, node affinity configurations, and resource requirements for Ray head and worker pods. The changes enable custom scheduling requirements particularly for GPU workloads and include merging tolerations and affinity settings from custom pod specifications. The implementation includes refactoring of affinity configurations to ensure proper handling of resource specifications in Ray clusters, along with improved test suite organization and comprehensive test coverage.Unit tests added: True
Estimated effort to review (1-5, lower is better): 2