-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-38802][K8S][TESTS] Add Support for spark.kubernetes.test.(driver|executor)RequestCores
#36087
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM. Thank you, @Yikun . Merged to master for Apache Spark 3.4.
…ver|executor)RequestCores` ### What changes were proposed in this pull request? This patch adds support for `spark.kubernetes.test.(driver|executor)RequestCores`, this help devs set specific cores info for (driver|executor)RequestCores. ### Why are the changes needed? In some cases (such as resource limited case), we want to set specifc `RequestCores`. See also: apache#35830 (review) ### Does this PR introduce _any_ user-facing change? No, test only ### How was this patch tested? IT passed Closes apache#36087 from Yikun/SPARK-38802. Authored-by: Yikun Jiang <yikunkero@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
@dongjoon-hyun Could we backport this to branch-3.3, this will very help to run branch-3.3 K8S in github action (such as validate specific docker images). also cc @HyukjinKwon |
And also this #36096 |
Here is a simple demo to show why we need them: Yikun/spark-docker#5
|
Sure, @Yikun . This test commit is backported to branch-3.3 according to the community request. |
…ver|executor)RequestCores` ### What changes were proposed in this pull request? This patch adds support for `spark.kubernetes.test.(driver|executor)RequestCores`, this help devs set specific cores info for (driver|executor)RequestCores. ### Why are the changes needed? In some cases (such as resource limited case), we want to set specifc `RequestCores`. See also: #35830 (review) ### Does this PR introduce _any_ user-facing change? No, test only ### How was this patch tested? IT passed Closes #36087 from Yikun/SPARK-38802. Authored-by: Yikun Jiang <yikunkero@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 8396382) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
@dongjoon-hyun Much thanks! |
What changes were proposed in this pull request?
This patch adds support for
spark.kubernetes.test.(driver|executor)RequestCores
, this help devs set specific cores info for (driver|executor)RequestCores.Why are the changes needed?
In some cases (such as resource limited case), we want to set specifc
RequestCores
.See also: #35830 (review)
Does this PR introduce any user-facing change?
No, test only
How was this patch tested?
IT passed