[SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment #3527

zwangsheng · 2022-09-20T08:55:03Z

Why are the changes needed?

On case spark on kubernetes, spark using env SPARK_USER_NAME as user name.
So kyuubi should build spark engine with this env when proxy user or using keytab.
This conf only affect on kubernetes case.

Ref: apache/spark#23017

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before make a pull request

zwangsheng · 2022-09-20T08:55:41Z

plz help check @pan3793 @turboFei @wForget

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala

codecov-commenter · 2022-09-21T01:50:36Z

Codecov Report

Merging #3527 (9596372) into master (73612b7) will increase coverage by 0.00%.
The diff coverage is 88.88%.

@@            Coverage Diff            @@
##             master    #3527   +/-   ##
=========================================
  Coverage     51.67%   51.67%           
  Complexity       13       13           
=========================================
  Files           482      482           
  Lines         26933    26943   +10     
  Branches       3760     3760           
=========================================
+ Hits          13917    13923    +6     
- Misses        11661    11664    +3     
- Partials       1355     1356    +1

Impacted Files	Coverage Δ
...ache/kyuubi/engine/spark/SparkProcessBuilder.scala	`86.66% <88.88%> (+0.41%)`	⬆️
...ache/kyuubi/operation/KyuubiOperationManager.scala	`79.45% <0.00%> (-2.74%)`	⬇️
...che/kyuubi/server/KyuubiTHttpFrontendService.scala	`64.23% <0.00%> (-0.73%)`	⬇️
...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala	`80.74% <0.00%> (-0.63%)`	⬇️
...g/apache/kyuubi/operation/BatchJobSubmission.scala	`74.64% <0.00%> (ø)`
...a/org/apache/kyuubi/service/TFrontendService.scala	`91.47% <0.00%> (+0.29%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

yaooqinn · 2022-09-21T02:34:30Z

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala

+    // For spark on kubernetes, spark pod using env SPARK_USER_NAME as current user
+    def setSparkUserName(userName: String): Unit = {
+      clusterManager().foreach(cm => {
+        if (cm.startsWith("k8s://")) {


there is no harm to set for all cluster managers？

spark.kubernetes.driverEnv. only affect on kubernetes case. We are only setting up this On Kubernetes for now.
As for SPARK_USER_NAME this env, it's affect in spark/kubernetes/entrypoint.sh

pan3793 · 2022-09-23T04:31:50Z

Thanks, merging to master/1.6

…deployment ### _Why are the changes needed?_ On case spark on kubernetes, spark using env `SPARK_USER_NAME` as user name. So kyuubi should build spark engine with this env when proxy user or using keytab. This conf only affect on kubernetes case. Ref: apache/spark#23017 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3527 from zwangsheng/feature/add_spark_user_name. Closes #3527 9596372 [zwangsheng] only k8s case ddd713f [zwangsheng] fix 48b9b22 [zwangsheng] add Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit 3738512) Signed-off-by: Cheng Pan <chengpan@apache.org>

pan3793 · 2022-09-29T15:50:23Z

@zwangsheng spark.kubernetes.executorEnv.[EnvironmentVariableName] does not take effect, the correct key is spark.executorEnv.[EnvironmentVariableName]
https://spark.apache.org/docs/latest/configuration.html

zwangsheng · 2022-10-08T01:50:03Z

@zwangsheng spark.kubernetes.executorEnv.[EnvironmentVariableName] does not take effect, the correct key is spark.executorEnv.[EnvironmentVariableName] https://spark.apache.org/docs/latest/configuration.html

😭sry, i will make follow PR.

…e spark.executorEnv ### _Why are the changes needed?_ Fix bug in #3527 Modify `spark.kubernetes.executorEnv` to `spark.executorEnv` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request - [x] Unit Test Closes #3591 from zwangsheng/follow/SPARK_USER_NAME. Closes #3527 7254fe9 [zwangsheng] fix executorEnv Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org>

…e spark.executorEnv ### _Why are the changes needed?_ Fix bug in #3527 Modify `spark.kubernetes.executorEnv` to `spark.executorEnv` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request - [x] Unit Test Closes #3591 from zwangsheng/follow/SPARK_USER_NAME. Closes #3527 7254fe9 [zwangsheng] fix executorEnv Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit fb6516a) Signed-off-by: Cheng Pan <chengpan@apache.org>

…AME on submitting Spark batch job to K8s ### _Why are the changes needed?_ Batch job should also submit with `SPARK_USER_NAME`. See more in pre pr #3527 . ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3664 from zwangsheng/feature/batch_with_user_name. Closes #3664 76dee3e [zwangsheng] style 0d40508 [zwangsheng] fix a1fcf7f [zwangsheng] fix Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org>

…AME on submitting Spark batch job to K8s ### _Why are the changes needed?_ Batch job should also submit with `SPARK_USER_NAME`. See more in pre pr #3527 . ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3664 from zwangsheng/feature/batch_with_user_name. Closes #3664 76dee3e [zwangsheng] style 0d40508 [zwangsheng] fix a1fcf7f [zwangsheng] fix Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit 5f8b57b) Signed-off-by: Cheng Pan <chengpan@apache.org>

add

48b9b22

github-actions bot added the module:server label Sep 20, 2022

pan3793 changed the title ~~[FEATURE][SPARK] Build Spark engine with env SPARK_USER_NAME using proxyName or keytab shortUserName~~ [FEATURE][SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment Sep 20, 2022

pan3793 changed the title ~~[FEATURE][SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment~~ [SPARK] Build Spark engine with env SPARK_USER_NAME using proxyName or keytab shortUserName Sep 20, 2022

pan3793 reviewed Sep 20, 2022

View reviewed changes

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala Outdated Show resolved Hide resolved

pan3793 reviewed Sep 20, 2022

View reviewed changes

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala Outdated Show resolved Hide resolved

pan3793 changed the title ~~[SPARK] Build Spark engine with env SPARK_USER_NAME using proxyName or keytab shortUserName~~ [SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment Sep 20, 2022

fix

ddd713f

pan3793 approved these changes Sep 20, 2022

View reviewed changes

pan3793 added this to the v1.6.1 milestone Sep 20, 2022

pan3793 assigned zwangsheng Sep 20, 2022

turboFei approved these changes Sep 20, 2022

View reviewed changes

turboFei reviewed Sep 20, 2022

View reviewed changes

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala Outdated Show resolved Hide resolved

only k8s case

9596372

yaooqinn reviewed Sep 21, 2022

View reviewed changes

pan3793 closed this in 3738512 Sep 23, 2022

zwangsheng mentioned this pull request Oct 8, 2022

[KYUUBI #3527][SPARK][FOLLOWUP] spark.kubernetes.executorEnv should be spark.executorEnv #3591

Closed

4 tasks

zwangsheng mentioned this pull request Oct 19, 2022

[SPARK][K8S][BATCH] Kyuubi should set env SPARK_USER_NAME on submitting Spark batch job to K8s #3664

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment #3527

[SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment #3527

zwangsheng commented Sep 20, 2022 •

edited by pan3793

Loading

zwangsheng commented Sep 20, 2022 •

edited

Loading

codecov-commenter commented Sep 21, 2022 •

edited

Loading

yaooqinn Sep 21, 2022

zwangsheng Sep 21, 2022

pan3793 commented Sep 23, 2022

pan3793 commented Sep 29, 2022

zwangsheng commented Oct 8, 2022

[SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment #3527

[SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment #3527

Conversation

zwangsheng commented Sep 20, 2022 • edited by pan3793 Loading

Why are the changes needed?

How was this patch tested?

zwangsheng commented Sep 20, 2022 • edited Loading

codecov-commenter commented Sep 21, 2022 • edited Loading

Codecov Report

yaooqinn Sep 21, 2022

Choose a reason for hiding this comment

zwangsheng Sep 21, 2022

Choose a reason for hiding this comment

pan3793 commented Sep 23, 2022

pan3793 commented Sep 29, 2022

zwangsheng commented Oct 8, 2022

zwangsheng commented Sep 20, 2022 •

edited by pan3793

Loading

zwangsheng commented Sep 20, 2022 •

edited

Loading

codecov-commenter commented Sep 21, 2022 •

edited

Loading