[SPARK-38921][K8S][Tests] Use k8s-client to create queue resource in Volcano IT

[Yikun]

What changes were proposed in this pull request?

Use fabric8io/k8s-client to create queue resource in Volcano IT.

Why are the changes needed?

Use k8s-client to create volcano queue to

• Make code easy to understand
• Enable abity to set queue capacity dynamically. This will help to support running Volcano test in a resource limited env (such as github action).

Does this PR introduce any user-facing change?

No

How was this patch tested?

Volcano IT passed

[Yikun]

[info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (10 seconds, 308 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (24 seconds, 557 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (27 seconds, 589 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (13 seconds, 256 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (22 seconds, 275 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (13 seconds, 411 milliseconds)
[info] Run completed in 1 minute, 56 seconds.
[info] Total number of tests run: 6
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 145 s (02:25), completed 2022-4-16 16:59:30

This is manually volcano test results.

[Yikun]

Here is complete test results (all k8s test also include volcano) with Yikun@55af9d8 (will add a pr after this patch merged):

https://github.com/Yikun/spark/runs/6046699187?check_suite_focus=true

[Yikun]

All exsiting CI passed: https://github.com/Yikun/spark/actions/runs/2176142324

[Yikun]

@dongjoon-hyun Could you take a look? thanks!

[dongjoon-hyun]

Thank you, @Yikun .

• I don't think this is an urgent refactoring because there is no test coverage change here.
• Refactoring always increases the complexity of backporting process. This is not helpful during 3.3 release process.
• Hence, I'd like to recommend to keep this PR open for a while until Apache Spark 3.3 release and revisit after that.

[Yikun]

@dongjoon-hyun Sure, I'm fine with it.

[dongjoon-hyun]

Thank you, @Yikun .

[dongjoon-hyun]

Hi @Yikun . I'm revisiting K8s PRs for Apache Spark 3.4.
Could you rebase this PR and make all CIs green, please?

[Yikun]

@dongjoon-hyun Thanks for taking care of this! Will rebase and retest soon

[Yikun]

info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (9 seconds, 265 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (22 seconds, 460 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (22 seconds, 562 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (14 seconds, 628 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (26 seconds, 966 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (16 seconds, 390 milliseconds)
[info] Run completed in 1 minute, 58 seconds.
[info] Total number of tests run: 6
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 132 s (02:12), completed 2022-8-11 11:47:56

[dongjoon-hyun]

+1, LGTM.

[dongjoon-hyun]

Merged to master for Apache Spark 3.4.0.

[Yikun]

Backport to branch-3.3 (3.3.2), See also #38789 (comment) .