Move spark e2e integration from self-hosted to github-hosted

[Yikun]

This patch moves spark e2e integration from self-hosted runner to github-hosted runner.

• Change self-hosted runner (ubuntu-20.04-spark) to github-hosted runner (ubuntu-20.04)
• Upgrade actions/docker actions to latest to cleanup infra warning
• Add java/scala cache to speed up test.
• Remove Replace mirror because we don't need it in github-hosted runner
• Add resource limited conf to make it work
  • apache/spark support it in master (future 3.4): apache/spark@72d58d5
  • backport to spark branch-3.3: apache/spark@821997b)
• Add volcano log like: [WIP] Add Volcano log collect in Spark IT #2205

Time cost: from 12 mins to 20 mins, I believe it's OK.

Related: #1704

Signed-off-by: Yikun Jiang yikunkero@gmail.com

[Yikun]

cc @william-wang @Thor-wl @martin-g

After this, we can use the github-hosted runner instead the self-hosted runner (currently, a vm in Huawei Cloud ECS).

This way has already used in apache/spark apache/spark@5e6aab4 since Jul. 2022, it's been very stable according to last half year build history.

[Yikun]

[info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (19 seconds, 931 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (51 seconds, 656 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (53 seconds, 628 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (21 seconds, 197 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (35 seconds, 323 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (37 seconds, 470 milliseconds)
[info] Run completed in 4 minutes, 21 seconds.
[info] Total number of tests run: 6
[info] Suites: completed 3, aborted 0
[info] Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

[william-wang]

/lgtm

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: william-wang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [william-wang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment