[SPARK-32918][SHUFFLE] RPC implementation to support control plane coordination for push-based shuffle

[zhouyejoe]

What changes were proposed in this pull request?

This is one of the patches for SPIP SPARK-30602 which is needed for push-based shuffle.
Summary of changes:
This PR introduces a new RPC to be called within Driver. When the expected shuffle push wait time reaches, Driver will call this RPC to facilitate coordination of shuffle map/reduce stages and notify external shuffle services to finalize shuffle block merge for a given shuffle. Shuffle services also respond back the metadata about a merged shuffle partition back to the caller.

Why are the changes needed?

Refer to the SPIP in SPARK-30602.

Does this PR introduce any user-facing change?

No

How was this patch tested?

This code snippets won't be called by any existing code and will be tested after the coordinated driver changes gets merged in SPARK-32920.

Lead-authored-by: Min Shen mshen@linkedin.com

[AmplabJenkins]

Can one of the admins verify this patch?

[mridulm]

A minor comment. Rest looks good to me.

[attilapiros]

It is very hard (even impossible) to reason about code changes which are not used (as well as not even tested) so please consider cutting the push based shuffle feature as whole into PRs which covers a bit more meaningful parts on its own.

[Victsm]

@attilapiros thanks for that feedback, and we acknowledge the sentiment for the issues with small patches in a large feature.
This is an oversight on my end when initially breaking SPARK-30602 into sub-tasks.
I thought SPARK-32918 could be a reasonable sized patch but it turned out to be not the case.
Both SPARK-32918 and SPARK-32919 are meant to be getting some of the prerequisites in place before getting to SPARK-32920 and SPARK-32921, which are relative major changes to key Spark driver components i.e. DAGScheduler and MapOutputTracker.
The original intent was to make patches for SPARK-32920 and SPARK-32921 focusing on changes to these corresponding components only, given the potentially vigorous reviews we would receive for changing these components.

[mridulm]

@tgravescs, @attilapiros, @Ngone51 are there any other comments ?
Thanks.

[tgravescs]

kicked the builds again as the error should have been fixed by 6da8ade

[mridulm]

Thanks for the +1 @tgravescs ! The scala 2.13 error is unrelated.
Merging to master.

[mridulm]

Thanks for working on this @zhouyejoe !
And thanks for the reviews @tgravescs, @attilapiros, @Ngone51, @otterc and @Victsm !

[zhouyejoe]

Thanks everyone on reviewing this PR.