[SQL] [WIP] Join test coverage improvements

[JoshRosen]

This WIP patch aims to improve the test coverage of Spark SQL's join operators by adding SparkPlanTest tests for the physical operators.

(Some of these changes are extracted from #7904 for easier review).

[SparkQA]

Test build #39980 has finished for PR 7985 at commit 7a845bf.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class VectorSlicer(override val uid: String)

[SparkQA]

Test build #39981 has finished for PR 7985 at commit 0fa69a7.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class InnerJoinSuite extends SparkPlanTest

[JoshRosen]

BroadcastNestedLoopJoin can't handle non-outer joins, but there was nothing to guard against this (this would never end up being planned though).

[SparkQA]

Test build #39984 has finished for PR 7985 at commit be61fbe.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

My goal here is to keep adding tests until I hit 100% line coverage (and ideally 100% branch coverage) on SortMergeJoin.

[SparkQA]

Test build #39993 has finished for PR 7985 at commit 2048cad.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class VectorSlicer(override val uid: String)

[JoshRosen]

Open question of whether I should be using EnsureRequirements here or not. On the one hand, this allows us to test whether these operators properly declare their requirements / input preconditions in a way that the optimizer can satisfy them.

On the other hand, this makes it harder to exercise precise control over the input iterators that the physical operators see, leaving certain parts of the test orchestration a little more "up to coincidence." For instance, let's say that I want to precisely control the input: I can create left and right plans with single partitions and just trust that the Exchange is not going to unnecessarily re-shuffle before sorting, but that's brittle if someone breaks Exchange to mis-handle this case (see #7988, which specifically fixes an issue that would have broken this strategy).

I guess the way forward is to state the Exchange / EnsureRequirements behavior requirements here, add unit tests for them in PlannerSuite, then assume those requirements here and build on top of them.

[davies]

@JoshRosen Should we merge this before #7904 ?

[JoshRosen]

I have decided to merge these changes into the SMJ patch since it's easier to develop both patches at the same time.

[SparkQA]

Test build #40333 has finished for PR 7985 at commit c700df8.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.