Skip to content

Commit

Permalink
[SPARK-40382][SQL][TESTS][FOLLOW-UP] Make the tests pass with ANSI en…
Browse files Browse the repository at this point in the history
…abled

### What changes were proposed in this pull request?

This PR is a followup of #37825, that change the types in the test relation to make the tests pass with ANSI enalbed.

### Why are the changes needed?

To recover the test coverage. Currently it fails with ANSI mode on: https://github.com/apache/spark/actions/runs/3246829492/jobs/5326051798#step:9:20487

```
[info] - SPARK-40382: eliminate multiple distinct groups due to superficial differences *** FAILED *** (5 milliseconds)
[info]   org.apache.spark.sql.AnalysisException: [DATATYPE_MISMATCH.BINARY_OP_WRONG_TYPE] Cannot resolve "(b + c)" due to data type mismatch: the binary operator requires the input type ("NUMERIC" or "INTERVAL DAY TO SECOND" or "INTERVAL YEAR TO MONTH" or "INTERVAL"), not "STRING".
[info]   at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.dataTypeMismatch(package.scala:68)
[info]   at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$5(CheckAnalysis.scala:223)
[info]   at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$5$adapted(CheckAnalysis.scala:210)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:295)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:294)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:294)
[info]   at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[info]   at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[info]   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:294)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:2
```

### Does this PR introduce _any_ user-facing change?

No, test-only.

### How was this patch tested?
Manually ran the tests locally.

Closes #38250 from HyukjinKwon/SPARK-40382-followup.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
HyukjinKwon committed Oct 14, 2022
1 parent 9d28909 commit 8a28e91
Showing 1 changed file with 3 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ class RewriteDistinctAggregatesSuite extends PlanTest {
val nullInt = Literal(null, IntegerType)
val nullString = Literal(null, StringType)
val testRelation = LocalRelation($"a".string, $"b".string, $"c".string, $"d".string, $"e".int)
val testRelation2 = LocalRelation($"a".double, $"b".int, $"c".int, $"d".int, $"e".int)

private def checkRewrite(rewrite: LogicalPlan): Unit = rewrite match {
case Aggregate(_, _, Aggregate(_, _, _: Expand)) =>
Expand Down Expand Up @@ -77,7 +78,7 @@ class RewriteDistinctAggregatesSuite extends PlanTest {
}

test("SPARK-40382: eliminate multiple distinct groups due to superficial differences") {
val input = testRelation
val input = testRelation2
.groupBy($"a")(
countDistinct($"b" + $"c").as("agg1"),
countDistinct($"c" + $"b").as("agg2"),
Expand All @@ -92,7 +93,7 @@ class RewriteDistinctAggregatesSuite extends PlanTest {
}

test("SPARK-40382: reduce multiple distinct groups due to superficial differences") {
val input = testRelation
val input = testRelation2
.groupBy($"a")(
countDistinct($"b" + $"c" + $"d").as("agg1"),
countDistinct($"d" + $"c" + $"b").as("agg2"),
Expand Down

0 comments on commit 8a28e91

Please sign in to comment.