Skip to content

Commit

Permalink
[SPARK-45050][SQL][CONNECT] Improve error message for UNKNOWN io.grpc…
Browse files Browse the repository at this point in the history
….StatusRuntimeException

### What changes were proposed in this pull request?

- Improve error message for UNKNOWN io.grpc.StatusRuntimeException

Before:
```
[info] - handle unknown exception *** FAILED *** (15 milliseconds)
[info]   org.apache.spark.SparkException:
[info]   at org.apache.spark.sql.connect.client.GrpcExceptionConverter$.toThrowable(GrpcExceptionConverter.scala:110)
[info]   at org.apache.spark.sql.connect.client.GrpcExceptionConverter$.convert(GrpcExceptionConverter.scala:41)
[info]   at org.apache.spark.sql.connect.client.GrpcExceptionConverter$$anon$1.hasNext(GrpcExceptionConverter.scala:49)
[info]   at org.apache.spark.sql.connect.client.SparkResult.org$apache$spark$sql$connect$client$SparkResult$$processResponses(SparkResult.scala:83)
[info]   at org.apache.spark.sql.connect.client.SparkResult.length(SparkResult.scala:153)
[info]   at org.apache.spark.sql.connect.client.SparkResult.toArray(SparkResult.scala:183)
[info]   at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2910)
[info]   at org.apache.spark.sql.Dataset.withResult(Dataset.scala:3350)
[info]   at org.apache.spark.sql.Dataset.collect(Dataset.scala:2909)
[info]   at org.apache.spark.sql.ClientE2ETestSuite.$anonfun$new$19(ClientE2ETestSuite.scala:118)
```
After:
```
[info] - handle unknown exception *** FAILED *** (21 milliseconds)
[info]   org.apache.spark.SparkException: io.grpc.StatusRuntimeException: UNKNOWN
[info]   at org.apache.spark.sql.connect.client.GrpcExceptionConverter$.toThrowable(GrpcExceptionConverter.scala:110)
[info]   at org.apache.spark.sql.connect.client.GrpcExceptionConverter$.convert(GrpcExceptionConverter.scala:41)
[info]   at org.apache.spark.sql.connect.client.GrpcExceptionConverter$$anon$1.hasNext(GrpcExceptionConverter.scala:49)
[info]   at org.apache.spark.sql.connect.client.SparkResult.org$apache$spark$sql$connect$client$SparkResult$$processResponses(SparkResult.scala:83)
[info]   at org.apache.spark.sql.connect.client.SparkResult.length(SparkResult.scala:153)
[info]   at org.apache.spark.sql.connect.client.SparkResult.toArray(SparkResult.scala:183)
[info]   at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2910)
[info]   at org.apache.spark.sql.Dataset.withResult(Dataset.scala:3350)
[info]   at org.apache.spark.sql.Dataset.collect(Dataset.scala:2909)
[info]   at org.apache.spark.sql.ClientE2ETestSuite.$anonfun$new$19(ClientE2ETestSuite.scala:118)
```

### Why are the changes needed?

- Better readability of the exception message

### Does this PR introduce _any_ user-facing change?

- No

### How was this patch tested?

- build/sbt "connect-client-jvm/testOnly *ClientE2ETestSuite"

### Was this patch authored or co-authored using generative AI tooling?

Closes #42771 from heyihong/SPARK-45050.

Authored-by: Yihong He <yihong.he@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit e82805d)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
heyihong authored and HyukjinKwon committed Sep 7, 2023
1 parent c98ade3 commit 916b6f5
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,18 @@ class ClientE2ETestSuite extends RemoteSparkSession with SQLHelper with PrivateM
assert(df.collect().length == 501)
}

test("handle unknown exception") {
var df = spark.range(1)
val limit = spark.conf.get("spark.connect.grpc.marshallerRecursionLimit").toInt + 1
for (a <- 1 to limit) {
df = df.union(spark.range(a, a + 1))
}
val ex = intercept[SparkException] {
df.collect()
}
assert(ex.getMessage.contains("io.grpc.StatusRuntimeException: UNKNOWN"))
}

test("many tables") {
withSQLConf("spark.sql.execution.arrow.maxRecordsPerBatch" -> "10") {
val numTables = 20
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
private def toThrowable(ex: StatusRuntimeException): Throwable = {
val status = StatusProto.fromThrowable(ex)

val fallbackEx = new SparkException(status.getMessage, ex.getCause)
val fallbackEx = new SparkException(ex.toString, ex.getCause)

val errorInfoOpt = status.getDetailsList.asScala
.find(_.is(classOf[ErrorInfo]))
Expand Down

0 comments on commit 916b6f5

Please sign in to comment.