Skip to content

Commit

Permalink
[SPARK-25883][BACKPORT][SQL][MINOR] Override method prettyName in `…
Browse files Browse the repository at this point in the history
…from_avro`/`to_avro`

Back port apache#22890 to branch-2.4.
It is a bug fix for this issue:
https://issues.apache.org/jira/browse/SPARK-26063

## What changes were proposed in this pull request?

Previously in from_avro/to_avro, we override the method `simpleString` and `sql` for the string output. However, the override only affects the alias naming:
```
Project [from_avro('col,
...
, (mode,PERMISSIVE)) AS from_avro(col, struct<col1:bigint,col2:double>, Map(mode -> PERMISSIVE))logicalclocks#11]
```
It only makes the alias name quite long: `from_avro(col, struct<col1:bigint,col2:double>, Map(mode -> PERMISSIVE))`).

We should follow `from_csv`/`from_json` here, to override the method prettyName only, and we will get a clean alias name

```
... AS from_avro(col)logicalclocks#11
```

## How was this patch tested?

Manual check

Closes apache#23047 from gengliangwang/backport_avro_pretty_name.

Authored-by: Gengliang Wang <gengliang.wang@databricks.com>
Signed-off-by: hyukjinkwon <gurwls223@apache.org>
  • Loading branch information
gengliangwang authored and kai-chi committed Aug 1, 2019
1 parent cc53fe0 commit 1a3251a
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 14 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -51,13 +51,7 @@ case class AvroDataToCatalyst(child: Expression, jsonFormatSchema: String)
deserializer.deserialize(result)
}

override def simpleString: String = {
s"from_avro(${child.sql}, ${dataType.simpleString})"
}

override def sql: String = {
s"from_avro(${child.sql}, ${dataType.catalogString})"
}
override def prettyName: String = "from_avro"

override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val expr = ctx.addReferenceObj("this", this)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,7 @@ case class CatalystDataToAvro(child: Expression) extends UnaryExpression {
out.toByteArray
}

override def simpleString: String = {
s"to_avro(${child.sql}, ${child.dataType.simpleString})"
}

override def sql: String = {
s"to_avro(${child.sql}, ${child.dataType.catalogString})"
}
override def prettyName: String = "to_avro"

override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val expr = ctx.addReferenceObj("this", this)
Expand Down

0 comments on commit 1a3251a

Please sign in to comment.