[SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null #3538

YanTangZhai · 2014-12-01T13:24:37Z

val jsc = new org.apache.spark.api.java.JavaSparkContext(sc)
val jhc = new org.apache.spark.sql.hive.api.java.JavaHiveContext(jsc)
val nrdd = jhc.hql("select null from spark_test.for_test")
println(nrdd.schema)
Then the error is thrown as follows:
scala.MatchError: NullType (of class org.apache.spark.sql.catalyst.types.NullType$)
at org.apache.spark.sql.types.util.DataTypeConversions$.asJavaDataType(DataTypeConversions.scala:43)

update

Update

update

Update

SparkQA · 2014-12-01T13:30:05Z

Test build #23989 has started for PR 3538 at commit 896c7b7.

This patch merges cleanly.

SparkQA · 2014-12-01T14:33:44Z

Test build #23989 has finished for PR 3538 at commit 896c7b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-01T14:33:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23989/
Test PASSed.

marmbrus · 2014-12-01T21:50:32Z

sql/core/src/main/scala/org/apache/spark/sql/types/util/DataTypeConversions.scala

@@ -62,6 +62,7 @@ protected[sql] object DataTypeConversions {
    case IntegerType => JDataType.IntegerType
    case LongType => JDataType.LongType
    case ShortType => JDataType.ShortType
+    case NullType => JDataType.StringType


Why not also add a null type to JDataType?

marmbrus · 2014-12-01T21:52:56Z

It would also be good to add a test case for this to JavaSQLSuite

… if sql has null

SparkQA · 2014-12-02T04:05:26Z

Test build #24020 has started for PR 3538 at commit 4b4bb34.

This patch merges cleanly.

… if sql has null

SparkQA · 2014-12-02T04:15:07Z

Test build #24021 has started for PR 3538 at commit e052dff.

This patch merges cleanly.

SparkQA · 2014-12-02T05:07:32Z

Test build #24020 has finished for PR 3538 at commit 4b4bb34.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class NullType extends DataType

AmplabJenkins · 2014-12-02T05:07:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24020/
Test PASSed.

SparkQA · 2014-12-02T05:17:12Z

Test build #24021 has finished for PR 3538 at commit e052dff.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class NullType extends DataType

AmplabJenkins · 2014-12-02T05:17:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24021/
Test PASSed.

marmbrus · 2014-12-02T22:02:10Z

sql/core/src/main/java/org/apache/spark/sql/api/java/DataType.java

+	/**
+	 * Gets the NullType object.
+	 */
+	public static final NullType NullType = new NullType();


Indentation is off here.

…if sql has null val jsc = new org.apache.spark.api.java.JavaSparkContext(sc) val jhc = new org.apache.spark.sql.hive.api.java.JavaHiveContext(jsc) val nrdd = jhc.hql("select null from spark_test.for_test") println(nrdd.schema) Then the error is thrown as follows: scala.MatchError: NullType (of class org.apache.spark.sql.catalyst.types.NullType$) at org.apache.spark.sql.types.util.DataTypeConversions$.asJavaDataType(DataTypeConversions.scala:43) Author: YanTangZhai <hakeemzhai@tencent.com> Author: yantangzhai <tyz0303@163.com> Author: Michael Armbrust <michael@databricks.com> Closes #3538 from YanTangZhai/MatchNullType and squashes the following commits: e052dff [yantangzhai] [SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null 4b4bb34 [yantangzhai] [SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null 896c7b7 [yantangzhai] fix NullType MatchError in JavaSchemaRDD when sql has null 6e643f8 [YanTangZhai] Merge pull request #11 from apache/master e249846 [YanTangZhai] Merge pull request #10 from apache/master d26d982 [YanTangZhai] Merge pull request #9 from apache/master 76d4027 [YanTangZhai] Merge pull request #8 from apache/master 03b62b0 [YanTangZhai] Merge pull request #7 from apache/master 8a00106 [YanTangZhai] Merge pull request #6 from apache/master cbcba66 [YanTangZhai] Merge pull request #3 from apache/master cdef539 [YanTangZhai] Merge pull request #1 from apache/master (cherry picked from commit 1066427) Signed-off-by: Michael Armbrust <michael@databricks.com>

marmbrus · 2014-12-02T22:18:20Z

Thanks for fixing this! I've merged to master and 1.2 (I fixed the indentation while merging).

YanTangZhai and others added 9 commits August 6, 2014 21:07

Merge pull request #1 from apache/master

cdef539

update

Merge pull request #3 from apache/master

cbcba66

Update

Merge pull request #6 from apache/master

8a00106

Update

Merge pull request #7 from apache/master

03b62b0

Update

Merge pull request #8 from apache/master

76d4027

update

Merge pull request #9 from apache/master

d26d982

Update

Merge pull request #10 from apache/master

e249846

Update

Merge pull request #11 from apache/master

6e643f8

Update

fix NullType MatchError in JavaSchemaRDD when sql has null

896c7b7

marmbrus reviewed Dec 1, 2014
View reviewed changes

[SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError…

4b4bb34

… if sql has null

[SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError…

e052dff

… if sql has null

marmbrus reviewed Dec 2, 2014
View reviewed changes

asfgit closed this in 1066427 Dec 2, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null #3538

[SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null #3538

YanTangZhai commented Dec 1, 2014

SparkQA commented Dec 1, 2014

SparkQA commented Dec 1, 2014

AmplabJenkins commented Dec 1, 2014

marmbrus Dec 1, 2014

marmbrus commented Dec 1, 2014

SparkQA commented Dec 2, 2014

SparkQA commented Dec 2, 2014

SparkQA commented Dec 2, 2014

AmplabJenkins commented Dec 2, 2014

SparkQA commented Dec 2, 2014

AmplabJenkins commented Dec 2, 2014

marmbrus Dec 2, 2014

marmbrus commented Dec 2, 2014

[SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null #3538

[SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null #3538

Conversation

YanTangZhai commented Dec 1, 2014

SparkQA commented Dec 1, 2014

SparkQA commented Dec 1, 2014

AmplabJenkins commented Dec 1, 2014

marmbrus Dec 1, 2014

Choose a reason for hiding this comment

marmbrus commented Dec 1, 2014

SparkQA commented Dec 2, 2014

SparkQA commented Dec 2, 2014

SparkQA commented Dec 2, 2014

AmplabJenkins commented Dec 2, 2014

SparkQA commented Dec 2, 2014

AmplabJenkins commented Dec 2, 2014

marmbrus Dec 2, 2014

Choose a reason for hiding this comment

marmbrus commented Dec 2, 2014