[SPARK-23158] [SQL] Move HadoopFsRelationTest test suites to from sql/hive to sql/core #20331

gatorsmile · 2018-01-19T17:22:40Z

What changes were proposed in this pull request?

The test suites that extend HadoopFsRelationTest are not in sql/hive packages, but their directories are in sql/hive. We should move them to sql/core.

How was this patch tested?

The existing tests.

gatorsmile · 2018-01-19T17:27:43Z

...src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcHadoopFsRelationSuite.scala

-  }
-}
-
-class HiveOrcHadoopFsRelationSuite extends OrcHadoopFsRelationSuite {


This is Hive only. Thus, create a separate file for it.

gatorsmile · 2018-01-19T17:28:05Z

...src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcHadoopFsRelationSuite.scala

-      spark.range(0, 10).write
-        .orc(file.getCanonicalPath)
-      val expectedCompressionKind =
-        OrcFileOperator.getFileReader(file.getCanonicalPath).get.getCompression


OrcFileOperator is defined in sql\hive.

@gatorsmile . This test case should be tested on native implementation, too.
HiveOrcHadoopFsRelationSuite test coverage is only hive implementation.

gatorsmile · 2018-01-19T17:28:15Z

...src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcHadoopFsRelationSuite.scala

-      assert(maybeOrcFile.isDefined)
-      val orcFilePath = maybeOrcFile.get.toPath.toString
-      val expectedCompressionKind =
-        OrcFileOperator.getFileReader(orcFilePath).get.getCompression


The same here.

gatorsmile · 2018-01-19T17:29:00Z

retest this please

SparkQA · 2018-01-19T19:13:38Z

Test build #86392 has finished for PR 20331 at commit f7693f0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class HadoopFsRelationTest extends QueryTest with SQLTestUtils with SharedSQLContext
class JsonHadoopFsRelationSuite extends HadoopFsRelationTest
class OrcHadoopFsRelationSuite extends HadoopFsRelationTest
class ParquetHadoopFsRelationSuite extends HadoopFsRelationTest
class SimpleTextHadoopFsRelationSuite extends HadoopFsRelationTest with PredicateHelper
class SimpleTextSource extends TextBasedFileFormat with DataSourceRegister
class SimpleTextOutputWriter(path: String, dataSchema: StructType, context: TaskAttemptContext)
class HiveOrcHadoopFsRelationSuite extends OrcHadoopFsRelationSuite

SparkQA · 2018-01-19T19:25:43Z

Test build #86393 has finished for PR 20331 at commit f7693f0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class HadoopFsRelationTest extends QueryTest with SQLTestUtils with SharedSQLContext
class JsonHadoopFsRelationSuite extends HadoopFsRelationTest
class OrcHadoopFsRelationSuite extends HadoopFsRelationTest
class ParquetHadoopFsRelationSuite extends HadoopFsRelationTest
class SimpleTextHadoopFsRelationSuite extends HadoopFsRelationTest with PredicateHelper
class SimpleTextSource extends TextBasedFileFormat with DataSourceRegister
class SimpleTextOutputWriter(path: String, dataSchema: StructType, context: TaskAttemptContext)
class HiveOrcHadoopFsRelationSuite extends OrcHadoopFsRelationSuite

SparkQA · 2018-01-19T23:17:51Z

Test build #86402 has finished for PR 20331 at commit b83f859.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class HadoopFsRelationTest extends QueryTest with SQLTestUtils
class JsonHadoopFsRelationSuite extends HadoopFsRelationTest with SharedSQLContext
abstract class OrcHadoopFsRelationBase extends HadoopFsRelationTest
class ParquetHadoopFsRelationSuite extends HadoopFsRelationTest with SharedSQLContext
class HiveOrcHadoopFsRelationSuite extends OrcHadoopFsRelationBase with TestHiveSingleton

gatorsmile · 2018-01-19T23:33:45Z

retest this please

SparkQA · 2018-01-20T02:06:57Z

Test build #86403 has finished for PR 20331 at commit b83f859.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class HadoopFsRelationTest extends QueryTest with SQLTestUtils
class JsonHadoopFsRelationSuite extends HadoopFsRelationTest with SharedSQLContext
abstract class OrcHadoopFsRelationBase extends HadoopFsRelationTest
class ParquetHadoopFsRelationSuite extends HadoopFsRelationTest with SharedSQLContext
class HiveOrcHadoopFsRelationSuite extends OrcHadoopFsRelationBase with TestHiveSingleton

SparkQA · 2018-01-20T19:27:53Z

Test build #86412 has finished for PR 20331 at commit 9c85b18.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2018-01-20T19:30:10Z

cc @cloud-fan

dongjoon-hyun

Please keep the test coverage for the following two test cases for native ORCFileFormat.

test("SPARK-13543: Support for specifying compression codec for ORC via option()")
test("Default compression codec is snappy for ORC compression")

dongjoon-hyun · 2018-01-22T02:26:36Z

...src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcHadoopFsRelationSuite.scala

-      spark.range(0, 10).write
-        .orc(file.getCanonicalPath)
-      val expectedCompressionKind =
-        OrcFileOperator.getFileReader(file.getCanonicalPath).get.getCompression


@gatorsmile . This test case should be tested on native implementation, too.
HiveOrcHadoopFsRelationSuite test coverage is only hive implementation.

jiangxb1987 · 2018-01-23T19:39:15Z

...c/test/scala/org/apache/spark/sql/execution/datasources/json/JsonHadoopFsRelationSuite.scala

-      val df =
-        spark.read.format(dataSourceName).option("multiLine", true).schema(schema).load(path)
-      checkAnswer(df, Row(null, expected))
+    withSQLConf(SQLConf.MAX_RECORDS_PER_FILE.key -> "2") {


Just curious, why this change?

The test will fail if SQLConf.MAX_RECORDS_PER_FILE.key is less than 2

I think the default value won't be less than 2, we don't need to be so careful...

dongjoon-hyun · 2018-03-11T20:15:09Z

Please remember this comment during next update, @gatorsmile . Thanks.

fix

f7693f0

gatorsmile mentioned this pull request Jan 19, 2018

[SPARK-23094] Fix invalid character handling in JsonDataSource #20302

Closed

gatorsmile commented Jan 19, 2018

View reviewed changes

fix

b83f859

fix test

9c85b18

dongjoon-hyun requested changes Jan 22, 2018

View reviewed changes

jiangxb1987 reviewed Jan 23, 2018

View reviewed changes

gatorsmile closed this Jul 30, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-23158] [SQL] Move HadoopFsRelationTest test suites to from sql/hive to sql/core #20331

[SPARK-23158] [SQL] Move HadoopFsRelationTest test suites to from sql/hive to sql/core #20331

gatorsmile commented Jan 19, 2018

gatorsmile Jan 19, 2018 •

edited

Loading

dongjoon-hyun Jan 22, 2018

gatorsmile Jan 19, 2018

dongjoon-hyun Jan 22, 2018

gatorsmile Jan 19, 2018

gatorsmile commented Jan 19, 2018

SparkQA commented Jan 19, 2018

SparkQA commented Jan 19, 2018

SparkQA commented Jan 19, 2018

gatorsmile commented Jan 19, 2018

SparkQA commented Jan 20, 2018

SparkQA commented Jan 20, 2018

gatorsmile commented Jan 20, 2018

dongjoon-hyun left a comment

dongjoon-hyun Jan 22, 2018

jiangxb1987 Jan 23, 2018

gatorsmile Jan 23, 2018

cloud-fan Mar 8, 2018

dongjoon-hyun commented Mar 11, 2018 •

edited

Loading

[SPARK-23158] [SQL] Move HadoopFsRelationTest test suites to from sql/hive to sql/core #20331

[SPARK-23158] [SQL] Move HadoopFsRelationTest test suites to from sql/hive to sql/core #20331

Conversation

gatorsmile commented Jan 19, 2018

What changes were proposed in this pull request?

How was this patch tested?

gatorsmile Jan 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gatorsmile commented Jan 19, 2018

SparkQA commented Jan 19, 2018

SparkQA commented Jan 19, 2018

SparkQA commented Jan 19, 2018

gatorsmile commented Jan 19, 2018

SparkQA commented Jan 20, 2018

SparkQA commented Jan 20, 2018

gatorsmile commented Jan 20, 2018

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Mar 11, 2018 • edited Loading

gatorsmile Jan 19, 2018 •

edited

Loading

dongjoon-hyun commented Mar 11, 2018 •

edited

Loading