Skip to content

Commit

Permalink
[SPARK-6037][SQL] Avoiding duplicate Parquet schema merging
Browse files Browse the repository at this point in the history
`FilteringParquetRowInputFormat` manually merges Parquet schemas before computing splits. However, it is duplicate because the schemas are already merged in `ParquetRelation2`. We don't need to re-merge them at `InputFormat`.

Author: Liang-Chi Hsieh <viirya@gmail.com>

Closes #4786 from viirya/dup_parquet_schemas_merge and squashes the following commits:

ef78a5a [Liang-Chi Hsieh] Avoiding duplicate Parquet schema merging.

(cherry picked from commit 4ad5153)
Signed-off-by: Cheng Lian <lian@databricks.com>
  • Loading branch information
viirya authored and liancheng committed Feb 27, 2015
1 parent b83a93e commit 25a109e
Showing 1 changed file with 7 additions and 16 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -434,22 +434,13 @@ private[parquet] class FilteringParquetRowInputFormat
return splits
}

Option(globalMetaData.getKeyValueMetaData.get(RowReadSupport.SPARK_METADATA_KEY)).foreach {
schemas =>
val mergedSchema = schemas
.map(DataType.fromJson(_).asInstanceOf[StructType])
.reduce(_ merge _)
.json

val mergedMetadata = globalMetaData
.getKeyValueMetaData
.updated(RowReadSupport.SPARK_METADATA_KEY, setAsJavaSet(Set(mergedSchema)))

globalMetaData = new GlobalMetaData(
globalMetaData.getSchema,
mergedMetadata,
globalMetaData.getCreatedBy)
}
val metadata = configuration.get(RowWriteSupport.SPARK_ROW_SCHEMA)
val mergedMetadata = globalMetaData
.getKeyValueMetaData
.updated(RowReadSupport.SPARK_METADATA_KEY, setAsJavaSet(Set(metadata)))

globalMetaData = new GlobalMetaData(globalMetaData.getSchema,
mergedMetadata, globalMetaData.getCreatedBy)

val readContext = getReadSupport(configuration).init(
new InitContext(configuration,
Expand Down

0 comments on commit 25a109e

Please sign in to comment.