This repository has been archived by the owner on Sep 2, 2024. It is now read-only.
Revert PARQUET-1414 because it causes empty pages to be written in Spark #37
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@mccheah can you add this to the FORK.md file so that we remember to bring it back? Also seems like there are 2 commits here, was that intentional? |
|
Yeah notice both of them apply to PARQUET-1414. Think the second commit was an improvement made to the first. |
|
Updated the |
|
lgtm |
vinooganesh
approved these changes
Feb 13, 2019
robert3005
suggested changes
Feb 13, 2019
|
@robert3005 if we squash and merge, we can just revert this PR in the future to get these changes back, right? |
|
yeah, mostly concerned about the title to maintain clean history |
|
Tracking upstream Spark at https://issues.apache.org/jira/browse/SPARK-26874 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PARQUET-1414 isn't compatible with the Spark Parquet Data Source, as Spark can inadvertently cause empty pages to be written.
The problem is that in Spark's ParquetWriteSupport, Spark may choose not to write records, particularly if they are empty in optional columns, but will still indicate to the Parquet file writer that records are processed. When Spark indicates to Parquet that records are processed, Parquet might attempt to flush the page to the column chunk, since the number of records counting towards the page row limit introduced in PARQUET-1414 only accounts for the number of signals of processed rows, and not the number of values in the page directly. Suppose the Parquet page row limit is N. Then if Spark receives N null cells for a column, the Parquet writer will think there are N records filling the page (when in fact there are zero), and then write the empty page.
This is most likely to occur in Parquet writes where a column is sparsely populated.
We revert here for now, targeting to patch Spark in the near future, and then re-applying PARQUET-1414 following.