Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-5775] [SQL] BugFix: GenericRow cannot be cast to SpecificMutableRow when nested data and partitioned table #4792

Closed
wants to merge 4 commits into from
Closed

[SPARK-5775] [SQL] BugFix: GenericRow cannot be cast to SpecificMutableRow when nested data and partitioned table #4792

wants to merge 4 commits into from
+217 −24

Conversation

liancheng
Copy link
Contributor

This PR adapts @anselmevignon's #4697 to master and branch-1.3. Please refer to PR description of #4697 for details.

Review on Reviewable

@SparkQA
Copy link

SparkQA commented Feb 26, 2015

Test build #28012 has started for PR 4792 at commit ca6e038.

  • This patch merges cleanly.

@liancheng
Copy link
Contributor Author

@marmbrus As discussed offline, I tried to use SpecificMutableRow throughout the whole Parquet support implementation, but it touches to many places, which I think is too risky as 1.3 release is so close. So falled back to current implementation. I can do the refactoring later on master (mainly focus on the CatalystConverter class hierarchy).

@liancheng liancheng mentioned this pull request Feb 26, 2015
Closed
@marmbrus
Copy link
Contributor

Can't we do the same check that we do inside of parquet to see if the row is all primitives once and switch on that, instead of doing per tuple extraction and run time type checking?

@SparkQA
Copy link

SparkQA commented Feb 26, 2015

Test build #28012 has finished for PR 4792 at commit ca6e038.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28012/
Test PASSed.

@yhuai yhuai mentioned this pull request Feb 27, 2015
Closed
@yhuai
Copy link
Contributor

yhuai commented Feb 27, 2015

liancheng#4

@liancheng
Copy link
Contributor Author

@marmbrus @yhuai Thanks for the suggestion and PR, much better!

@liancheng
Merge pull request #4 from yhuai/spark-5775-yin
cee55cf 
Remove runtime pattern matching.
@SparkQA
Copy link

SparkQA commented Feb 27, 2015

Test build #28039 has started for PR 4792 at commit cee55cf.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 27, 2015

Test build #28039 has finished for PR 4792 at commit cee55cf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28039/
Test PASSed.

yhuai
yhuai reviewed Feb 27, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableOperations.scala
@@ -126,6 +126,9 @@ private[sql] case class ParquetTableScan(
conf)

if (requestedPartitionOrdinals.nonEmpty) {
// This check if based on CatalystConverter.createRootConverter.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a typo at here... if => is.

@SparkQA
Copy link

SparkQA commented Feb 28, 2015

Test build #28117 has started for PR 4792 at commit 538f506.

  • This patch merges cleanly.

@liancheng
Copy link
Contributor Author

@yhuai Thanks for the review! I've addressed the comments. Will merge this to master and branch-1.3 after Jenkins approves.

@SparkQA
Copy link

SparkQA commented Feb 28, 2015

Test build #28117 has finished for PR 4792 at commit 538f506.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28117/
Test PASSed.

@asfgit asfgit closed this in e6003f0 Feb 28, 2015
asfgit pushed a commit that referenced this pull request Feb 28, 2015
@liancheng
[SPARK-5775] [SQL] BugFix: GenericRow cannot be cast to SpecificMutab…
aa39460 
…leRow when nested data and partitioned table

This PR adapts anselmevignon's #4697 to master and branch-1.3. Please refer to PR description of #4697 for details.

<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/4792)
<!-- Reviewable:end -->

Author: Cheng Lian <lian@databricks.com>
Author: Cheng Lian <liancheng@users.noreply.github.com>
Author: Yin Huai <yhuai@databricks.com>

Closes #4792 from liancheng/spark-5775 and squashes the following commits:

538f506 [Cheng Lian] Addresses comments
cee55cf [Cheng Lian] Merge pull request #4 from yhuai/spark-5775-yin
b0b74fb [Yin Huai] Remove runtime pattern matching.
ca6e038 [Cheng Lian] Fixes SPARK-5775

(cherry picked from commit e6003f0)
Signed-off-by: Cheng Lian <lian@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@liancheng @SparkQA @marmbrus @AmplabJenkins @yhuai