[SPARK-5068][SQL]fix bug query data when path doesn't exists #3907
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
jeanlyn
changed the title
|
Hi @marmbrus. Any suggestions? |
| import org.apache.hadoop.mapred.JobID | ||
| import org.apache.hadoop.mapred.TaskAttemptID | ||
| import org.apache.hadoop.mapred.TaskID | ||
| import org.apache.hadoop.mapred._ |
There was a problem hiding this comment.
I think I would leave this as individual imports for consistency.
There was a problem hiding this comment.
thx for your suggestions.
jeanlyn
force-pushed
the
origin/SPARK-5068
branch
from
0db35b3 to
d4a5b9a
Compare
jeanlyn
changed the title
jeanlyn
changed the title
|
ok to test |
|
Test build #25181 has started for PR 3907 at commit
|
|
Test build #25181 has finished for PR 3907 at commit
|
|
Test PASSed. |
|
Is this superseded by #4356 ? if so can this be closed? |
|
OK.I close this one |
asfgit
pushed a commit
that referenced
this pull request
Apr 12, 2015
…ontext This PR follow up PR #3907 & #3891 & #4356. According to marmbrus liancheng 's comments, I try to use fs.globStatus to retrieve all FileStatus objects under path(s), and then do the filtering locally. [1]. get pathPattern by path, and put it into pathPatternSet. (hdfs://cluster/user/demo/2016/08/12 -> hdfs://cluster/user/demo/*/*/*) [2]. retrieve all FileStatus objects ,and cache them by undating existPathSet. [3]. do the filtering locally [4]. if we have new pathPattern,do 1,2 step again. (external table maybe have more than one partition pathPattern) chenghao-intel jeanlyn Author: lazymam500 <lazyman500@gmail.com> Author: lazyman <lazyman500@gmail.com> Closes #5059 from lazyman500/SPARK-5068 and squashes the following commits: 5bfcbfd [lazyman] move spark.sql.hive.verifyPartitionPath to SQLConf,fix scala style e1d6386 [lazymam500] fix scala style f23133f [lazymam500] bug fix 47e0023 [lazymam500] fix scala style,add config flag,break the chaining 04c443c [lazyman] SPARK-5068: fix bug when partition path doesn't exists #2 41f60ce [lazymam500] Merge pull request #1 from apache/master
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
the issue is descript on SPARK-5068 and this PR is fix the same problem as PR3891.however,this pull request catch the exception when it is thrown instead of doing it serially when constructing the RDD