[SPARK-22384][SQL][followup] Refine partition pruning when attribute is wrapped in Cast #21712
Conversation
| "aa" :: "ab" :: "ba" :: "bb" :: Nil) | ||
| } | ||
|
|
||
| test("getPartitionsByFilter: cast(chunk as boolean)=1 (not a valid partition predicate)") { |
There was a problem hiding this comment.
nit: cast(chunk as boolean)=true
| * Returns true iff we can safely cast the `from` type to `to` type without any truncating or | ||
| * precision lose, e.g. int -> long, date -> timestamp. | ||
| */ | ||
| def canSafeCast(from: AtomicType, to: AtomicType): Boolean = (from, to) match { |
There was a problem hiding this comment.
Is this only applied for AtomicType?
There was a problem hiding this comment.
yes, at least for now. Hive partition predicate doesn't support complex type.
| @@ -122,6 +122,22 @@ class HiveClientSuite(version: String) | |||
| "aa" :: Nil) | |||
| } | |||
|
|
|||
| test("getPartitionsByFilter: cast(chunk as int)=1 (not a valid partition predicate)") { | |||
There was a problem hiding this comment.
I ran this test with current master branch, seems it can also pass without this change?
There was a problem hiding this comment.
yes this can pass, we can tell it by #12586
Cast.mayTruncate returns true for string to int, but not string to boolean. Here I just wanna test both of the cases, to improve test coverage.
|
Test build #92602 has finished for PR 21712 at commit
|
|
Test build #92608 has finished for PR 21712 at commit
|
|
Thanks! Merged to master. |
|
Thanks for ping me. LGTM |
What changes were proposed in this pull request?
As mentioned in #21586 ,
Cast.mayTruncateis not 100% safe, string to boolean is allowed. Since changingCast.mayTruncatealso changes the behavior of Dataset, here I propose to add a newCast.canSafeCastfor partition pruning.How was this patch tested?
new test cases