[GLUTEN-8499][VL] Fix the logic of cast expression #8500
Conversation
|
Run Gluten ClickHouse CI on ARM |
|
Run Gluten ClickHouse CI on ARM |
|
Velox doesn't support ANSI mode yet, we should fallback to Vanilla spark once user set ANSI mode. It will be big issue on Spark4.0 |
|
Run Gluten ClickHouse CI on ARM |
Yes, there are still a lot of works to support ansi mode, until then we have to fallback it to avoid data consistency issues. |
Should we start the work of ANSI mode support? |
It depends on customer requirement. Not sure its importance yet. Do you have request to support this year? |
What changes were proposed in this pull request?
The cast expression in ansi mode not only returns null on error, but also has different calculation logic from non-ansi mode, so we should fall back the cast expression in ansi mode.
Like
cast string to long, they define differentallowDecimal:https://github.com/apache/spark/blob/3c831ebf7cd81c2ed774c692b2da7a35ce0fa252/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java#L1612
https://github.com/apache/spark/blob/3c831ebf7cd81c2ed774c692b2da7a35ce0fa252/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java#L1813
Closes #8499
How was this patch tested?
unit test