[SPARK-44731][PYTHON][CONNECT] Make TimestampNTZ works with literals in Python Spark Connect #42445
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HyukjinKwon
force-pushed
the
SPARK-44731
branch
from
e4ad9dc
to
eb7598f
Compare
zhengruifeng
approved these changes
Aug 11, 2023
zhengruifeng
pushed a commit
that referenced
this pull request
Aug 11, 2023
…in Python Spark Connect ### What changes were proposed in this pull request? This PR proposes: - Share the namespaces for `to_timestamp_ntz`, `to_timestamp_ltz` and `to_unix_timestamp` in Spark Connect. They were missed. - Adds the support of `TimestampNTZ` for literal handling in Python Spark Connect (by respecting `spark.sql.timestampType`). ### Why are the changes needed? For feature parity, and respect timestamp ntz in resampling in pandas API on Spark ### Does this PR introduce _any_ user-facing change? Yes, this virtually fixes the same bug: #42392 in Spark Connect with Python. ### How was this patch tested? Unittests reenabled. Closes #42445 from HyukjinKwon/SPARK-44731. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org> (cherry picked from commit 73b0376) Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
|
merged to master and branch-3.5 |
hvanhovell
pushed a commit
to hvanhovell/spark
that referenced
this pull request
Aug 13, 2023
…in Python Spark Connect ### What changes were proposed in this pull request? This PR proposes: - Share the namespaces for `to_timestamp_ntz`, `to_timestamp_ltz` and `to_unix_timestamp` in Spark Connect. They were missed. - Adds the support of `TimestampNTZ` for literal handling in Python Spark Connect (by respecting `spark.sql.timestampType`). ### Why are the changes needed? For feature parity, and respect timestamp ntz in resampling in pandas API on Spark ### Does this PR introduce _any_ user-facing change? Yes, this virtually fixes the same bug: apache#42392 in Spark Connect with Python. ### How was this patch tested? Unittests reenabled. Closes apache#42445 from HyukjinKwon/SPARK-44731. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
valentinp17
pushed a commit
to valentinp17/spark
that referenced
this pull request
Aug 24, 2023
…in Python Spark Connect ### What changes were proposed in this pull request? This PR proposes: - Share the namespaces for `to_timestamp_ntz`, `to_timestamp_ltz` and `to_unix_timestamp` in Spark Connect. They were missed. - Adds the support of `TimestampNTZ` for literal handling in Python Spark Connect (by respecting `spark.sql.timestampType`). ### Why are the changes needed? For feature parity, and respect timestamp ntz in resampling in pandas API on Spark ### Does this PR introduce _any_ user-facing change? Yes, this virtually fixes the same bug: apache#42392 in Spark Connect with Python. ### How was this patch tested? Unittests reenabled. Closes apache#42445 from HyukjinKwon/SPARK-44731. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
ragnarok56
pushed a commit
to ragnarok56/spark
that referenced
this pull request
Mar 2, 2024
…in Python Spark Connect ### What changes were proposed in this pull request? This PR proposes: - Share the namespaces for `to_timestamp_ntz`, `to_timestamp_ltz` and `to_unix_timestamp` in Spark Connect. They were missed. - Adds the support of `TimestampNTZ` for literal handling in Python Spark Connect (by respecting `spark.sql.timestampType`). ### Why are the changes needed? For feature parity, and respect timestamp ntz in resampling in pandas API on Spark ### Does this PR introduce _any_ user-facing change? Yes, this virtually fixes the same bug: apache#42392 in Spark Connect with Python. ### How was this patch tested? Unittests reenabled. Closes apache#42445 from HyukjinKwon/SPARK-44731. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR proposes:
to_timestamp_ntz,to_timestamp_ltzandto_unix_timestampin Spark Connect. They were missed.TimestampNTZfor literal handling in Python Spark Connect (by respectingspark.sql.timestampType).Why are the changes needed?
For feature parity, and respect timestamp ntz in resampling in pandas API on Spark
Does this PR introduce any user-facing change?
Yes, this virtually fixes the same bug: #42392 in Spark Connect with Python.
How was this patch tested?
Unittests reenabled.