[SPARK-47365][PYTHON] Add toArrow() DataFrame method to PySpark #45481
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HyukjinKwon
reviewed
Mar 16, 2024
ianmcook
changed the title
ianmcook
changed the title
ianmcook
changed the title
ianmcook
commented
Apr 23, 2024
grundprinzip
reviewed
May 7, 2024
grundprinzip
reviewed
May 7, 2024
ianmcook
changed the title
xinrong-meng
reviewed
May 7, 2024
|
Okay I'm fine with this |
|
Thanks. I rebased. I also took a closer look at your changes in #46129 and made a few changes for consistency with the new structure you introduced there:
I also added Please let me know if that all looks OK. |
ianmcook
commented
May 9, 2024
HyukjinKwon
approved these changes
May 9, 2024
HyukjinKwon
reviewed
May 9, 2024
HyukjinKwon
reviewed
May 9, 2024
Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
ianmcook
changed the title
|
Merged to master. |
JacobZheng0927
pushed a commit
to JacobZheng0927/spark
that referenced
this pull request
May 11, 2024
### What changes were proposed in this pull request? - Add a PySpark DataFrame method `toArrow()` which returns the contents of the DataFrame as a [PyArrow Table](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html), for both local Spark and Spark Connect. - Add a new entry to the **Apache Arrow in PySpark** user guide page describing usage of the `toArrow()` method. - Add a new option to the method `_collect_as_arrow()` to provide more useful output when there are zero records returned. (This keeps the implementation of `toArrow()` simpler.) ### Why are the changes needed? In the Apache Arrow community, we hear from a lot of users who want to return the contents of a PySpark DataFrame as a PyArrow Table. Currently the only documented way to do this is to return the contents as a pandas DataFrame, then use PyArrow (`pa`) to convert that to a PyArrow Table. ```py pa.Table.from_pandas(df.toPandas()) ``` But going through pandas adds significant overhead which is easily avoided since internally `toPandas()` already converts the contents of Spark DataFrame to Arrow format as an intermediate step when `spark.sql.execution.arrow.pyspark.enabled` is `true`. Currently it is also possible to use the experimental `_collect_as_arrow()` method to return the contents of a PySpark DataFrame as a list of PyArrow RecordBatches. This PR adds a new non-experimental method `toArrow()` which returns the more user-friendly PyArrow Table object. This PR also adds a new argument `empty_list_if_zero_records` to the experimental method `_collect_as_arrow()` to control what the method returns in the case when the result data has zero rows. If set to `True` (the default), the existing behavior is preserved, and the method returns an empty Python list. If set to `False`, the method returns returns a length-one list containing an empty Arrow RecordBatch which includes the schema. This is used by `toArrow()` which requires the schema even if the data has zero rows. For Spark Connect, there is already a `SparkSession.client.to_table()` method that returns a PyArrow table. This PR uses that to expose `toArrow()` for Spark Connect. ### Does this PR introduce _any_ user-facing change? - It adds a DataFrame method `toArrow()` to the PySpark SQL DataFrame API. - It adds a new argument `empty_list_if_zero_records` to the experimental DataFrame method `_collect_as_arrow()` with a default value which preserves the method's existing behavior. - It exposes `toArrow()` for Spark Connect, via the existing `SparkSession.client.to_table()` method. - It does not introduce any other user-facing changes. ### How was this patch tested? This adds a new test and a new helper function for the test in `pyspark/sql/tests/test_arrow.py`. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#45481 from ianmcook/SPARK-47365. Lead-authored-by: Ian Cook <ianmcook@gmail.com> Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon
pushed a commit
that referenced
this pull request
Jun 2, 2024
### What changes were proposed in this pull request? - Add support for passing a PyArrow Table to `createDataFrame()`. - Document this on the **Apache Arrow in PySpark** user guide page. - Fix an issue with timestamp and struct columns in `toArrow()`. ### Why are the changes needed? This seems like a logical next step after the addition of a `toArrow()` DataFrame method in #45481. ### Does this PR introduce _any_ user-facing change? Users will have the ability to pass PyArrow Tables to `createDataFrame()`. There are no changes to the parameters of `createDataFrame()`. The only difference is that `data` can now be a PyArrow Table. ### How was this patch tested? Many tests were added, for Spark Classic and Spark Connect. I ran the tests locally with older versions of PyArrow installed (going back to 10.0). ### Was this patch authored or co-authored using generative AI tooling? No Closes #46529 from ianmcook/SPARK-48220. Authored-by: Ian Cook <ianmcook@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
riyaverm-db
pushed a commit
to riyaverm-db/spark
that referenced
this pull request
Jun 7, 2024
### What changes were proposed in this pull request? - Add support for passing a PyArrow Table to `createDataFrame()`. - Document this on the **Apache Arrow in PySpark** user guide page. - Fix an issue with timestamp and struct columns in `toArrow()`. ### Why are the changes needed? This seems like a logical next step after the addition of a `toArrow()` DataFrame method in apache#45481. ### Does this PR introduce _any_ user-facing change? Users will have the ability to pass PyArrow Tables to `createDataFrame()`. There are no changes to the parameters of `createDataFrame()`. The only difference is that `data` can now be a PyArrow Table. ### How was this patch tested? Many tests were added, for Spark Classic and Spark Connect. I ran the tests locally with older versions of PyArrow installed (going back to 10.0). ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#46529 from ianmcook/SPARK-48220. Authored-by: Ian Cook <ianmcook@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
toArrow()which returns the contents of the DataFrame as a PyArrow Table, for both local Spark and Spark Connect.toArrow()method._collect_as_arrow()to provide more useful output when there are zero records returned. (This keeps the implementation oftoArrow()simpler.)Why are the changes needed?
In the Apache Arrow community, we hear from a lot of users who want to return the contents of a PySpark DataFrame as a PyArrow Table. Currently the only documented way to do this is to return the contents as a pandas DataFrame, then use PyArrow (
pa) to convert that to a PyArrow Table.But going through pandas adds significant overhead which is easily avoided since internally
toPandas()already converts the contents of Spark DataFrame to Arrow format as an intermediate step whenspark.sql.execution.arrow.pyspark.enabledistrue.Currently it is also possible to use the experimental
_collect_as_arrow()method to return the contents of a PySpark DataFrame as a list of PyArrow RecordBatches. This PR adds a new non-experimental methodtoArrow()which returns the more user-friendly PyArrow Table object.This PR also adds a new argument
empty_list_if_zero_recordsto the experimental method_collect_as_arrow()to control what the method returns in the case when the result data has zero rows. If set toTrue(the default), the existing behavior is preserved, and the method returns an empty Python list. If set toFalse, the method returns returns a length-one list containing an empty Arrow RecordBatch which includes the schema. This is used bytoArrow()which requires the schema even if the data has zero rows.For Spark Connect, there is already a
SparkSession.client.to_table()method that returns a PyArrow table. This PR uses that to exposetoArrow()for Spark Connect.Does this PR introduce any user-facing change?
toArrow()to the PySpark SQL DataFrame API.empty_list_if_zero_recordsto the experimental DataFrame method_collect_as_arrow()with a default value which preserves the method's existing behavior.toArrow()for Spark Connect, via the existingSparkSession.client.to_table()method.How was this patch tested?
This adds a new test and a new helper function for the test in
pyspark/sql/tests/test_arrow.py.Was this patch authored or co-authored using generative AI tooling?
No