Skip to content

Commit

Permalink
[SPARK-46016][DOCS][PS] Fix pandas API support list properly
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR proposes to fix a critical issue in the [Supported pandas API documentation](https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/supported_pandas_api.html) where many essential APIs such as `DataFrame.max`, `DataFrame.min`, `DataFrame.mean`, `and DataFrame.median`, etc. were incorrectly marked as not implemented - marked as "N" - as below:

<img width="291" alt="Screenshot 2023-11-24 at 12 37 49 PM" src="https://github.com/apache/spark/assets/44108233/95c5785c-711c-400c-b2ec-0db034e90fd8">

 The root cause of this issue was that the script used to generate the support list excluded functions inherited from parent classes. For instance, `CategoricalIndex.max` is actually supported by inheriting the `Index` class but was not directly implemented in `CategoricalIndex`, leading to it being marked as unsupported:

<img width="397" alt="Screenshot 2023-11-24 at 12 30 08 PM" src="https://github.com/apache/spark/assets/44108233/90e92996-a88a-4a20-bb0c-4909097e2688">

### Why are the changes needed?

The current documentation inaccurately represents the state of supported pandas API, which could significantly hinder user experience and adoption. By correcting these inaccuracies, we ensure that the documentation reflects the true capabilities of Pandas API on Spark, providing users with reliable and accurate information.

### Does this PR introduce _any_ user-facing change?

No. This PR only updates the documentation to accurately reflect the current state of supported pandas API.

### How was this patch tested?

Manually build documentation, and check if the supported pandas API list is correctly generated as below:

<img width="299" alt="Screenshot 2023-11-24 at 12 36 31 PM" src="https://github.com/apache/spark/assets/44108233/a2da0f0b-0973-45cb-b22d-9582bbeb51b5">

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#43996 from itholic/fix_supported_api_gen.

Authored-by: Haejoon Lee <haejoon.lee@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 132bb63)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
itholic authored and HyukjinKwon committed Nov 24, 2023
1 parent a855262 commit 351a5f8
Showing 1 changed file with 2 additions and 14 deletions.
16 changes: 2 additions & 14 deletions python/pyspark/pandas/supported_api_gen.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,23 +138,11 @@ def _create_supported_by_module(
# module not implemented
return {}

pd_funcs = dict(
[
m
for m in getmembers(pd_module, isfunction)
if not m[0].startswith("_") and m[0] in pd_module.__dict__
]
)
pd_funcs = dict([m for m in getmembers(pd_module, isfunction) if not m[0].startswith("_")])
if not pd_funcs:
return {}

ps_funcs = dict(
[
m
for m in getmembers(ps_module, isfunction)
if not m[0].startswith("_") and m[0] in ps_module.__dict__
]
)
ps_funcs = dict([m for m in getmembers(ps_module, isfunction) if not m[0].startswith("_")])

return _organize_by_implementation_status(
module_name, pd_funcs, ps_funcs, pd_module_group, ps_module_group
Expand Down

0 comments on commit 351a5f8

Please sign in to comment.