BUG: Aggregation on arrow array return same type

[liang3zy22]

• closes BUG: Groupby-aggregate on a boolean column returns a different datatype with pyarrow than with numpy #53030 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[rhshadrach]

Shouldn't the bool_arrow result here be double[pyarrow]?

[liang3zy22]

No, it is float64. The result DataFrame's bool_arrow is also float64. So in age_series function, if preserve_type is False, the pyarrow result would be same as numpy result.

[rhshadrach]

I'm asking what it should be. If I add a collection of bool[pyarrow], I expect to get int64[pyarrow]. Likewise, if I divide two int64[pyarrow], I expect to get double[pyarrow].

[liang3zy22]

Oh, yeah. It should be double[pyarrow]. So we need to cast the numpy result to pyarrow types. I am thinking the current cast function "maybe_cast_pointwise_result" need to cast numpy result array to corresponding pyarrow array, not original dtype.

[mroeschke]

I don't think this a great long term solution. I think this needs a more general solution so that this just doesn't apply to agg_series

[liang3zy22]

Yes, a general solution should be in cast function to handle the pyarrow situation. I will update the pr later.

[jbrockmendel]

would #53089 address this? in general we want to get logic out of maybe_cast_pointwise_result

[liang3zy22]

would #53089 address this? in general we want to get logic out of maybe_cast_pointwise_result

@jbrockmendel , I have tried pr53089. It can't fix issue 53030. But I can try to work on pr53089 to fix issue 53030.

[jbrockmendel]

the reason here doesn't seem right to me. are you sure?

[liang3zy22]

I have updated it to make the reason more clear. Please review it.

[jbrockmendel]

what cases get here?

[liang3zy22]

The convert_dtypes(result, dtype_backend="pyarrow") function sometimes will return a np.dtype. So I used current codes to handle this situation. Let me do more investigation to get root cause and better solution.

[jbrockmendel]

i dont understand the reason at all. what breaks if you remove this "skipif" entirely?

[liang3zy22]

In some environment, this test case will fail. The error log is as below:

pandas/tests/groupby/aggregate/test_aggregate.py:1619: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/core/generic.py:6487: in astype
    elif is_extension_array_dtype(dtype) and self.ndim > 1:
pandas/core/dtypes/common.py:1319: in is_extension_array_dtype
    return registry.find(dtype) is not None
pandas/core/dtypes/base.py:522: in find
    return dtype_type.construct_from_string(dtype)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'pandas.core.dtypes.dtypes.ArrowDtype'>, string = 'bool[pyarrow]'

    @classmethod
    def construct_from_string(cls, string: str) -> ArrowDtype:
        """
        Construct this type from a string.
    
        Parameters
        ----------
        string : str
            string should follow the format f"{pyarrow_type}[pyarrow]"
            e.g. int64[pyarrow]
        """
        if not isinstance(string, str):
            raise TypeError(
                f"'construct_from_string' expects a string, got {type(string)}"
            )
        if not string.endswith("[pyarrow]"):
            raise TypeError(f"'{string}' must end with '[pyarrow]'")
        if string == "string[pyarrow]":
            # Ensure Registry.find skips ArrowDtype to use StringDtype instead
            raise TypeError("string[pyarrow] should be constructed by StringDtype")
    
        base_type = string[:-9]  # get rid of "[pyarrow]"
        try:
>           pa_dtype = pa.type_for_alias(base_type)
E           NameError: name 'pa' is not defined

So I added skipif to check if TYPE_CHECKING is true then pyarrow will be imported in core/dtypes/dtypes.py.

[jbrockmendel]

the decorator you want for this test is @skip_if_no_pyarrow

[liang3zy22]

@jbrockmendel , I have done some debug, found something about convert_dtypes function.
The convert_dtypes function's first parameter input_array could be np.ndarray with object dtype. Then the line:
inferred_dtype = lib.infer_dtype(input_array)
will be executed. Since lib.infer_dtype returns a str, then inferred_dtype also is str. So following lines will be executed:

if isinstance(inferred_dtype, str):
    # If we couldn't do anything else, then we retain the dtype
    inferred_dtype = input_array.dtype

At this time, inferred_dtype is object dtype.
In the 'pyarrow' backend codes part, all if condition failed. So this line is executed:
base_dtype = inferred_dtype
base_dtype is object dtype also. Then after pa_type = to_pyarrow_type(base_dtype), pa_type is None since base_dtype is object.
So in this situation, convert_dtypes will return object dtype as long as input_array is np.ndarray with object dtype.

[jbrockmendel]

what if you do infer_objects() followed by convert_dtypes?

[liang3zy22]

I have tried using infer_objects with following codes:

            if result.dtype == object:
                df = pandas.DataFrame(result)
                result = df.infer_objects().values
            pyarrow_type = convert_dtypes(result, dtype_backend="pyarrow")

It doesn't work. The dtype of result still can be object after infer_objects.

[mroeschke]

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.