ENH: EA._from_scalars

[jbrockmendel]

• closes API: EA interface - strictness of _from_sequence #33254
• closes API: inconsistency in casting back to its EA dtype (try_cast_to_ea) #31108 (someone double-check me on this)

xref #33254 _from_sequence is generally not very strict. This implements _from_scalars for a few EAs as a POC to get rid of an ugly kluge in maybe_cast_pointwise_result. The changed behavior in test_resample_categorical_data_with_timedeltaindex looks like a clear improvement to me.

cc @jorisvandenbossche

Things to work out before this leaves the "WIP" zone:

• implement/document it on the base class, none of this "hasattr" shenanigans
• decide whether dtype should be required. Not specifying dtype has different implications for different EA subclasses, which is not ideal. I think part of the motivation is to be able to preserve pyarrow/masked "flavors" when doing pointwise ops.
• see if this fixes BUG: groupby.agg with udf incorrectly inferring pyarrow timestamp dtype #49163, BUG: UDFs with apply returning nanoseconds when compared to native binops that return seconds #52411 update Nope!

[jorisvandenbossche]

decide whether dtype should be required.

The reason to not do this would be that when starting with an int64[pyarrow], you could end up with float64[pyarrow]?
However, given that this is essentially a full type system (all possible types are available in ArrowDtype), that would essentially make this non-strict for those?

(so I would maybe start with requiring a dtype)

There are also some corner cases listed in my original PR: #38315 (comment) (need to read through that again myself as well)

[jorisvandenbossche]

You can probably add a simple _from_scalars in the base class that just calls _from_sequence. We will need some fallback like that for external EAs anyway (unless we keep this hasattr check, but since we fallback to _from_sequence below anyway, can better do this in _from_scalars I think)

[jbrockmendel]

Agreed this makes sense.

ATM _from_sequence can raise anything, so we catch anything, which isn't an ideal pattern. Thoughts on documenting _from_scalars as only raising ValueError/TypeError and eventually enforcing that?

[jorisvandenbossche]

Thoughts on documenting _from_scalars as only raising ValueError/TypeError and eventually enforcing that?

Yes, I think that's certainly fine

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[liang3zy22]

@jbrockmendel , I have done some investigation based on your PR. I have found one issue:
For pyarrow type, preserve_dtype would be True in agg_series. Then maybe_cast_pointwise_result should be called.
The maybe_cast_pointwise_result function have one default parameter, same_type to be True. So the result would be cast to the original pyarrow types. It caused issue 53030. I have changed the same_type parameter to be False by default. The issue 53030 is fixed, but some pytest test cases failed.
So in what situation should cast the result to original pyarrow types? And in what situation should cast the result to the corresponding pyarrow types?

[jbrockmendel]

@jorisvandenbossche i think the last round of comments and OP issues have been addressed. any other thoughts?

[mroeschke]

Thinking aloud. Once we allow EA authors to specify their "is_scalar" passing equivalent, should that be incorporated here somehow?

[jbrockmendel]

Yes. Assuming at some point we add to the EADtype something like a _is_recognized_scalar method, the default implementation of that method might look like (assuming away NAs for the moment)

def _is_recognized_scalar(self, scalar) -> bool:
    cls = self.construct_array_class()
    try:
        cls._from_scalars([scalar], dtype=self)
    except TypeError:
        return False
    return True

We could alternately implement a default from_scalars in terms of _is_recognized_scalar. _find_compatible_dtype from #53106 could also be the primitive from which the other two are constructed (though i dont think it can be constructed from either).

[jbrockmendel]

If there are no further comments, I'm planning to merge this at the end of the week.