Incorrect dtype for concat with empty float and datetime columns

[TomAugspurger]

Code Sample, a copy-pastable example if possible

In [40]: a = pd.DataFrame({"A": pd.array(['2000'], dtype='datetime64[ns]')})

In [41]: b = pd.DataFrame({"A": pd.array([1.0], dtype='float64')})

In [42]: pd.concat([a.iloc[:0], b.iloc[:0]])['A'].dtype
Out[42]: dtype('<M8[ns]')

Problem description

The return dtype of concatentating a float64 and datetime64[ns] should be object dtype. That's typically the case when both are non-empty

In [45]: pd.concat([a, b])['A'].dtype
Out[45]: dtype('O')

But when either is empty, we get incorrect results

In [42]: pd.concat([a.iloc[:0], b.iloc[:0]])['A'].dtype
Out[42]: dtype('<M8[ns]')

In [43]: pd.concat([a.iloc[:0], b])['A'].dtype
Out[43]: dtype('float64')

In [44]: pd.concat([a, b.iloc[:0]])['A'].dtype
Out[44]: dtype('<M8[ns]')

All of those should be object dtype.

[jbrockmendel]

@jorisvandenbossche is this fixed by one of your recent or upcoming PRs?

[jorisvandenbossche]

This is not fixed on master.

While certainly a bug, I think it is not directly related to the recent "skipping empties or not" discussion. It seems that this empty float array already gets casted to datetime64 in JoinUnit.get_reindexed_values, so before those values are passed as to_concat arrays to the contact machinery.

I opened a general issue at #39122, where I mentioned this case as example bug for DataFrame.

[mroeschke]

These almost looked fixed. Could add a few tests for the cases that work and xfail the one that doesnt

In [23]: In [40]: a = pd.DataFrame({"A": pd.array(['2000'], dtype='datetime64[ns]')})
    ...:
    ...: In [41]: b = pd.DataFrame({"A": pd.array([1.0], dtype='float64')})

In [24]: In [42]: pd.concat([a.iloc[:0], b.iloc[:0]])['A'].dtype
Out[24]: dtype('O')

In [25]: In [42]: pd.concat([a.iloc[:0], b.iloc[:0]])['A'].dtype
Out[25]: dtype('O')

In [26]: In [43]: pd.concat([a.iloc[:0], b])['A'].dtype
Out[26]: dtype('O')

In [27]: In [44]: pd.concat([a, b.iloc[:0]])['A'].dtype
Out[27]: dtype('<M8[ns]')

[GustavoGB]

Hello guys! Can I take this one ?

[AwaishK]

take

[liang3zy22]

Take. PR already submitted as above.