ENH: Handle conversion from Polars with pd.DataFrame

[braaannigan]

Is your feature request related to a problem?

I have a Polars DataFrame polars_df and want to convert it to a Pandas DataFrame. However, when I call pd.DataFrame(polars_df) it transposes the polars dataframe.

Describe the solution you'd like

I would like pd.DataFrame to output a Pandas DataFrame that hasn't been transposed.

API breaking implications

In the __init__ method of the DataFrame class (line 604) it would check if the type string was polars.internal.dataframe and if so it would call the to_pandas method of the object.

Describe alternatives you've considered

In my own code I can just use the to_pandas method of a polars dataframe, but 3rd-party libraries do not do this.

Additional context

import pandas as pd
import polars as pl

df = pl.DataFrame({'a':[0,1],'b':[1,2]})
pd.DataFrame(df)

gives output like:

	0	1
0	0	1
1	3	4

so the rows have been transposed and the column names have been dropped

[jreback]

we cannot depend on polars

instead polars can implement thr dataframe protocol

pls open an issue on that repo

closing as this is out of scope

[braaannigan]

Thanks for your comments @jreback

Looking through the PR on the dataframe protocol it seems like the conversion has to be called explicitly. Could there be a check within pd.DataFrame to see if the object isn't one of the usual suspects but has the dataframe namespace then it uses that namespace to construct the pandas dataframe?

[jreback]

there will be further work on this but polars would need to support this in the first place

[braaannigan]

Cheers, thanks for replying

[abubelinha]

we cannot depend on polars
instead polars can implement thr dataframe protocol
pls open an issue on that repo

@braaannigan did you finally open that issue? If so, please share the link.
I also think it's important to facilitate compatibility between both systems

Thanks
@abubelinha

[MarcoGorelli]

polars has already implemented it pola-rs/polars#6581

[abubelinha]

Thanks @MarcoGorelli
So @jreback this issue can/should be reopen now?

[MarcoGorelli]

🤔 why should it be reopened? you can do

In [8]: df = pl.DataFrame({'a': [1,2,3], 'b': [4,5,6]})

In [9]: pd.api.interchange.from_dataframe(df)
Out[9]:
   a  b
0  1  4
1  2  5
2  3  6

[abubelinha]

Sorry, I may have misunderstood.
I thought the issue was letting Pandas import a Polars dataframe (keeping column names and without transposing) with a simple syntax like this:

pldf # a Polars dataframe
pddf = pd.DataFrame(pldf) # a Pandas dataframe

Anyway, the issue remained unanswered until now.
So, this is the final answer?

pd.core.interchange.from_dataframe.from_dataframe(pldf)

[MarcoGorelli]

pd.DataFrame(pldf) is unsupported, the answer to that was

we cannot depend on polars
instead polars can implement thr dataframe protocol

, which polars has done 🎉

Or you can just use to_pandas pola-rs/polars#6756

[abubelinha]

Thanks for clarifying 👍

[datapythonista]

we cannot depend on polars

While I agree that the dataframe interchange protocol is a better approach, it's worth noting that we can actually depend on polars, since we're already doing it on xarray to provide a DataFrame.to_xarray. So, I don't think a DataFrame.to_polars should be out of question.

I'm surely more in favor of removing to_xarray than to have a soft dependency on polars or any other library that we can export to. And if having these methods is useful (and I think it is), I think having third party packages implementing Python entrypoints we can create our methods automatically from seems the most reasonable approach to me.

[MarcoGorelli]

Sure, just noting though that they already have from_pandas:

In [1]: pl.from_pandas(pd.DataFrame({'a': [1,2,3]}))
Out[1]:
shape: (3, 1)
┌─────┐
│ a   │
│ --- │
│ i64 │
╞═════╡
│ 1   │
│ 2   │
│ 3   │
└─────┘

which I think pd.DataFrame.to_polars would be duplicative of?

[jreback]

we don't rely on xarray explicitly - for a type check
rather it's a runtime conversion TO xarray

-1 on adding any additional even implicit dependencies

-0.25 on removing xarray (this is an orthogonal library so it's pretty useful)