Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Converting Spark to Polars and back #17857

Open
ivanchai-1 opened this issue Jul 25, 2024 · 2 comments
Open

Converting Spark to Polars and back #17857

ivanchai-1 opened this issue Jul 25, 2024 · 2 comments
Labels
enhancement New feature or an improvement of an existing feature

Comments

@ivanchai-1
Copy link

Description

We have a problem converting uploads from Spark to Polars.
This conversion is implemented in Pandas, but it takes a very long time to work on huge data. I would like to see this function in Polars for faster task calculations.

pandas_df = spark_df.toPandas()
polars_df = pl.DataFrame(pandas_df)
↧↧↧↧
polars_df = pl.DataFrame(spark_df)
@ivanchai-1 ivanchai-1 added the enhancement New feature or an improvement of an existing feature label Jul 25, 2024
@deanm0000
Copy link
Collaborator

https://stackoverflow.com/questions/73203318/how-to-transform-spark-dataframe-to-polars-dataframe

@deanm0000 deanm0000 reopened this Jul 25, 2024
@ion-elgreco
Copy link
Contributor

@deanm0000 apache/spark#45481, toArrow() will be available in spark 4.0 afaik

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or an improvement of an existing feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ion-elgreco @deanm0000 @ivanchai-1