Enable dataset writer to write hive partitioned parquet datasets

[lmocsi]

Description

Enable dataset writer to write hive partitioned datasets (writing all / many destination files simultaneously).

[deanm0000]

and streaming from lazy...ya know, while you're at it.

[lmocsi]

This should be the other side of pl.scan_pyarrow_dataset (maybe pl.write_pyarrow_dataset).

[deanm0000]

write_parquet already takes arguments to use pyarrow dataset writer. See the example here https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.DataFrame.write_parquet.html.

[ion-elgreco]

This should be the other side of pl.scan_pyarrow_dataset (maybe pl.write_pyarrow_dataset).

Can't you just do df.to_arrow() and then wrap it inside pyarrow.dataset.dataset?

[deanm0000]

This should be the other side of pl.scan_pyarrow_dataset (maybe pl.write_pyarrow_dataset).

Can't you just do df.to_arrow() and then wrap it inside pyarrow.dataset.dataset?

well that's what write_parquet does if you set those parameters. It can't stream though. So if you've got a lazyframe then you can't stream that into a dataset.

[uditrana]

Does someone have an example showing how you would take a Polars dataframe and write out parquets by partition?

I am running into this use case frequently converting big datasets into smaller pieces for deep learning ingestion

[deanm0000]

@uditrana you can use df.write_parquet with use_pyarrow=True and feed pyarrow_options with a dict where one key is partition_cols https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.DataFrame.write_parquet.html#polars.DataFrame.write_parquet

[uditrana]

If I pass in additional kwargs (along with partition_cols) to pyarrow_options, then they should match the signature of pyarrow.write_to_dataset and not pyarrow.write_to_table, correct?

[deanm0000]

yup, it just uses pyarrow.write_to_dataset under the hood.

[ion-elgreco]

May I suggest to use delta-rs instead combined with delta torch

[jmakov]

May I suggest to use delta-rs instead combined with delta torch

Is streaming supported for .scan_delta?

[29antonioac]

Hi, quick question, is the aim of this issue to not rely on pyarrow? I've found using pyarrow to write partitioned datasets can be memory intensive and I thought doing it in Rust side could be beneficial on low-memory systems.

Thanks for the hard work!

[lmocsi]

yup, it just uses pyarrow.write_to_dataset under the hood.

Are you referring to ds.write_dataset()? If yes, then is has some drawbacks, too: apache/arrow#39768

[deanm0000]

Hi, quick question, is the aim of this issue to not rely on pyarrow?

Yes

I've found using pyarrow to write partitioned datasets can be memory intensive and I thought doing it in Rust side could be beneficial on low-memory systems.

Probably not, pyarrow is c and c++ so I wouldn't think it'd be materially different in memory usage than rust. It maybe implemented differently but it's not like pyarrow is implemented in pure Python.

[eromoe]

How to agg.sink_parquet with partition ? My dataset is super large, I want to split it into year/ month partitions, and I found write_parquet method require collect first...

[Smotrov]

It there any option to make sink_json partitioned? Usually have huge datasets.
If it could sink it to a cloud (AWS S3) it would be amazing.

[cmdlineluser]

@Smotrov I don't believe so.

Other formats are on the list in:

Support Hive partitioning logic in other readers besides Parquet

• Hive partitioning tracking issue #15441

Perhaps JSON can be added.

[nameexhaustion]

Support for writing hive partitioned parquet is added by #17324
IPC support is tracked at #17481

[EpicUsaMan]

Support for writing hive partitioned parquet is added by #17324 IPC support is tracked at #17481

But sink_parquet still doesn't support hive partitioning which leads to two bottlenecks:

• compression in one thread
• not utilizing whole SSD while writing using sink_parquet

So could we reopen this or should I create new issue? Because I didn't find such

[lmocsi]

I guess you should create a new one, and reference this one there.

[EpicUsaMan]

I guess you should create a new one, and reference this one there.

Maintainer said that it will be in polars 2.0