[Python][Parquet] Empty row groups left behind after hitting max_rows_per_file in ds.write_dataset

[ion-elgreco]

Describe the bug, including details regarding any error messages, version, and platform.

The pyarrow.dataset.write_dataset function leaves an empty row_group behind in the the parquet file after the writer hits the limit of max_rows_per_file. See reproducible example below:

import pyarrow.dataset as ds
import pyarrow.parquet as pq
ds.write_dataset(data, "test_dataset", max_rows_per_file=1024*32, max_rows_per_group=1024 * 16, min_rows_per_group=8*1024, format='parquet')
metadata = pq.read_metadata("test_dataset/part-0.parquet")

for i in range(metadata.num_row_groups):
    print(metadata.row_group(i))

<pyarrow._parquet.RowGroupMetaData object at 0x7f92b5168180>
  num_columns: 1
  num_rows: 16384
  total_byte_size: 61
<pyarrow._parquet.RowGroupMetaData object at 0x7f92b5185bc0>
  num_columns: 1
  num_rows: 16384
  total_byte_size: 61
<pyarrow._parquet.RowGroupMetaData object at 0x7f92b5185bc0>
  num_columns: 1
  num_rows: 0
  total_byte_size: 14

Component(s)

Python

[AlenkaF]

Thank you for opening up an issue @ion-elgreco!

I agree this is probably a bug in datasets. It would be in Arrow C++ I believe. @mapleFU would you mind sharing your view on this?

[mapleFU]

@ion-elgreco What's the version of arrow you're now using? I guess that's from bad handling of max_rows_per_group 🤔

[ion-elgreco]

@mapleFU it occurs with the latest one, so v15

We figured this out during this issue, a user mentioned he tried it on v15 up to v9: delta-io/delta-rs#2169 (comment)

[mapleFU]

I think that's a problem in dataset writer, and the problem is not related to core parquet lib, I'll try to reproduce and fix it tomorrow

[mapleFU]

Update: dataset writer DoWriteRecordBatch could cause this problem currently, I can fix it tomorrow, but currently I don't know if other place would causing this 🤔

[mapleFU]

@ion-elgreco Would you mind try: #39995 ? I guess this might fix the problem

[ion-elgreco]

@mapleFU is there some guide I can follow to compile from source? Never touched any C++ codebase before

[mapleFU]

https://arrow.apache.org/docs/developers/python.html
It's a bit complex because it need to build that

[ion-elgreco]

@mapleFU ok I need to find some time to set this up then, I'll try it out in the weekend