Build sst file in more resource friendly way

[jiacai2050]

Describe This Problem

In current implementation, sst write involves two loop:

1. First loop to calculate bloom filters of each row group
2. Second loop to write to parquet

Two loops mean more cpu usage, what's worse it that this may eat too many memory.

Proposal

It's best we can reduce build procedure to one loop, this depends on apache/arrow-rs#3356

If this is not possible, then we may need to spit RecordBatch to disk in order to reduce memory consumption.

Additional Context

No response

[jiacai2050]

Glad parquet has accepted my proposal, we can just wait version release for both data fusion and parquet.

[ShiKaiWi]

Proposal

Current procedure:

1. Fetch all record batches from the input stream to organize them in row groups;
2. Build the metadata based on the row groups;
3. Encode all the row groups and obtain the encoded bytes;
4. Upload the encoded bytes into OSS;

New procedure:

while true {
 1. Fetch enough rows from the input stream to form a row group, break if the input stream is exhausted;
 2. Collect the necesary information from the row group for building final metadata;
 3. Encode the row group and upload the encoded bytes;
}
4. Encode and upload the final metadata to OSS.

[ShiKaiWi]

However, after reviewing the api of parquet writer, async writing has been supported yet, and the relating issue is apache/arrow-rs#1269.

To solve this problem, I guess there are two ways:

• Implement this feature in the upstream which may takes more time to move on.
• Write the local file first and then upload to OSS.

Personally, I vote for the first way.