Add a Rust-backed engine for write_deltalake

[wjones127]

Description

Right now we've built on top of the PyArrow writers. This requires a lot of complex code that is essentially duplicating logic in Rust. The main motivation for writing it was that the Rust implementation wasn't ready, so it was faster to build on top of PyArrow. That might not be true anymore.

We can update the signature of write_deltalake() to take an engine parameter (sort of like how Pandas read_parquet has this parameter), which would let users choose to use the pyarrow engine or the Rust engine for now. Eventually we can switch the default and deprecate the pyarrow implementation.

We should be on the lookout for issues that block this. First, we need to make sure the same unit tests pass with the new writer. So we should parametrize all tests by the engine.

Second, we should be on the lookout for performance issues. We have a set of benchmarks here:

delta-rs/python/tests/test_benchmark.py

Line 28 in 56a9728

def test_benchmark_write(benchmark, sample_table, tmp_path):

, which we can add to. One known issue (that might have been solved) is: #1225

Use Case

Related Issue(s)

[roeap]

we should use this opportunity to also consolidate our writer implementations. right now we have one in /writer as well as /operations/writer. The one in operations being the newer one. Originally it was meant to have two implementations only for a short while, but this has been the case for many months now 😆.

The /writer one was mainly kept around to keep supporting the kafka-delta-ingest use case to also be able to writer json data.

[ion-elgreco]

@roeap I am mainly exposing operations/write to Python, not writer

[roeap]

I'll re-open this to keep track until we have a more full-featured rust writer?

[ion-elgreco]

@roeap yeah good one!