feat: merge operation

[Blajda]

Description

Implement the Merge operation using Datafusion.

Currently the implementation rewrites the entire DeltaTable limiting the files that are rewritten will be performed in future work.

Related Issue(s)

• progresses Implement merge command #850

Documentation

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[ion-elgreco]

@Blajda @wjones127 does that mean the full table will be loaded in memory first so it can rewrite the whole table?

[Blajda]

does that mean the full table will be loaded in memory first so it can rewrite the whole table?

Yes since this is using a nested loop join both the source and target need to be fully loaded into memory. There would need to be follow up improvements to perform pruning and to use a Hash Join / Merge Join.

[ion-elgreco]

does that mean the full table will be loaded in memory first so it can rewrite the whole table?

Yes since this is using a nested loop join both the source and target need to be fully loaded into memory. There would need to be follow up improvements to perform pruning and to use a Hash Join / Merge Join.

Good to know, still useful for datasets that fit in memory then. Hopefully it can be merged soon ;) Then I could try taking a go on the Python bindings and start using it as :)

[wjones127]

What do you think about putting this operation behind a feature flag?

I don't feature flag is necessary, but we could label it "experimental" maybe. I think as long as the docs give up-to-date notes on limitations this should be fine.

[Blajda]

Thanks @wjones127 for taking the time to review this. I'll break future enhancements into smaller chunks to make it easier.

[roeap]

Hoping that merging main will resolve some of the failures, which seem to be related to recently added deletion vector fields...

[Blajda]

Updated code to add new dv field. hopefully it passes all test now.

[roeap]

There is one more failure - probably related to a mypy update o.a. - we could just # type: ignore that line for now.

Other then that we are good to go :)

[Blajda]

😕 Seems to fail with the ignore and without. check-python is passes locally for me

[roeap]

yea, this is a bit annoying :( - unfortunately I do not have permissions to disable or bypass required checks ...

Should we try once more without? Local checks are passing for me as well without the comment.

Alternatively we could try # noqa, which is quite common, but not sure if mypy honors it.

[roeap]

or maybe even both, not sure how make works. i.e. would it fail early if now the ruff check is complaining? otherwise # noqa would silence ruff ...

[Blajda]

@roeap @wjones127 Please provide approval on the workflows. The lint change should hopefully allow everything to pass now.

[ion-elgreco]

@roeap @wjones127 can you approve the workflow, so this can get merged in the next release?

[Blajda]

@wjones127 Please approve the workflow. Merged with the latest changes in main.

[Blajda]

@wjones127 Thanks for reviewing and merging.