feat: support partition prune api

[KnightChess]

Description

Add filtering capabilities to table API, currently only partition fields are applicable. Multiple predicates are AND together.

hudi_table.read_snapshot(&["foo != a", "bar >= 100"]);

Supported operators are: >, >=, <, <=, =, !=.

For #47

How are the changes test-covered

• N/A
• Automated tests (unit and/or integration tests)
• Manual tests
• Details are described below

[KnightChess]

@xushiyan cc

[KnightChess]

@xushiyan hello, Is there a problem with my test method in my local?

[KnightChess]

hello, Is there a problem with my test method in my local?

ignore checkstyle, fetch the latest commit, it work again, but can not report python test error.

[codecov]

Codecov Report

Attention: Patch coverage is 90.07092% with 14 lines in your changes missing coverage. Please review.

Project coverage is 89.32%. Comparing base (e23e6ed) to head (f1ce54d).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/core/src/table/partition.rs	88.76%	10 Missing ⚠️
crates/core/src/table/mod.rs	88.00%	3 Missing ⚠️
crates/core/src/table/fs_view.rs	96.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #119      +/-   ##
==========================================
+ Coverage   87.82%   89.32%   +1.50%     
==========================================
  Files          14       15       +1     
  Lines         731      834     +103     
==========================================
+ Hits          642      745     +103     
  Misses         89       89              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[KnightChess]

@xushiyan cc:

[xushiyan]

@KnightChess awesome contribution! let me take a look. i might push some quick fixes just FYI to move faster.

[xushiyan]

@KnightChess good progress. i think the main functionalities are there. A major change is to re-model the Partition struct, we need to decouple it from datafusion struct

[xushiyan]

@KnightChess do you think you can address the main comment in the next few days? then i can polish further if needed and land this. Trying to get this in the upcoming release within 2 weeks 🙂 (cutting RC branch with a week)

[KnightChess]

@xushiyan sorry for reply late, I will address these two days

[KnightChess]

@xushiyan Hello, I couldn't find an implementation similar to ScalarValue, and I am not very familiar with Arrow yet, and. There is a certain learning curve involved, which might delay the progress of this PR. Could you please help improve this PR?

[KnightChess]

@xushiyan cc, I try to use arrow Scalar<ArrayRef> to replace datafusion ScalarValue, and modified some suggestions to repair.

[xushiyan]

@xushiyan cc, I try to use arrow Scalar<ArrayRef> to replace datafusion ScalarValue, and modified some suggestions to repair.

@KnightChess Thanks. I was traveling. Will take a look later today.

[xushiyan]

I was wrapping up my vacation 😄 just now getting back to update this:

Changes I've made:

• Add PartitionPruner to handle pruning logic, taking care of hudi configs like hive style and url encoded paths.
• Taking filters from parameters in form of foo > 10, bar != a, etc, only apply AND to them.
• Limit the operators to only >, >=, <, <=, =, != to avoid rabbit hole of implementing sql parser; IN and NOT IN can be achieved anyway.
• API to reset should not be needed now, as the pruning with different filters should be applied to loaded fs view directly.

There are more follow up work to do on datafusion integration side, which I'll jot down in the GH issue.

[KnightChess]

@xushiyan thanks review