refactor: Move hive partitioning/multi-file handling outside of readers

[coastalwhite]

This is a very early stage PR exploring whether it is possible to take the multi-file handling and hive partitioning out of the readers. This will also make the multi-file and hive partitioning completely file type agnostic.

Currently, this PR does not do anything unless POLARS_NEW_MULTIFILE is set to 1.

Here is a rough todo list.

• Row Indexes
• Include File Paths
• Projection Pushdown
• Slicing
  • Positive
  • Negative
• Hive Predicates
• Non hive predicates
• Lazy Loading of Non-Hive data
• Allow Missing Columns
• All file types
  • Parquet
  • CSV
  • IPC
  • NDJson
• New streaming engine sink

ping @ritchie46, @nameexhaustion

[codecov]

Codecov Report

Attention: Patch coverage is 9.71660% with 669 lines in your changes missing coverage. Please review.

Project coverage is 78.91%. Comparing base (6e4d717) to head (e01c9d2).
Report is 32 commits behind head on main.

Files with missing lines	Patch %	Lines
...rates/polars-mem-engine/src/executors/hive_scan.rs	0.00%	370 Missing ⚠️
...es/polars-mem-engine/src/executors/scan/parquet.rs	13.25%	72 Missing ⚠️
crates/polars-mem-engine/src/executors/scan/csv.rs	3.44%	56 Missing ⚠️
crates/polars-expr/src/expressions/apply.rs	13.88%	31 Missing ⚠️
crates/polars-expr/src/expressions/binary.rs	16.66%	20 Missing ⚠️
crates/polars-schema/src/schema.rs	0.00%	18 Missing ⚠️
crates/polars-mem-engine/src/planner/lp.rs	60.00%	16 Missing ⚠️
crates/polars-expr/src/expressions/column.rs	20.00%	12 Missing ⚠️
crates/polars-expr/src/expressions/window.rs	0.00%	9 Missing ⚠️
...polars-stream/src/nodes/io_sources/parquet/init.rs	0.00%	8 Missing ⚠️
... and 14 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #20203      +/-   ##
==========================================
- Coverage   79.13%   78.91%   -0.23%     
==========================================
  Files        1572     1567       -5     
  Lines      219839   219920      +81     
  Branches     2462     2465       +3     
==========================================
- Hits       173970   173546     -424     
- Misses      45301    45806     +505     
  Partials      568      568              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.