fix: Properly project unordered column in parquet prefiltered

[coastalwhite]

Related to #20175.

[nameexhaustion]

😄

[coastalwhite]

@nameexhaustion any chance you can take this PR from here? You are way more knowledgeable about the hive partitioning code than me

[nameexhaustion]

Will check

[nameexhaustion]

revert sorting projections in the projection pushdown optimizer from 29373d1

[coastalwhite]

Maybe we should just project unsorted columns in the optimizer. That makes IO code a lot simpler.

[nameexhaustion]

I have changed this for Parquet. I've left the other scans as they may rely on sorted projections.

[codecov]

Codecov Report

Attention: Patch coverage is 34.95146% with 67 lines in your changes missing coverage. Please review.

Project coverage is 79.61%. Comparing base (36b1244) to head (43a6457).
Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
...m/src/nodes/io_sources/parquet/row_group_decode.rs	0.00%	63 Missing ⚠️
crates/polars-schema/src/schema.rs	71.42%	2 Missing ⚠️
...ates/polars-stream/src/physical_plan/lower_expr.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #20189      +/-   ##
==========================================
- Coverage   79.63%   79.61%   -0.02%     
==========================================
  Files        1564     1564              
  Lines      217472   217868     +396     
  Branches     2474     2477       +3     
==========================================
+ Hits       173188   173463     +275     
- Misses      43715    43836     +121     
  Partials      569      569              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nameexhaustion]

In summary it was extremely tricky because we have an optimization where we skip loading hive columns from the actual file, but we still load them in the correct positions as if they were loaded from the file.

[nameexhaustion]

For a full-projection scan().collect() this materializes hive columns that exist in the file in the position based on the file schema.But in the case where projections were pushed down the resulting columns are actually not properly ordered because the columns in df no longer match schema - this is still fine because we add a Select {} node on top of the scan in projection pushdown to get the correct order.

[coastalwhite]

I think just in general. No IO source should have to reorder its columns in a projection. It might be better to provide the IO sources with a schema and a bitmap on which columns to load. The complexity of reordering the columns can then be handled by almost free SELECT immediately afterward.

This will only get more complicated as we expand the hive partitioning support. We should try to divide those problems as much as possible. The efficiency hit would be absolute minimal. Ideally, I would like to remove the materialize_hive_partitions from the polars-io/parquet directory entirely.

Doesn't have to be this PR though.