You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Comet has native code for decoding Parquet structures into Arrow arrays. This issue is for discussing delegating to the parquet crate instead for these operations.
The benefits of this approach include:
Support for complex types. The parquet crate already supports reading maps and structs. We could implement the same support in the Comet native code but it is probably a lot of work
Support for StringView and benefitting from related performance optimizations (see [1] and [2] for details)
Benefit from ongoing optimization work and active community
Reduce maintenance efforts in Comet
Possible downsides of this approach:
Lose the performance benefit of re-using mutable buffers? (although this also comes with a maintenance cost)
What is the problem the feature request solves?
Comet has native code for decoding Parquet structures into Arrow arrays. This issue is for discussing delegating to the parquet crate instead for these operations.
The benefits of this approach include:
Possible downsides of this approach:
[1] https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1/
[2] https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2/
Describe the potential solution
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: