Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Round trip FIXED_LEN_BYTE_ARRAY data properly in Parquet writer #15600

Merged
merged 10 commits into from
May 8, 2024

Conversation

etseidl
Copy link
Contributor

@etseidl etseidl commented Apr 25, 2024

Description

#13437 added the ability to consume FIXED_LEN_BYTE_ARRAY encoded data and represent it as lists of UINT8. When trying to write this data back to Parquet there are two problems. 1) the notion of fixed length is lost, and 2) the UINT8 data is written as a list of INT32 which can quadruple the storage required. This PR addresses both issues by adding fields to the input and output metadata to allow for preserving the form of the original data.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@etseidl etseidl requested a review from a team as a code owner April 25, 2024 18:21
Copy link

copy-pr-bot bot commented Apr 25, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Apr 25, 2024
@etseidl
Copy link
Contributor Author

etseidl commented Apr 25, 2024

CC @vuule

@vuule vuule requested review from mhaseeb123 and vuule April 25, 2024 19:02
@vuule vuule added feature request New feature or request cuIO cuIO issue non-breaking Non-breaking change labels Apr 25, 2024
@vuule
Copy link
Contributor

vuule commented Apr 25, 2024

/ok to test

Copy link
Contributor

@vuule vuule left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a few small things. Looks great overall.

cpp/src/io/parquet/writer_impl.cu Outdated Show resolved Hide resolved
cpp/tests/io/parquet_writer_test.cpp Outdated Show resolved Hide resolved
@etseidl etseidl requested a review from vuule May 3, 2024 15:29
@vuule
Copy link
Contributor

vuule commented May 3, 2024

/ok to test

@vuule
Copy link
Contributor

vuule commented May 8, 2024

/ok to test

@vuule
Copy link
Contributor

vuule commented May 8, 2024

/merge

@rapids-bot rapids-bot bot merged commit 5f1f0dd into rapidsai:branch-24.06 May 8, 2024
70 checks passed
@etseidl etseidl deleted the fixed_len_roundtrip branch May 8, 2024 05:32
rapids-bot bot pushed a commit that referenced this pull request May 22, 2024
…PI (#15613)

Several recent PRs (#15081, #15411, #15600) added the ability to control some aspects of Parquet file writing on a per-column basis. During discussion of #15081 it was [suggested](#15081 (comment)) that these options be exposed by cuDF-python in a manner similar to pyarrow. This PR adds the ability to control per-column encoding, compression, binary output, and fixed-length data width, using fully qualified Parquet column names. For example, given a cuDF table with an integer column 'a', and a `list<int32>` column 'b', the fully qualified column names would be 'a' and 'b.list.element'.

Addresses "Add cuDF-python API support for specifying encodings" task in #13501.

Authors:
  - Ed Seidl (https://github.com/etseidl)
  - Vukasin Milovanovic (https://github.com/vuule)
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Muhammad Haseeb (https://github.com/mhaseeb123)
  - GALI PREM SAGAR (https://github.com/galipremsagar)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #15613
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cuIO cuIO issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants