Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Add support for escape characters in CSV #11984

Open
revans2 opened this issue Oct 25, 2022 · 0 comments
Open

[FEA] Add support for escape characters in CSV #11984

revans2 opened this issue Oct 25, 2022 · 0 comments
Labels
0 - Backlog In queue waiting for assignment cuIO cuIO issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. Spark Functionality that helps Spark RAPIDS

Comments

@revans2
Copy link
Contributor

revans2 commented Oct 25, 2022

Is your feature request related to a problem? Please describe.
By default Spark uses an escape character \ to escape things like double quotes within other double quotes i.e. "\"". Pandas by default uses two double quotes right next to each other. Spark does not support this method of escaping.

Describe the solution you'd like
We would like an option in the csv reader setting to allow setting an escape character. There should be an option to either disable the double double quote escaping, or have it automatically disabled if an escape character is provided.

Describe alternatives you've considered
write our own CSV parser.

@revans2 revans2 added feature request New feature or request Needs Triage Need team to review and classify Spark Functionality that helps Spark RAPIDS labels Oct 25, 2022
@GregoryKimball GregoryKimball added 0 - Backlog In queue waiting for assignment cuIO cuIO issue and removed Needs Triage Need team to review and classify labels Oct 30, 2022
@GregoryKimball GregoryKimball added the libcudf Affects libcudf (C++/CUDA) code. label Apr 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0 - Backlog In queue waiting for assignment cuIO cuIO issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. Spark Functionality that helps Spark RAPIDS
Projects
None yet
Development

No branches or pull requests

2 participants