Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When encountering a malformed line in an input file, log it and continue. #86

Open
docktermj opened this issue Oct 4, 2021 · 2 comments
Open

When encountering a malformed line in an input file, log it and continue. #86

docktermj opened this issue Oct 4, 2021 · 2 comments

Comments

@docktermj
Copy link
Contributor

Is your feature request related to a problem? Please describe.
We've notice input files with lines that are malformed (i.e. CSV delimiters are corrupted by delimiters in the data itself)

Describe the solution you'd like
When the stream-producer encounters such an error:

  1. Log the error
  2. Continue reading the next records.
@github-actions github-actions bot added the triage Need to triage label Oct 4, 2021
@docktermj docktermj changed the title When encountering a mal-formed line in an input file, log it and continue. When encountering a malformed line in an input file, log it and continue. Oct 4, 2021
@roncewind
Copy link
Contributor

How about added a command line option to log failed lines?

@jamietypovsky jamietypovsky removed the triage Need to triage label Oct 18, 2021
@jamietypovsky
Copy link

jamietypovsky commented Oct 18, 2021 •
edited
Loading

It is ok to log all of the information to an error log file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@docktermj @roncewind @jamietypovsky