Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support ndjson with a codec #2700

Closed
dlvenable opened this issue May 16, 2023 · 1 comment · Fixed by #4533
Closed

Support ndjson with a codec #2700

dlvenable opened this issue May 16, 2023 · 1 comment · Fixed by #4533
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@dlvenable
Copy link
Member

dlvenable commented May 16, 2023

Is your feature request related to a problem? Please describe.

The current json codec supports parsing JSON arrays. However, many users would like to parse ndjson formatted files.

{"key": "value1"}
{"key": "value2"}
{"key": "value3"}
{"key": "value4"}

Describe the solution you'd like

Create a new codec: ndjson.

Describe alternatives you've considered (Optional)

We can already ready nd-json files using a combination of:

  • Parsing with the newline codec.
  • Using the parse_json processor.
@dlvenable
Copy link
Member Author

Additionally, we have seen a number of situations where JSON objects are not always properly separated by a newline.

I believe we can implement this codec to have a loose enforcement of different objects by default. Thus allowing for parsing the following:

{"key": "value1"}
{"key": "value2"}{"key": "value3"}
{"key": "value4"} {"key": "value5"}
{"key": "value6"}

{"key": "value7"}

This would produce seven events.

This change would also provide something that isn't currently possible with the newline codec and parse_json combination.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

Successfully merging a pull request may close this issue.

1 participant