Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds an ndjson input codec for reading ND-JSON and lenient formats #4533

Merged
merged 1 commit into from
May 14, 2024

Conversation

dlvenable
Copy link
Member

@dlvenable dlvenable commented May 13, 2024

Description

Adds the ndjson codec. This also supports lenient formats where there are not newlines.

Issues Resolved

Resolves #2700

Testing

Tested with:

{"name" :  "one", "value":  1}
{"name" :  "two", "value":  2}{"name" :  "three", "value":  3}
{"name" :  "four", "value":  4}

{}


{"name" :  "five", "value":  5}

{}{"name" :  "six", "value":  6}
{"name" :  "seven", "value":  7}

With stdout sink:

data-prepper  | {"name":"one","value":1}
data-prepper  | {"name":"two","value":2}
data-prepper  | {"name":"three","value":3}
data-prepper  | {"name":"seven","value":7}
data-prepper  | {"name":"four","value":4}
data-prepper  | {"name":"five","value":5}
data-prepper  | {"name":"six","value":6}

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ore lenient formats that do not have the newline.

Signed-off-by: David Venable <dlv@amazon.com>
@dlvenable dlvenable merged commit 59e9fd6 into opensearch-project:main May 14, 2024
40 of 47 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request May 14, 2024
…ore lenient formats that do not have the newline. (#4533)

Signed-off-by: David Venable <dlv@amazon.com>
(cherry picked from commit 59e9fd6)
dlvenable added a commit that referenced this pull request May 14, 2024
…ore lenient formats that do not have the newline. (#4533) (#4536)

Signed-off-by: David Venable <dlv@amazon.com>
(cherry picked from commit 59e9fd6)

Co-authored-by: David Venable <dlv@amazon.com>
@dlvenable dlvenable deleted the ndjson-codec branch June 12, 2024 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support ndjson with a codec
3 participants