Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Forcefully emit batches on input timeout error #916

Merged
merged 11 commits into from
Jun 12, 2020
Merged

Forcefully emit batches on input timeout error #916

merged 11 commits into from
Jun 12, 2020

Conversation

dominiklohmann
Copy link
Member

Note: This does not fix the underlying issue completely, but is a decent workaround that improves the latency between source and importer.

@dominiklohmann dominiklohmann added the bug Incorrect behavior label Jun 11, 2020
@dominiklohmann dominiklohmann marked this pull request as ready for review June 12, 2020 08:17
lava
lava previously requested changes Jun 12, 2020
View reviewed changes
Copy link
Member

@lava lava left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good; the change request is for printing an error on parser failure, the other items are mostly discussion points.

libvast/src/format/csv.cpp Outdated Show resolved Hide resolved
libvast/src/format/pcap.cpp Outdated Show resolved Hide resolved
libvast/src/format/syslog.cpp Outdated Show resolved Hide resolved
tobim
tobim requested changes Jun 12, 2020
View reviewed changes
Copy link
Member

@tobim tobim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the time being this is a reasonable workaround, I hope we'll be able to revert the changes to the source in the future.

libvast/vast/system/source.hpp Outdated Show resolved Hide resolved
libvast/vast/system/source.hpp Outdated Show resolved Hide resolved
libvast/vast/system/source.hpp Outdated Show resolved Hide resolved
libvast/vast/format/json.hpp Outdated Show resolved Hide resolved
tobim
tobim requested changes Jun 12, 2020
View reviewed changes
libvast/vast/format/json.hpp Outdated Show resolved Hide resolved
libvast/vast/format/json.hpp Outdated Show resolved Hide resolved
tobim
tobim approved these changes Jun 12, 2020
View reviewed changes
Copy link
Member

@tobim tobim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we should have an integration test for this, but we need to add parallel execution capabilities to the testing tool first.

@dominiklohmann dominiklohmann merged commit dbf1a79 into master Jun 12, 2020
@dominiklohmann dominiklohmann deleted the topic/force-emit-batches branch June 12, 2020 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mavam mavam mavam left review comments

@tobim tobim tobim approved these changes

@lava lava Awaiting requested review from lava

Assignees
No one assigned
Labels
bug Incorrect behavior
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dominiklohmann @lava @mavam @tobim