-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feat: adjust fields for ECS compatibility #28
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Docs review: I left some comments inline for your consideration.
I see some other changes I'd like to make that pre-date your recent changes. Any chance I could put them in a PR, and include them in your version bump?
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left a suggestion inline.
In addition, please consider these formatting fixes to sections not included in this PR:
Lines 26-29:
[NOTE]
========
* The `command` field of this event will be the command run.
* The `message` field of this event will be the entire stdout of the command.
========
Lines 31-35:
IMPORTANT: The exec input ultimately uses `fork` to spawn a child process.
Using fork duplicates the parent process address space (in our case, **logstash and the JVM**); this is mitigated with OS copy-on-write but ultimately you can end up allocating lots of memory just for a "simple" executable.
If the exec input fails with errors like `ENOMEM: Cannot allocate memory` it is an indication that there is not enough non-JVM-heap physical memory to perform the fork.
If you'd rather me put these changes in a separate PR, I am happy to do so.
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, this looks on-track. I'm concerned by the specs being configured to run differently in CI (does the behaviour differ on docker in general, or on our CI docker? why?)
I've also left a nit about using floats for our intermediate storage mechanism for duration, if we are indeed attempting to track actual nanos.
lib/logstash/inputs/exec.rb
Outdated
@@ -124,5 +134,9 @@ def wait_until_end_of_interval(duration) | |||
end | |||
end | |||
|
|||
# convert seconds (float) to nanoseconds |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: can we avoid transitioning through a lossy format, and hold nanos as our intermediate?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure we can, but our over-head won't make it super precise.
forgot JRuby is no longer using currentTimeMillis
for Time.now
but actually does a native call directly and has nanos -> thus it's a good and valid point - updated, thanks for the advice.
spec/inputs/exec_spec.rb
Outdated
|
||
expect(queue.map(&:to_hash)).to be_empty | ||
end | ||
end if ENV['CI'] != 'true' # in Docker the behavior differs - missing command files are not raised |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oof. I don't like skipping specs in CI. Can you explain why this is needed? what doesn't raise?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry I forgot about this one - should have mentioned the behavior difference in Docker vs non-Docker.
normally one gets Errno::ENOENT: No such file or directory - invalid_command
for a missing command,
however the behavior is different in Docker - popen
does not raise - we simply get a non-zero exit code:
"message"=>"", "process"=>{"command_line"=>"invalid_command 1 2 3", "exit_code"=>127}
haven't looked into this one as it's existing behavior - thought we rather have this as a known behavior spec-ed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there are two different behaviours, depending on which platform it is run, I would prefer we clearly specify both behaviours instead of specifying one of the behaviours and conditionally executing that spec.
If possible, it could be helpful to emulate the underlying change in behaviour and define our specs so that we can validate our effective behavour in CI regardless of which platform it is executed on (It would be hard to stub IO::popen
because we rely on its side-effect of setting $?
, but can we stub run_command
?).
So our specs output would look something like:
when command fails
on a platform where IO.popen raises ENOENT
it does not enqueue an event
on a platform where IO.popen does not raise ENOENT (Docker)
it enqueues an event tagged with the error code
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 if we're going this far as to specify the behavior that I would rather move this out of this PR.
Have opened an issue #29 + I will create a draft with the suggested spec.
The problematic (newly added) spec has been removed.
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
@karenzone I've updated the docs based on your last suggestion sorry it took so long: 8536b8a |
spec/inputs/exec_spec.rb
Outdated
|
||
expect(queue.map(&:to_hash)).to be_empty | ||
end | ||
end if ENV['CI'] != 'true' # in Docker the behavior differs - missing command files are not raised |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there are two different behaviours, depending on which platform it is run, I would prefer we clearly specify both behaviours instead of specifying one of the behaviours and conditionally executing that spec.
If possible, it could be helpful to emulate the underlying change in behaviour and define our specs so that we can validate our effective behavour in CI regardless of which platform it is executed on (It would be hard to stub IO::popen
because we rely on its side-effect of setting $?
, but can we stub run_command
?).
So our specs output would look something like:
when command fails
on a platform where IO.popen raises ENOENT
it does not enqueue an event
on a platform where IO.popen does not raise ENOENT (Docker)
it enqueues an event tagged with the error code
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
===== IMPORTANT | ||
|
||
The exec input ultimately uses `fork` to spawn a child process. | ||
IMPORTANT: The exec input ultimately uses `fork` to spawn a child process. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for getting this extra changes in. They render nicely and look good. Post-merge LGTM
Previously plugin was setting the top level
command
andhost
fields for every event (which aren't ECS compliant).Also changes behavior not to override fields if they exists in the decoded payload (e.g. no longer force the
host
field if such a field is decoded from the command's output).