Feat: adjust fields for ECS compatibility

[kares]

Previously plugin was setting the top level command and host fields for every event (which aren't ECS compliant).

Also changes behavior not to override fields if they exists in the decoded payload (e.g. no longer force the host field if such a field is decoded from the command's output).

[karenzone]

Docs review: I left some comments inline for your consideration.

I see some other changes I'd like to make that pre-date your recent changes. Any chance I could put them in a PR, and include them in your version bump?

[karenzone]

Left a suggestion inline.

In addition, please consider these formatting fixes to sections not included in this PR:

Lines 26-29:

[NOTE]
========
* The `command` field of this event will be the command run.
* The `message` field of this event will be the entire stdout of the command.
========

Lines 31-35:

IMPORTANT: The exec input ultimately uses `fork` to spawn a child process.
Using fork duplicates the parent process address space (in our case, **logstash and the JVM**); this is mitigated with OS copy-on-write but ultimately you can end up allocating lots of memory just for a "simple" executable.
If the exec input fails with errors like `ENOMEM: Cannot allocate memory` it is an indication that there is not enough non-JVM-heap physical memory to perform the fork.

If you'd rather me put these changes in a separate PR, I am happy to do so.

[yaauie]

In general, this looks on-track. I'm concerned by the specs being configured to run differently in CI (does the behaviour differ on docker in general, or on our CI docker? why?)

I've also left a nit about using floats for our intermediate storage mechanism for duration, if we are indeed attempting to track actual nanos.

[yaauie]

nit: can we avoid transitioning through a lossy format, and hold nanos as our intermediate?

[kares]

sure we can, but our over-head won't make it super precise.
forgot JRuby is no longer using currentTimeMillis for Time.now but actually does a native call directly and has nanos -> thus it's a good and valid point - updated, thanks for the advice.

[yaauie]

oof. I don't like skipping specs in CI. Can you explain why this is needed? what doesn't raise?

[kares]

sorry I forgot about this one - should have mentioned the behavior difference in Docker vs non-Docker.
normally one gets Errno::ENOENT: No such file or directory - invalid_command for a missing command,
however the behavior is different in Docker - popen does not raise - we simply get a non-zero exit code:
"message"=>"", "process"=>{"command_line"=>"invalid_command 1 2 3", "exit_code"=>127}

haven't looked into this one as it's existing behavior - thought we rather have this as a known behavior spec-ed.

[yaauie]

If there are two different behaviours, depending on which platform it is run, I would prefer we clearly specify both behaviours instead of specifying one of the behaviours and conditionally executing that spec.

If possible, it could be helpful to emulate the underlying change in behaviour and define our specs so that we can validate our effective behavour in CI regardless of which platform it is executed on (It would be hard to stub IO::popen because we rely on its side-effect of setting $?, but can we stub run_command?).

So our specs output would look something like:

• when command fails
  • on a platform where IO.popen raises ENOENT
    • it does not enqueue an event
  • on a platform where IO.popen does not raise ENOENT (Docker)
    • it enqueues an event tagged with the error code

[kares]

👍 if we're going this far as to specify the behavior that I would rather move this out of this PR.
Have opened an issue #29 + I will create a draft with the suggested spec.
The problematic (newly added) spec has been removed.

[kares]

@karenzone I've updated the docs based on your last suggestion sorry it took so long: 8536b8a

[yaauie]

If there are two different behaviours, depending on which platform it is run, I would prefer we clearly specify both behaviours instead of specifying one of the behaviours and conditionally executing that spec.

If possible, it could be helpful to emulate the underlying change in behaviour and define our specs so that we can validate our effective behavour in CI regardless of which platform it is executed on (It would be hard to stub IO::popen because we rely on its side-effect of setting $?, but can we stub run_command?).

So our specs output would look something like:

• when command fails
  • on a platform where IO.popen raises ENOENT
    • it does not enqueue an event
  • on a platform where IO.popen does not raise ENOENT (Docker)
    • it enqueues an event tagged with the error code

[karenzone]

Thank you for getting this extra changes in. They render nicely and look good. Post-merge LGTM