Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add ecs support #179

Merged
merged 12 commits into from
Mar 25, 2021
Merged

Conversation

kaisecheng
Copy link
Contributor

Elastic Common Schema Support
target uses client as the default in ECS; legacy keeps geoip as default

Co-authored-by: Ry Biesemeyer ry.biesemeyer@elastic.co

Fix: #163

kaisecheng and others added 4 commits March 16, 2021 14:22
Co-authored-by: Ry Biesemeyer <ry.biesemeyer@elastic.co>
@kaisecheng kaisecheng requested a review from yaauie March 16, 2021 16:05
Copy link
Contributor

@yaauie yaauie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First: 🌮🌮🌮 (I meant 🎉 but 🌮 are also good 😂 ). Thank you much for carrying this forward.

I've left a couple notes in-line.


I also have thoughts on user experience that I completely failed to define in the original requirements 🤦🏼 :

I believe we should auto-target when the source specifies an ip sub-field, and should otherwise require a target in ECS mode instead of guessing client.

For example, when the user specifies source => "[host][ip]" and doesn't provide a target we can safely assume target => "[host]", but if they specify a non-ECS source => "host_ip" we can't reasonably assume enough in the context of ECS to properly categorize it.

Pseudo-code:

def default_ecs_target
  return @source[0...-4] if @source.end_with?('[ip]')

  fail(LogStash::ConfigurationError, "GeoIP Filter in ECS-Compatiblity mode requires a `target` when `source` is not an `ip` sub-field")
end

lib/logstash/filters/geoip.rb Outdated Show resolved Hide resolved
def setup_target_field
@target ||= ecs_select[disabled:'geoip', v1:'client']

if ecs_compatibility != :disabled and !ECS_TARGET_FIELD.include?(@target)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 this will warn when a user explicitly specifies a field in the canonical field reference syntax ([source]) but not in the top-level-field shorthand syntax (source).

We may need to normalize the input before validating.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch!

spec/filters/geoip_ecs_spec.rb Outdated Show resolved Hide resolved
spec/filters/geoip_ecs_spec.rb Show resolved Hide resolved
auto set `target` from `source`
normalize `target`
@kaisecheng kaisecheng requested review from yaauie and removed request for yaauie March 22, 2021 14:05
Copy link
Contributor

@yaauie yaauie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏼 great spec coverage!

I've left a couple nitpicks about constraining assertions in specs, and a suggested refactoring of the target detection to make it easier to read and reason about.

lib/logstash/filters/geoip.rb Outdated Show resolved Hide resolved
spec/filters/geoip_ecs_spec.rb Outdated Show resolved Hide resolved
spec/filters/geoip_ecs_spec.rb Outdated Show resolved Hide resolved
spec/filters/geoip_ecs_spec.rb Outdated Show resolved Hide resolved
spec/filters/geoip_ecs_spec.rb Outdated Show resolved Hide resolved
spec/filters/geoip_ecs_spec.rb Outdated Show resolved Hide resolved
@kaisecheng kaisecheng requested a review from yaauie March 23, 2021 16:26
Copy link
Contributor

@karenzone karenzone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left comment in line. Otherwise, builds cleanly and LGTM.

@@ -89,6 +89,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
[cols="<,<,<",options="header",]
|=======================================================================
|Setting |Input type|Required
| <<plugins-{type}s-{plugin}-ecs_compatibility>> | <<string,string>>|No
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doc: Please order alphabetically in both the table and doc description. Otherwise, build cleanly and LGTM>

Copy link
Contributor

@yaauie yaauie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 LGTM

@kaisecheng kaisecheng merged commit f9de15b into logstash-plugins:master Mar 25, 2021
@kaisecheng
Copy link
Contributor Author

published at v7.1.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement ECS-Compatibility Mode
3 participants