[Filebeat] Fix fileset field namespacing #20170
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965
andrewkroh
added
bug
Filebeat
|
Pinging @elastic/siem (Team:SIEM) |
botelastic
bot
added
needs_team
💔 Build Failed
Expand to view the summary
Build stats
Test stats 🧪
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
3 tasks
andrewkroh
added a commit
to andrewkroh/beats
that referenced
this pull request
Jul 23, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965 (cherry picked from commit ea7c05f)
6 tasks
andrewkroh
added
v7.9.0
and removed
needs_backport
andrewkroh
added a commit
to andrewkroh/beats
that referenced
this pull request
Jul 23, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965 (cherry picked from commit ea7c05f)
6 tasks
andrewkroh
added a commit
that referenced
this pull request
Jul 23, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in #19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes #19965 (cherry picked from commit ea7c05f)
andrewkroh
added a commit
that referenced
this pull request
Jul 23, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in #19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes #19965 (cherry picked from commit ea7c05f)
v1v
added a commit
to v1v/beats
that referenced
this pull request
Jul 27, 2020
…ne-2.0 * upstream/master: (41 commits) adding possibility to override content-type checks, it was breaking certain webhooks that is not able to set content-headers at all. Still defaults to application/json (elastic#20232) fix: use a fixed worker type for tests (elastic#20130) [Ingest Manager] Prepare packaging for endpoint and asc files (elastic#20186) [Packetbeat] HTTP: Improve support for 100-continue elastic#15830 (elastic#19349) Increase index.max_docvalue_fields_search to 200 (elastic#20218) [Ingest Manager] Prevent closing closed reader (elastic#20214) [Metricbeat] Use MySQL Host Parser in Query metricset (elastic#20191) Testing: Ignore timestamp from cylance/protect dataset (elastic#20211) [Filebeat] Ignore cylance.protect timestamps while testing (elastic#20207) [CI] remove codecov step (elastic#20102) [docs] Indicate that SYSTEM user is required on Windows to use Endpoint (elastic#20172) Remove f5/firepass rsa2elk fileset (elastic#20160) [Elastic Agent] Improve GRPC stop to be more relaxed. (elastic#20118) Fix fileset field prefixing (elastic#20170) Fix terminating pod autodiscover issue (elastic#20084) Call host parser only once when building light metricsets (elastic#20149) [CI] fix null string with contains (elastic#20182) [Ingest Manager] Fix failing unit tests on windows (elastic#20127) [Filebeat] Update crowdstrike module (elastic#20138) [docs] Add x-pack role to relevant metricsets (elastic#20167) ...
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965 (cherry picked from commit 84a227a)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
When the fields.yml file is constructed it is done by appending files together and adding some indenting.
In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml.
This generally allows for all of the filesets fields to become children of the module.
The problem we had was that the new filesets added in #19713 expected that their fields would be root fields
(not children to the module namespace). In cases where the module already existed and had declared
a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields
(e.g. microsoft.rsa.* instead of rsa.*).
The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana
request payload limit.
Why is it important?
Reduces the number of fields in the Filebeat mapping and allows Kibana to generate a smaller index pattern.
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
filebeat setupRelated issues