Filebeat setup is not creating index pattern in kibana #19965
Comments
liza-mae
added
bug
Filebeat
|
Pinging @elastic/integrations (Team:Integrations) |
|
I hit the same problem with 7.9.0 BC1 on centOS 7. Kibana debug logs didn't show anything related to it. |
|
@LeeDr after looking at this closer, I would consider it a blocker since it is causing wider problems in Kibana when creating this index manually and more so when it is the default index pattern.
|
|
I'm seeing the same issue when trying to create a kibana index pattern for Filebeat: Metricbeat seems to work just fine. Maybe it's related to the field count in Filebeat? I don't think user should have to manually change |
|
To elaborate on @kaiyan-sheng's last comment, while creating the index pattern in Kibana, it requests all of the fields for filebeat-* and gets back about 6700 fields (almost a 1MB response) then tries to PUT the index pattern it created and fails with HTTP 413. This is same error code we see when Filebeat tries to insert the index pattern that it generates (although the error was silently ignored due to #20121).
|
|
This is quite a many fields... Is the error because of too many fields in general, or does it indicate some other error? |
|
I tried a small hack and I was able to slim down the size of the index pattern request body from Filebeat by omitting some of the parameters with default values (e.g. count=0, analyzed=false). This allows Filebeat setup to work, but even with that working there are still problems in Kibana. It seems like Kibana still wants to update the index pattern when visiting Discover or opening the Index Pattern UI and this triggers the HTTP 413 despite have already pushed an index pattern. |
|
I have identified about 2000 fields that should not be present. That's about 1/3 of the fields in the mapping so when we fix that it should help this issue.
The fields returned in that list were meant to be named |
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in #19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes #19965
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965 (cherry picked from commit ea7c05f)
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965 (cherry picked from commit ea7c05f)
Summary of Changes
|
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in #19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes #19965 (cherry picked from commit ea7c05f)
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in #19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes #19965 (cherry picked from commit ea7c05f)
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965 (cherry picked from commit 84a227a)
Install and run ES/Kibana.
Install Filebeat.
Due to #19964 - I tried running
./filebeat setup --dashboardsinstead to get around it and it seemed to pass,Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
However, it is not creating the index and even using filebeat setup does not create index pattern .
The text was updated successfully, but these errors were encountered: