[Filebeat-8.0]: module system is configured but has no enabled fileset: error on running filebeat setup command.

[amolnater-qasource]

Kibana version: 8.0 Snapshot Kibana Cloud-qa environment

Host OS: Ubuntu 20(.tar), Centos 8(.rpm), Debian 10(.deb) and MAC

Build details:

Build: 48414
Commit: e5070e0ebad538dd152fbc8e016091a2580047b4
Artifact Link: https://snapshots.elastic.co/8.0.0-94dc1348/downloads/beats/filebeat/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz

Steps to reproduce:

1. Download and extract filebeat artifact.
2. Update the filebeat.yml:

cloud.id: "XXXXXXXXXXXXXXXXXXXXXXXLXdlc3QxLmdjcC5jbG91ZC5lcy5pbyQ4MGM4NDgyYjRlODE0Yzkxxx=="
cloud.auth: "elastic:IEtKXRAmrH4zuGxxxxxxx"

3. Run: ./filebeat modules enable system.
4. Run command:./filebeat setup -e.
5. Observe below error:

filebeat.yml:
filebeat.zip

CLI-detailed steps:
CLI-filebeat.txt

Expected Result:
No error should be there on running filebeat setup -e command, and filebeat should show data under Discover tab.

Reference Document:
https://www.elastic.co/guide/en/beats/filebeat/7.15/filebeat-installation-configuration.html

NOTE:

• We are unable to get filebeat data on Ubuntu 20(.tar), Centos 8(.rpm), Debian 10(.deb) and MAC hosts.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Reviewed & mentioned to @andresrc

CC: @EricDavisX

[kvch]

This is expected behaviour. From 8.0, all filesets are disabled by default and users have to enable them manually. An error is returned when nothing is enabled to let users know if they forgot turn on modules/filesets.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[kvch]

I talked with @amolnater-qasource offline, but I am sharing it here as well. To avoid this error, you have to enable syslog and audit filesets in the file modules.d/system.yml:

- module: system
  # Syslog
  syslog:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Authorization logs
  auth:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

[amolnater-qasource]

Hi @kvch
Thanks for sharing the update.
On updating both syslog and auth to true under modules.d/system.yml.
We are successfully able to get data under Discover tab.

QUERY:
Further under Discover tab: we got a new index could you please confirm if this is an issue or expected?:

We have only enabled system module.

@EricDavisX
We have updated our test content for Filebeat installation as per this update.

Please let us know if anything else is required.
Thanks

[kvch]

@ruflin Is this new "apm--transaction..." template expected?

[ruflin]

@simitt @sqren You might be able to help with the above? I assume this is coming from APM?

[EricDavisX]

Hi @kvch Thanks for sharing the update. On updating both syslog and auth to true under modules.d/system.yml. We are successfully able to get data under Discover tab.

@EricDavisX We have updated our test content for Filebeat installation as per this update.

Hi @amolnater-qasource can you do a Filebeat docs check to see if it was updated to indicate this new expectation and general info on how to update it to 'see' any data come in? If it needs it, we can log a separate docs ticket and ref this.

[sorenlouv]

@ruflin Is this new "apm--transaction..." template expected?

Yes, it's automatically created when the APM UI app is opened.
It's possible to disable this

[amolnater-qasource]

Hi @sqren
We haven't accessed APM UI and haven't done anything related to APM.
These datasets were there on running Filebeat with both syslog and auth filesets from system.yml.
You can refer screenshot at: #29175 (comment)
agent.type= filebeat.

Is that expected to get this for even Filebeat or it is an issue?

@EricDavisX
Under docs yes it is there that we need to enable the required filesets.

However till 7.16 we never enabled these, as by default these filesets gets enabled on running ./filebeat modules enable system for any module.
On 8.0 its set to false even after enabling system, user has to manually do it as confirmed at #29175 (comment)
It might be confusing for the first time.

Further for datastreams it is only mentioned to show for filebeat-* index.
On confirmation from @sqren we will log the required ticket.

cc: @ruflin
Thanks

[sorenlouv]

Is that expected to get this for even Filebeat or it is an issue?

I don't see any problems here. The user can delete the data view (index pattern) manually and stop it from being created again by setting xpack.apm.autocreateApmIndexPattern: false.

[amolnater-qasource]

Thanks @sqren for sharing the feedback.

@EricDavisX

• Documents do have these updates available for 8.0.
• There is no discussion of the new index("apm--transaction...") under the docs.
• However as confirmed above it is not an issue that a new index is created with filebeat-*.

Could you please confirm if any action is required for the same or we should mark this as done?

Thanks!

[EricDavisX]

excellent summary - @amolnater-qasource please scan through the docs repo and put a ticket in if we don't see one for the docs, then we can call it done. :)

[amolnater-qasource]

Hi @EricDavisX
As per feedback we have logged a docs ticket for these changes regarding Filebeat installation at elastic/observability-docs#1309

Thanks

[metalshanked]

Hi @kvch , @amolnater-qasource
I am facing the error when using custom inputs for filebeat modules. Can anyone please advise on how to solve this in 8.0.0? (It works great in 7.x)

filebeat.modules:   
  - module: panw
    panos:
      enabled: true
      var:
        input: custom
      input:
        type: kafka
        hosts:

"message":"Error creating runner from config: could not create module registry for filesets: module panw is configured but has no enabled filesets","service.name":"filebeat","ecs.version":"1.6.0"}

I updated panw.yml in modules.d to enabled and now the above error goes away but a new error crops up

...Unable to encode event: unsupported"

Same events which worked fine with 7.x show the above error

[renzedj]

Hi @kvch Thanks for sharing the update. On updating both syslog and auth to true under modules.d/system.yml. We are successfully able to get data under Discover tab.

After enabling modules, as well as syslogs in the modules, I'm still having the same issue.