[Filebeat][proposal] Disable filesets that are not configured

[andrewkroh]

The default behavior for Filebeat modules is that all filesets are enabled by default. This puts a burden on users of modules when they upgrade versions. When a new fileset is added this is a breaking change to the end-user because they must take some action to change their configuration in order maintain the same behavior.

For example, when we add a new fileset to AWS module the end-user must add the new fileset to the configuration file just to disable it. If they don't then they will end up with errors because configuration is required for the new fileset to function.

A similar problem also manifests itself to users of the CLI when trying to enable particular filesets of a module. They get all filesets enabled by default. If they want to enable a subset of filesets they must use a wildcard option (that's undocumented AFAIK) -M aws.*.enabled=false.

My proposal is to change the default behavior of filesets to disabled by default when they do not appear in the configuration.

[kaiyan-sheng]

I agree, right now when user trying to run Filebeat with only cloudtrail fileset enabled, the config file looks like this:

filebeat.modules:
- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: https://sqs.us-east-1.amazonaws.com/428152502467/test-fb-ks
    var.credential_profile_name: elastic-beats
    var.visibility_timeout: 300s
    var.endpoint: amazonaws.com
  elb.enabled: false
  s3access.enabled: false
  vpcflow.enabled: false

output.elasticsearch:
  hosts: ["localhost:9200"]

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[exekias]

+1 on doing this, filesets requiring special settings should be manually enabled to avoid errors and wasting cpu cycles

[exekias]

btw this is a breaking change, right? How do you feel about how it will affect existing deployments?

[narph]

+1 , we are seeing issues with this behavior as well in azure, has this gotten any traction so far?

[elasticmachine]

Pinging @elastic/agent (Team:Agent)