Filebeat Kubernetes Hints - Incorrect Fileset Being Used

[Evesy]

I originally posted this here https://discuss.elastic.co/t/filebeat-hints-using-incorrect-fileset-dataset/187909 but I'm reposting here as I've tested again on a fully 7.x setup with a simpler module configuration with the same outcome, so I do believe it to be a bug.

I have a pod with the below annotations:

    at/logging: "true"
    co.elastic.logs/fileset.stderr: error
    co.elastic.logs/fileset.stdout: access
    co.elastic.logs/module: testing

I then have a filebeat module defined that looks like the below (Carried over from 6.x):

.
 |-access
 |---config
 |-----access.yml
 |---ingest
 |-----pipeline.json
 |---manifest.yml
 |-_meta
 |---config.yml
 |---fields.yml
 |-error
 |---config
 |-----error.yml
 |---ingest
 |-----pipeline.json
 |---manifest.yml

In that pod I am running echo 'STDOUT' >> /proc/1/fd/1 in a loop. I can see the logs are correctly making it through the docker logging driver by cat'ing the log file on the host:

{"log":"STDOUT\n","stream":"stdout","time":"2019-06-28T13:16:09.8800378Z"}
{"log":"STDOUT\n","stream":"stdout","time":"2019-06-28T13:16:11.887124162Z"}
{"log":"STDOUT\n","stream":"stdout","time":"2019-06-28T13:16:13.893990818Z"}

I'd expect based on the annotations & module definition that any stdout logs would use the access fileset (The ingest pipeline for this adds a field with value I definitely went through the stdout pipeline), and stderr logs would use the error fileset (The ingest pipeline for this adds a field with value I definitely went through the stderr pipeline). This setup is working as expected on Filebeat 6.6.1, Elasticsearch 6.8.0.

However, in practice, I see stdout messages picking between the two filesets seemingly at random:

Our Filebeat config is as below:

http:
  enabled: true
  host: localhost
  port: 5066

logging.level: debug

filebeat.registry.flush: 2s

# Overwrite pipelines with new config
filebeat.overwrite_pipelines: true

filebeat.config:
  input:
    # Mounted `filebeat-prospectors` configmap:
    path: ${path.config}/prospectors.d/*.yml
    # Reload prospectors configs as they change:
    reload.enabled: false
  modules:
    path: ${path.config}/modules.d/*.yml
    # Reload module configs as they change:
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      include_annotations: ['at/logging','co.elastic.logs/module','co.elastic.logs/fileset.stderr','co.elastic.logs/fileset.stdout']

processors:
- add_cloud_metadata: ~
- drop_event:
    when:
      not:
        equals:
          kubernetes.annotations.at/logging: 'true'
- drop_event:
    when:
      equals:
        kubernetes.container.name: 'istio-proxy'
- drop_event:
    when:
      equals:
        kubernetes.container.name: 'istio-init'
- decode_json_fields:
    fields: ["message"]
    process_array: false
    max_depth: 1
    target: ""
    overwrite_keys: true

output.elasticsearch:
  hosts: ["<REDACTED>"]
  protocol: 'https'
  username: 'elastic'
  password: "<REDACTED>"
  index: "kubernetes-%{+yyyy.MM.dd}"
  bulk_max_size: 250
  indices:
    - index: "ingress-nginx-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "testing"

setup.template.enabled: false
setup.template.name: 'custom-beats'
setup.template.pattern: "beats-*"
setup.template.overwrite: true
setup.template.settings:
  index:
    number_of_shards: 2
    number_of_replicas: 1
    codec: best_compression

xpack.monitoring.enabled: false

I can't see anything in there that would cause this. The only thing specific to the testing module is to override the index it writes to (Which works, the messages are sent to ingress-nginx-*)

Attached is the testing module: module.tar.gz

Filebeat version: 7.2.0
Elasticsearch version: 7.1.0
Kubernetes version: v1.13.6-gke.13

[jareksm]

I'm experiencing the same issue with filebeat 7.3.0 and default nginx module. The filesets are assigned at random, sometimes stdout out goes to access or error, same with stderr. The only way for me to get any consistency is to set both filesets to access.

Filebeat: 7.3.0
ElasticSearch: 7.3.0
K8s: v1.13.5
Logs: cri format