Filebeat json parser is trying to parse excluded lines

[dosmanak]

• Version: 8.2.0
• Operating System: Irrelevant / Docker on Linux
• Discuss Forum URL: https://discuss.elastic.co/t/filebeat-json-parser-is-trying-to-parse-excluded-lines/304674

filebeat log is filled with errors "Error decoding JSON: invalid character"

Steps to Reproduce:

filebeat.yml

---
filebeat.inputs:
  - type: filestream
    id: 0
    paths:
      - '/data/mixed_json_plain.log'
    exclude_lines: [ "^PLAIN" ]
    parsers:
      - ndjson:
          add_error_key: true
          keys_under_root: true
          overwrite_keys: true
          message_key: message
          expand_keys: true

output.file:
  path: /data/filebeat_output
  rotate_every_kb: 100000
  rotate_on_startup: false
  codec.json:
    pretty: true

log sample file mixed_json_plain.log

{"id":1,"message":"lorem ipsum"}
PLAIN first line
PLAIN second line
{"id":2,"message":"dolor sit amet"}
{"id":3,"message":"consectetur adipiscing elit"}

To reproduce errors

docker run --rm -v $PWD:/data -v $PWD/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:8.2.0

When run with -e -d '*' it is visible on debug level, the line is exluded correctly.

Drop line as it does match one of the exclude patterns%sPLAIN first line

Full debug log is in linked Discuss Forum URL
Interresting lines are

{"log.level":"error","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"parser_json","log.origin":{"file.name":"readjson/json.go","file.line":74},"message":"Error decoding JSON: invalid character 'P' looking for beginning of value","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-05-31T15:20:47.977Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":351},"message":"Drop line as it does match one of the exclude patterns%sPLAIN first line","service.name":"filebeat","id":"0","source_file":"filestream::0::native::7639481-54","path":"/data/mixed_json_plain.log","state-id":"native::7639481-54","ecs.version":"1.6.0"}

[dosmanak]

I have changed to processors.decode_json_fields which works fine even on multiline json events without specifying so.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[kvch]

The log input behaves the same way, and produces the same error. Maybe in filestream we could make exclude_lines and include_lines parser, so we have more flexibility in filtering out lines.