[Filebeat] Discover won't load: Trying to retrieve too many docvalue_fields

[andrewkroh]

Kibana fails while loading Discover. This is the error observed in the network tab of firefox:

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "[status_exception] error while executing search",
  "attributes": {
    "error": {
      "type": "status_exception",
      "reason": "error while executing search",
      "caused_by": {
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
          {
            "shard": 0,
            "index": "filebeat-8.0.0-2020.07.23-000001",
            "node": "1CD091fmR8q1g3oPQaND0A",
            "reason": {
              "type": "illegal_argument_exception",
              "reason": "Trying to retrieve too many docvalue_fields. Must be less than or equal to: [100] but was [118]. This limit can be set by changing the [index.max_docvalue_fields_search] index level setting."
            }
          }
        ],
        "caused_by": {
          "type": "illegal_argument_exception",
          "reason": "Trying to retrieve too many docvalue_fields. Must be less than or equal to: [100] but was [118]. This limit can be set by changing the [index.max_docvalue_fields_search] index level setting.",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Trying to retrieve too many docvalue_fields. Must be less than or equal to: [100] but was [118]. This limit can be set by changing the [index.max_docvalue_fields_search] index level setting."
          }
        }
      }
    }
  }
}

For confirmed bugs, please report:

• Version: filebeat version 8.0.0 (amd64), libbeat 8.0.0 [e25d6c8 built 2020-07-23 17:06:47 +0000 UTC]
• Operating System: Mac OS
• Discuss Forum URL: N/A
• Steps to Reproduce:
  • Run filebeat setup
  • Index Some Data
  • Open Discover

Request from Kibana: request.txt

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[andrewkroh]

This works as a work-around:

setup.template.settings:
  index:
    max_docvalue_fields_search: 200

[nothing-personal-gh]

That workaround doesn't seem to work if I want to set the value to 400. This is my filebeat.yml

setup.template.settings:
  index.max_docvalue_fields_search: 400
  query.default_field: []

filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'

processors:
  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"

output.elasticsearch:
  hosts: ["http://host.docker.internal:9292"]
  indices:
    - index: "filebeat-%{[agent.version]}-%{+yyyy.MM}"

logging.json: true
logging.metrics.enabled: false

But the filebeat index is still created with 200. Looking at the PRs linked above, it seems the value 200 is hardcoded in
code and not read from the config.