Data View does not pass configured expand_wildcards value to Elasticsearch when querying field caps

[gwbrown]

Kibana version: 8.0 branch, git hash 4a711e2914907627ff2785d94c1c9191f06db3c0

Elasticsearch version: 8.0 branch, git hash 8d4512f1ebf44521a49f52633f3a1d91e51ad363

Server OS version: MacOS 11.6

Browser version: Firefox 95.0.1

Browser OS version: MacOS 11.6

Original install method (e.g. download page, yum, from source, etc.): Source

Describe the bug: When a data view is created with "Allow hidden and system indices" checked, it correctly uses expand_wildcards=all to query the list of indices on the Data View creation page, but not when that Data View is used in discover, and specifically in a _field_caps call.

Steps to reproduce:

1. Start a fresh 8.0 Elasticsearch cluster & Kibana instance.
2. Go to the "Create a Data View" page
3. Enable the "Allow hidden and system indices" toggle.
4. Type * into the Name field.
5. Watch the spinner on the Timestamp field spin forever, because it doesn't get the response it expects from the _field_caps API because it didn't include the parameter expand_wildcards=all in the request.

Alternatively:

1. Start a fresh 8.0 Elasticsearch cluster & Kibana instance.
2. Go to the "Create a Data View" page.
3. Enter .* in the Name field.
4. Select @timestamp as the timestamp field.
5. Change the Name field to *.
6. Save the Data View.
7. Load the Data View in Discover, watch it never load anything because it can't find any indices because it doesn't include expand_wildcards=all.

Expected behavior: The Discover view should be populated with documents from the system indices on the system (such as .kibana)

Errors in browser console (if relevant):

Uncaught (in promise) Error: No matching indices found: No indices match pattern "*"
    KbnError errors.ts:15
    DataViewMissingIndices errors.ts:18
    _request data_views_api_client.ts:26
errors.ts:15

Provide logs and/or server output (if relevant): Kibana offers to show the error:

KbnError@http://localhost:5601/fln/9007199254740991/bundles/plugin/kibanaUtils/kibana/kibanaUtils.plugin.js:4784:5
DataViewMissingIndices@http://localhost:5601/fln/9007199254740991/bundles/plugin/dataViews/kibana/dataViews.plugin.js:2834:5
_request/<@http://localhost:5601/fln/9007199254740991/bundles/plugin/dataViews/kibana/dataViews.plugin.js:3138:15

Any additional context: I checked this by hand, and if I copy the requests with no changes except adding expand_wildcards=all, the requests appear to return the values you'd expect.

[gwbrown]

I found this via a bug report from @bytebilly, and tracked it down using a custom build of ES that logs all requests, including their paths & parameters. Happy to help track this down from the ES side if that would be useful.

[elasticmachine]

Pinging @elastic/app-search-frontend (Team:AppSearch)

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServicesSv)

[mattkime]

@gwbrown I'm having trouble reproducing the problem. In step #5 its possible that there's a race condition of some sort. Could you share a har file while reproducing the problem?

I'm running 8.0.0-rc1

[gwbrown]

Sure, HARs attached - I did one for each of the repro cases I put in the ticket. If there's a race condition I've been super lucky, as it's quite reliable for me.

HARs taken on Kibana git hash 4a711e2914907627ff2785d94c1c9191f06db3c0 run via yarn start:
Kibana-122899-repro-hars.zip

[mattkime]

@gwbrown I'm pretty sure the problem you're seeing is specific to * but let me know if you can reproduce it with other index patterns.