[Index patterns] Remove field cache

[mattkime]

Summary

Removes index pattern field cache and field refresh ui. Field list is loaded via field caps api when index pattern is loaded. Field attributes that we wish to persist are now saved separately from the field list on the SavedObject. The field list will only contain scripted fields.

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[TDonselaar]

Hi,

This new feature is a step back for us. It removes the abilities to hide fields for other spaces.

Our use case:

We are running Kibana with multiple spaces. Every space is a customer.
The customer should not see all index fields that they aren't going to use in there dashboard.
We would update the saved object for the index pattern and simply remove fields in the json.
The removing of the fields in the saved object allows us to remove fields in the UI per space.

This all was working great.

With the new update the index field list is generated on-to-fly when kibana loads.
This results in us not able to hide fields for spaces.

We have manged to mitigate this by added custom code to kibana, so it just doesn't return the fields we don't want.
However, this solution is hacky.

Can we have the abilities to hide fields?

[Dosant]

@TDonselaar, thanks for the feedback. FYI seems like this enhancement issue would cover your use case.

Also, you can try to use field filters for this, which is a recommended way.

[TDonselaar]

Hi @Dosant Issue #17270 is indeed exactly what we need hopefully the team will add this feature.
Regarding Field filters it doesn't fit our use case.

[mattkime]

@TDonselaar

Regarding Field filters it doesn't fit our use case.

How so? It might not be ideal, but we should be able to find a work around.

[TDonselaar]

Filter fields just exclude the fields in the results, it doesn't stop them showing up in the filtering part of the UI.

Example when I add a filter field:

Then I go to a new dashboard and open a filter popup I still see the fields I added to the filter fields.

So you see that the customer can still use the field.

[mattkime]

I'm currently verifying field filter functionality. I'm seeing the same thing you are and it doesn't appear to be working.

[mattkime]

@TDonselaar You could also look at using field level security in elasticsearch, this might even be a preferred method of working as its being enforced at a lower level. - https://www.elastic.co/guide/en/elasticsearch/reference/current/field-level-security.html

[TDonselaar]

@TDonselaar You could also look at using field level security in elasticsearch, this might even be a preferred method of working as its being enforced at a lower level. - https://www.elastic.co/guide/en/elasticsearch/reference/current/field-level-security.html

I could think of two issue with my use case that might happen if I used field level security:

• The blocked fields are only filtered on the output/input for the request. The fields might still show up when Kibana requests a list of field mappings from elasticsearch.

• I use scripted fields and custom queries in Vega charts. The fields I use are fields that are hidden from the user. I think blocking them in elasticsearch would break my scripted fields and Vega charts.

I haven't tested field level security, so I'm just presuming that I'll run into these issues.

So #17270 option to hide fields would be a solution for my use case.

[mattkime]

@TDonselaar

The fields might still show up when Kibana requests a list of field mappings from elasticsearch.

This specifically does not happen.

I think blocking them in elasticsearch would break my scripted fields and Vega charts.

Thats true.

---

I think you should create a new issue with an enhancement request - a method of hiding fields from field lists. The fields still need to exist to be used in queries but shouldn't be presented to the user. The more detail about your specific needs the better.

[b0le]

We have encountered an issue that might be related to this change.

Scenario using Elastic Stack 7.15.1:

1. We began ingesting Kafka metrics using Metricbeat (default metricsets: partiition and consumergroup).
2. Checking the default Kafka dashboard that comes with Metricbeat reveals metrics are being visualized but the Controls visualization which allows for filtering by Kafka topic is reporting an error:

Filtering occurs on the "kafka.topic.name" field which does not exist on any documents in the "metricbeat-*" index pattern. Choose a different field or index documents that contain values for this field.

3. We can confirm that documents with "kafka.topic.name" field are indeed being ingested.
4. Checking Kibana's Index Pattern management page reveals that none of the kibana.* fields are listed.
5. There's no clear way of refreshing the fields list.

@mattkime

Field list is loaded via field caps api when index pattern is loaded.

When precisely does this happen, i.e. the loading of the index pattern? Is it performed on regular intervals, triggered by some specific event or is the load performed only on Kibana's startup?

[mattkime]

@b0le Could you create a new issue and include a har file captured while reproducing the problem? I'm particularly interested in the fields_for_wildcard request.

When precisely does this happen, i.e. the loading of the index pattern?

On page load or when navigating between Kibana applications.