[Feature Request] Add metrics to track latency issue due to search idle feature

[ruai0511]

Is your feature request related to a problem? Please describe

Shards refresh automatically in the background by default very second. When no search request lands on a shard for over 30 seconds, the shard will go into an idle state where it stops the implicit index refresh to improve performance. (more information on Search Idle here).

However, this causes another problem. After a shard goes idle, there are still write requests coming in (write requests doesn’t affect the search idle feature), so the next search request will need to force a refresh on the shard to get the most updated data - to make sure all the write requests we got during idle are reflected - and then serve the result. However, the additional step here to force a refresh consumes time and increases latency.

We'd like to observe how often an idle shard is being "woken up".

Describe the solution you'd like

As a first step to improve the performance due to this force refresh, we'd like to add metrics to track this issue. That is, to add a counter to reflect when a search request wakes up a shard, and then export it through an API for visibility. This counter makes sure we know the timestamp of a "blocking index refresh" happening, so that we can observe the relationship between a shard wakes up and high latency. The metrics tracker will be the first step of solving the issue Here.

Related component

Search

[andrross]

[Triage - attendees 1 2 3 4 5 6]
@ruai0511 Thanks for filing, looking forward to seeing progress here.