Query insight - identify important metrics in search workflow

[ansjcy]

Is your feature request related to a problem? Please describe.
As part of the query insight vision, We want to identify crucial staged and metrics in a search workflow so that we can add related query insight features around them, together with #11429

Describe the solution you'd like
Let's think about "what metrics should we instrument to provide better 'insights' to the queries" - Those metrics could be around cache utilization, query rewrite, and any other stage of the query execution.

we can formulate a strategy for identifying those crucial metrics and come up with plans for their collection, processing, and export. For example, we can do profiling on certain concerning queries to know what stage/workflow we should add datapoints on.

Describe alternatives you've considered
N/A

Additional context
Some interesting discussions around this topic in

• [RFC] Real-time Insights into Top N Queries by Latency and Resource Usage #11186

[ansjcy]

Done as part of #11904