Improving event loop delay representation

[mshustov]

Starting from v7.16, Kibana collects event loop delay distribution provided by nodejs natively in a form of a histogram.
The mean property of the histogram is transformed into ms units and emitted to the Kibana logs

memory: 690.3MB uptime: 0:00:25 load: [4.16,7.93,7.34] delay: 16.996

Also, the same value is provided via /api/status API, which in turn is used by Kibana Monitoring UI.

Using mean as a basic metric has at least one disadvantage: it's hard to tell how many event loop cycles were above a certain limit.
To simplify the investigation of performance problems, Kibana could

• include in the logs an extended sub-set of event loop delay histogram: { 50th: number; 95th: number; 99th: number }.
• provide the event loop delay histogram via /api/status API to allow Monitoring UI to provide more detailed metrics.

cc @pmuellr @kobelb to the question of improving the format of the logs to investigate Kibana performance problems.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pmuellr]

It would be awesome to have the times the EL block was "maxed out", but since the native APIs don't give us that, oh well. TBH, I want the stack trace at the time as well, think I'll be waiting a bit on that :-)

Since the native APIs do let us get p numbers, I think p50/p90/p99 would be nice addition. Just a bit more insight, and more importantly, basically the max.

[Bamieh]

@pmuellr I was hoping the exceeds metric would give that detail but I was unable to figure out what it actually does even after digging into the nodejs code

[TinaHeiligers]

I'll be pushing up a pr soon.