[elasticsearch/logging] Add slowlogs config option.

[tylersmalley]

For multiple reasons, it's helpful to log the queries to Elasticsearch performed by Kibana. This is often needed for auditing or inspection of slow queries.

Configuration would include an option to only log queries slower than a specified amount of time, producing something similar to slowlog. Setting this to 0, would essentially log everything.

For audit purposes, it's important to associate the query with a user and an action in Kibana (ex: which visualization, health check, task manager). We will need to come up with a solution to provide the ES proxy with this information so it could be included in the logs.

Identify if a query timeout limit was hit. If so, which timeout was responsible? Is it the elasticsearch.requestTimeout, ES search_timeout in request body,

In 8.0 we intend on packaging Filebeat with the Kibana package. Doing so could allow us to easily ingest this data to enable a UI view like what was described in #51224.

@kobelb and @joshdover wanted to ping you both here to make sure this isn't already being done/tracked anywhere since I could see it overlapping.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[tylersmalley]

@epixa I don't think we created an issue after our discussion at EAH. Is there anything you want to add here or that needs correction?

[epixa]

I have nothing to add

[joshdover]

I think there is some overlap here with #57546 (we haven't started work on that yet). In general, I'd just like to see that we're leveraging the new logging config. Maybe this means adding a new layout type for the elasticsearch logging context that is prepopulated with this more detailed information?

@restrry any thoughts?

[mshustov]

(we haven't started work on that yet). In general, I'd just like to see that we're leveraging the new logging config. Maybe this means adding a new layout type for the elasticsearch logging context that is prepopulated with this more detailed information?

log4j allows populating logs with custom data with MDC. We use metadata in the POJO format to provide data to the logger. Then we need to render metadata somehow. I'd expect that we ship preconfigured appender for elasticsearch.client context that has a pattern formatting metadata. For this, we need to extend the meta conversion pattern syntax to support keys lookup. An example of custom pattern for ES:

[%date][%context]  user:%meta{username} url:%meta{url} transferred:%meta{bytes}  - %meta{duration}

Another interesting question: how network layer access application layer data? Such as a username, for example. We will have to extend ElasticsearchService/KibanaRequest API, I suppose.

[spalger]

I've been wondering how easy it would be to attach elasticsearch query logs to every response automatically and automatically harvest those from kfetch/http service/etc on the front end.

[mshustov]

@spalger that's a good idea! In theory, elasticsearch client can attach a request metadata to a response header, and elasticsearch client in the browser can log response metadata in the console. Later, when we have the client-side logging service #33796 we can reuse the same pattern layout on the client.

[spalger]

Thanks, the thing I like about this approach is that it puts the data in the hands of the user and doesn't require storing it anywhere, so it could be on for certain types of users or always in OSS.

[mshustov]

@kobelb @jportner Does this task overlap in any way with #52125?
As I understand, in this task we want to have a performance audit of the ES query. While #52125 records on behalf of which user the operation was performed. Correct me if I wrong, pls.

[jportner]

That's correct, so it sounds like there's some overlap.
#52125 is a security audit log -- it's concerned with who initiated the action and what the outcome was. So:

• We don't want any "performance audit logs" to have user data attached
• It would be nice if the security audit log had any additional info available (such as "which timeout was responsible?" as described above)

[jbudz]

Current status

Migration to the new elasticsearch client is done and logging all queries under logging.verbose: true and elasticsearch.logQueries.

stdout

server    log   [06:16:46.727] [debug][data][elasticsearch][query] 200
PUT /_template/.management-beats
{"index_patterns":[".management-beats"],...

json

{"type":"log","@timestamp":"2020-11-02T06:21:04-06:00","tags":["debug","elasticsearch","data","query"],"p
id":4058,"message":"200\nPOST /_bulk?refresh=false&_source_includes=originId\n{\"update\":{\"_id\":\"task
:endpoint:user-artifact-packager:1.0.0\",\"_index\":\".kibana_task_manager\"}}\n{\"doc\":{\"task\":{\"run
At\":\"2020-11-02T12:22:04.687Z\",\"state\":\"{}\",\"attempts\":0,\"status\":\"idle\",\"startedAt\":null,
\"retryAt\":null,\"ownerId\":null,\"schedule\":{\"interval\":\"60s\"},\"taskType\":\"endpoint:user-artifa
ct-packager\",\"scope\":[\"securitySolution\"],\"params\":\"{\\\"version\\\":\\\"1.0.0\\\"}\",\"scheduled
At\":\"2020-11-02T12:16:51.317Z\"},\"updated_at\":\"2020-11-02T12:21:04.739Z\"}}\n"}

Recap from comments

• related Provide substitution for elasticsearch.logQueries in the new logging config #57546 for changing the kibana.yml setting name,
• Audit logging won't include performance metrics.

Remaining for this issue, feel free to edit:

• slow log settings
• response time from elasticsearch
• x-opaque-id

Do we want these logged to a separate file? Should slow log settings hold off until #57546 to avoid any configuration conflicts?

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)