[Metricbeat] Elasticsearch enrichment stats API

[cachedout]

Describe the enhancement:

Elasticsearch has recently added a new set of monitoring metrics to track metrics around enrichment policies.

Documents collected via monitoring internal collection from this new API are shaped as follows:

{
  "enrich_coordinator_stats": {
    "queue_size": 0,
    "executed_searches_total": 0,
    "node_id": "daM2_Qf6R6atMzbkdCUDvA",
    "remote_requests_current": 0,
    "remote_requests_total": 0
  },
  "timestamp": "2019-10-24T13:39:48.894Z",
  "interval_ms": 10000,
  "cluster_uuid": "4km3wpEWRp-ba2nntbAwOw",
  "type": "enrich_coordinator_stats",
  "source_node": {
    "transport_address": "10.0.2.15:9300",
    "uuid": "daM2_Qf6R6atMzbkdCUDvA",
    "ip": "10.0.2.15",
    "host": "10.0.2.15",
    "timestamp": "2019-10-24T13:39:48.660Z",
    "name": "packer-virtualbox-iso-1553019999"
  }
}

This issue is to discuss the possible addition of these new metrics to Metricbeat collection. Currently, the monitoring parity tests fail as a result of a divergence between internal collection and Metricbeat collection as a result of this change by the Elasticsearch project.

[ycombinator]

@cachedout I have a few questions after looking at the response from the Enrich Stats API:

1. The API response contains two array fields: executing_policies and coordinator_stats. The example you've shown above seems to correlate to an object in the coordinator_stats array. So do we not want to care about the executing_policies field for Stack Monitoring purposes?

2. The sample document you've shown above shows a single object. However, the coordinator_stats field in the API response is an array of objects. How does the array in the API response map to the object (document) in the index? Do we want to create one document per object in the array?

3. Is the API response expected to be the same regardless of which Elasticsearch node it's run against? Or will it vary depending on the node against which the API is called?

[cachedout]

Hi @ycombinator. These are good questions.

The API response contains two array fields: executing_policies and coordinator_stats. The example you've shown above seems to correlate to an object in the coordinator_stats array. So do we not want to care about the executing_policies field for Stack Monitoring purposes?

After playing around with this a bit and reading this PR it appears to me that executing_policies is not indexed into monitoring documents unless the executing_policies array is populated in the return from GET /_enrich/_stats. However, I would like to verify this with the Elasticsearch team. I am tagging @jakelandis and @martijnvg for further explanation and confirmation about the intended behavior of the collector.

The sample document you've shown above shows a single object. However, the coordinator_stats field in the API response is an array of objects. How does the array in the API response map to the object (document) in the index? Do we want to create one document per object in the array?

This is indeed curious, especially in light of this comment in the PR:

coordinators stats are now in a list instead of a map and has an additional field for the node id

This appears to me as though what's happening is that a single document is being written by the collector for each node, and that document includes the node id. However, again, I think it's best to clarify this with the ES team and see what @jakelandis and @martijnvg have to say.

Is the API response expected to be the same regardless of which Elasticsearch node it's run against? Or will it vary depending on the node against which the API is called?

I'm not sure I fully understand this question. Are you talking about the shape of the document or the contents?

[ycombinator]

Is the API response expected to be the same regardless of which Elasticsearch node it's run against? Or will it vary depending on the node against which the API is called?

I'm not sure I fully understand this question. Are you talking about the shape of the document or the contents?

The contents. Basically, are the contents expected to vary based on which node the API call is made against (e.g. like the GET /_nodes/_local/stats API call which is used to populate node_stats documents in .monitoring-es-*) or are they expected to be the same regardless of which node the API call is made against (e.g. like the GET /_ccr/stats API call which is used to populate ccr_stats and ccr_auto_follow_stats documents in .monitoring-es-*)?

I'm reasonably sure it's the latter but I want to confirm as it affects the implementation a bit.

[cachedout]

I'm reasonably sure it's the latter but I want to confirm as it affects the implementation a bit.

Thanks for the clarification. Makes sense. I agree that what you've described is the most likely scenario but I definitely think it's wise to wait for a definitive answer from the ES team on this and the other two questions.

[ycombinator]

One more question: I see that the Enrich Stats API doc is labeled as "X-Pack". Exactly what license levels is this API available at? Metricbeat will only attempt to call the API if it is expected to exist at the monitored Elasticsearch cluster's license level.

[jakelandis]

it appears to me that executing_policies is not indexed into monitoring documents unless the executing_policies array is populated in the return from GET /_enrich/_stats

Mostly correct, it will not be indexed unless there are running policies, and the running policies can be seen through GET _enrich/_stats. (ES doesn't internally call that API but if it did the result would be the same)

This appears to me as though what's happening is that a single document is being written by the collector for each node, and that document includes the node id. However, again, I think it's best to clarify this with the ES team and see what @jakelandis and @martijnvg have to say.

I believe that is correct. One document per node. I think the map->list is in reference to the change to the _stats endpoint which is not called for monitoring. @martijnvg may be able to confirm.

I see that the Enrich Stats API doc is labeled as "X-Pack". Exactly what license levels is this API available at?

Basic - we are working on getting it added to the matrix.

[ycombinator]

Thanks for the answers, @jakelandis. We also chatted off-issue about:

Is the API response expected to be the same regardless of which Elasticsearch node it's run against? Or will it vary depending on the node against which the API is called?

Jake said:

It shouldn’t matter which node the request goes to (however, you can save a hop if you go to master)