Beats will not log monitoring bulk index failure

[cwurm]

When the monitoring user does not have the required privileges to write into .monitoring-beats-* but does have other indexing privileges (e.g. to write to auditbeat-*), a Beat will not log any errors about the failed monitoring bulk index requests.

The reason is that while Elasticsearch will return 403 Forbidden when the user has no write privileges at all, it will return 200 for a _bulk request if a user has at least one write privilege - even if it's not for the monitoring index.

Instead, errors are going to be returned in the body, e.g. see the following (I had to add a debug statement into the code, this is not what a Beat actually prints):

2019-10-29T10:33:46.579Z        DEBUG   [elasticsearch] elasticsearch/client.go:835     POST http://localhost:9200/.monitoring-beats-7-2019.10.29/_bulk 200 {"took":0,"errors":true,"items":[{"index":{"_index":".monitoring-beats-7-2019.10.29","_id":null,"status":403,"error":{"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [auditbeat]"}}}]} {}

For normal event publishing, Libbeat parses the return body in a function called bulkCollectPublishFails:

beats/libbeat/outputs/elasticsearch/client.go

Lines 471 to 475 in 9b96d62

	// bulkCollectPublishFails checks per item errors returning all events
	// to be tried again due to error code returned for that items. If indexing an
	// event failed due to some error in the event itself (e.g. does not respect mapping),
	// the event will be dropped.
	func bulkCollectPublishFails(

But this does not happen for the body of monitoring data bulk requests.

I tried to fix this, but the two logics for sending collected vs. monitoring data are quite different so couldn't do it easily.

/cc @ycombinator - seems you've touched this code most recently