Rate limit Elasticsearch client indexing error logs

[AndersonQ]

Proposed commit message

rate limit Elasticsearch client indexing error logs

Use `elastic-agent-libs/periodic.Doer` to rate limit indexing error logs. This does not affects the logs sent to the event logger.

Checklist

• My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None

How to test this PR locally

Start Filebeat with the following configuration

filebeat.inputs:
  - type: filestream
    id: filestream-input-id
    enabled: true
    parsers:
      - ndjson:
          target: ""
          overwrite_keys: true
          expand_keys: true
          add_error_key: true
          ignore_decoding_error: false
    paths:
      - /tmp/flog.log

output:
  elasticsearch:
    hosts:
      - localhost:9200 # a valid ES is needed
    protocol: https
    username: elastic
    password: changeme
    allow_older_versions: true
    ssl.verification_mode: none

logging:
  level: info
  event_data:
    files:
      name: filebeat-events-data # that's the default, change it if you want another name.

Create the log file /tmp/flog.log with the following content:

{"message":"foo bar","int":10,"string":"str"}
{"message":"another message","int":20,"string":"str2"}
{"message":"index failure","int":"not a number","string":10}
{"message":"second index failure","int":"not a number","string":10}
A broken JSON

• check filebeat logs, instead of seeing a Cannot index event (status=STATUS): dropping event! for every dropped event, you should see Failed to index N events in last 10s: events were dropped!

Related issues

• Depends Add periodic/Doer for frequency-limited action execution elastic-agent-libs#220
• Closes Rate limit "Cannot index event" log messages #40157

Logs

• logs:

{"log.level":"warn","@timestamp":"2024-08-14T10:03:16.900+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.NewClient.func3","file.name":"elasticsearch/client.go","file.line":175},"message":"Failed to index 2 events in last 10s: events were dropped! Look at the event log to view the event and cause.","service.name":"filebeat","ecs.version":"1.6.0"}

• event log:

{"log.level":"warn","@timestamp":"2024-08-14T10:03:10.714+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus","file.name":"elasticsearch/client.go","file.line":517},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Meta:null, Fields:null, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}, EncodedEvent:(*elasticsearch.encodedEvent)(0xc00199d980)} (status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:368] failed to parse field [int] of type [long] in document with id 'Gt7pT5EBJuRc3c2Nhw20'. Preview of field's value: 'not a number'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"For input string: \\\"not a number\\\"\"}}, dropping event!","service.name":"filebeat","log.type":"event","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-08-14T10:03:10.714+0200","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus","file.name":"elasticsearch/client.go","file.line":517},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Meta:null, Fields:null, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}, EncodedEvent:(*elasticsearch.encodedEvent)(0xc00199da00)} (status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:48] failed to parse field [int] of type [long] in document with id 'G97pT5EBJuRc3c2Nhw20'. Preview of field's value: 'not a number'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"For input string: \\\"not a number\\\"\"}}, dropping event!","service.name":"filebeat","log.type":"event","ecs.version":"1.6.0"}

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 40157-rate-limit-cannot-index-logs upstream/40157-rate-limit-cannot-index-logs
git merge upstream/main
git push upstream 40157-rate-limit-cannot-index-logs

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 40157-rate-limit-cannot-index-logs upstream/40157-rate-limit-cannot-index-logs
git merge upstream/main
git push upstream 40157-rate-limit-cannot-index-logs

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

LGTM, just a suggestion to document why we did this.

[cmacknz]

I suggest adding a comment explaining why we are rate limiting these so future readers don't need to guess, and can avoid adding new per-event info logs.