Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buffer Overflow produce queue increases exponential. #1047

Open
jvs87 opened this issue Jun 13, 2024 · 0 comments
Open

Buffer Overflow produce queue increases exponential. #1047

jvs87 opened this issue Jun 13, 2024 · 0 comments

Comments

@jvs87
Copy link

jvs87 commented Jun 13, 2024

Problem

We are facing with a problem of buffer overflow, that produce queue increase and cant liberate chunks correctlly so at the end we lose data due to police of drop_oldest_chunks.

Here you can see how buffer increases and also the queue,

image image

...

Steps to replicate

We have cofigurated 7 workers and here is the Elasticsearch configuration

  <match NUEVOES>
    @type elasticsearch
    @id elasticsearch
    validate_client_version false
#    hosts "tdo-pro-elkmaster1.cloudready.cloud.si.orange.es,tdo-pro-elkmaster2.cloudready.cloud.si.orange.es,tdo-pro-elkmaster3.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot1.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot2.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot3.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot4.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot5.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot6.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot7.cloudready.cloud.si.orange.es,tdo-pro-elkdatawarm1.cloudready.cloud.si.orange.es"
    host "kibana.cloudready.cloud.si.orange.es"
    port 9200
    user 
    password 
    scheme https
    ssl_verify false
    ssl_version TLSv1_2
    logstash_format false
    index_name logs.${$.index_name}.%Y-%m-%d
    type_name _doc
    include_timestamp true
    reload_on_failure false
    reload_connections false
    log_es_400_reason true
    id_key _hash
    remove_keys _hash
    <buffer time, tag, $.index_name>
      @type memory
      timekey 60
      timekey_wait 5
      chunk_limit_size 100m
      queue_limit_length 1000
      flush_mode interval
      flush_interval 1s
      flush_at_shutdown true
      flush_thread_count 2
      overflow_action drop_oldest_chunk
    </buffer>
  </match>

Expected Behavior or What you need to ask

Is there any bug on our plugin version that produces that overflow or simply whe have to configurated it correctly?

Using Fluentd and ES plugin versions

  • OS version
  • EC2 instance
  • Fluentd v0.12 or v0.14/v1.0
    • fluentd 1.10.4
  • ES plugin 3.x.y/2.x.y or 1.x.y
    • elasticsearch (7.4.0)
    • elasticsearch-api (7.4.0)
    • elasticsearch-transport (7.4.0)
    • fluent-plugin-elasticsearch (4.0.8)
  • ES version 8.11.1
  • ES template(s) (optional)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant