Logs filling up and crashing Elasticsearch

[alicyn]

I have enabled logs to be exported to elasticsearch running in k8s (AKS). Ever since I enabled that, traces stopped showing in Zipkin/Grafana because Zipkin stores trace data in elasticsearch also (same ES cluster, different index). Each ES node has 10Gi allocated on a persistent volume claim (pvc).

It looks like the logs are filling up ES and causing it to crash. I see messages like this
{"type": "server", "timestamp": "2022-05-18T22:14:43,311Z", "level": "WARN", "component": "o.e.c.r.a.d.DiskThresholdDecider", "cluster.name": "staging-v2", "node.name": "es-cluster-2", "message": "after allocating, node [W4uTIFfWTDKNW1-UY328_g] would have less than the required threshold of 0b free (currently 3.3gb free, estimated shard size is 5gb), preventing allocation", "cluster.uuid": "25jm1xO6SRexj4IZQ9hzlw", "node.id": "W4uTIFfWTDKNW1-UY328_g" }

After some time, I eventually see "all shards failed" error message in Grafana and I get a 503 from my ES cluster.

Is there a best practice for handling data retention on ES? Is there a way to purge old data from ES?

Here is a snippet from my collector config:

service:
    telemetry:
      metrics:
        address: 0.0.0.0:8888
    extensions:
      - health_check
      - memory_ballast
    pipelines:
      logs:
        exporters:
          - otlp
          - elasticsearch
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp
          - filelog

[jkowall]

Elasticsearch ILM
OpenSearch ISM

Depending on which you are using, you can create an index policy. Managing these backends can be a lot of work to scale them properly. Good luck on your journey. :)