Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ElasticSearch ran out of disk space #2135

Closed
mdebarros opened this issue Apr 1, 2021 · 1 comment
Closed

ElasticSearch ran out of disk space #2135

mdebarros opened this issue Apr 1, 2021 · 1 comment
Assignees
@mdebarros
Labels
bug Something isn't working or it has wrong behavior on a Mojaloop Core service oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it
Milestone
Sprint 13.5

Comments

@mdebarros
Copy link
Member

Summary:
ElasticSearch deployment in OSS Dev1 environment has run out of disk space

{"type": "server", "timestamp": "2021-03-31T11:57:04,123Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "high disk watermark [90%] exceeded on [xM3x3yASQA6pUIyvb8MmDA][elasticsearch-master-0][/usr/share/elasticsearch/data/nodes/0] free: 5.6gb[5.7%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete", "cluster.uuid": "rK-ShzSYTuql7hNt5jVp3Q", "node.id": "xM3x3yASQA6pUIyvb8MmDA"  }

Severity:
Low

Priority:
Low

Expected Behavior

Apply ElasticSearch policy to rollover index's to ensure that disk space resources are not exhausted.

Steps to Reproduce

  1. Deploy ElasticSearch from Mojaloop Helm Repo
  2. Wait around 2 months
  3. Monitor disk space

Specifications

  • Component (if known): ElasticSearch
  • Version: 11.0.0 (master branch - should be v12.0.0)
  • Platform: Helm
  • Subsystem: n/a
  • Type of testing: Manual
  • Bug found/raised by: @vijayg10

Notes:

  • Severity when opened: Low
  • Priority when opened: Low
@mdebarros mdebarros added bug Something isn't working or it has wrong behavior on a Mojaloop Core service oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it labels Apr 1, 2021
@mdebarros mdebarros self-assigned this Apr 7, 2021
@mdebarros
Copy link
Member Author

mdebarros commented Apr 8, 2021 •
edited
Loading

Pull Requests:

This was referenced Apr 8, 2021
Merged
Merged
mdebarros added a commit to mojaloop/helm that referenced this issue Apr 8, 2021
@mdebarros
feat(bug): #2135 elasticSearch ran out of disk space (#434)
1af77fd 
- Added rollover policy, and updated readme to include how to manage the policy for standard logstash indexes within the EFK documentation - fixes for mojaloop/project#2135
- Updated EFK Chart versions (which was incorrectly missed in last PR)
mdebarros added a commit to mojaloop/event-stream-processor that referenced this issue Apr 8, 2021
@mdebarros
Merge pull request #36 from mdebarros/bug/#2135-ElasticSearch-ran-out…
0499aef 
…-of-disk-space

Added rollover policy, and updated readme to include how to manage the policy - fixes for mojaloop/project#2135
@elnyry-sam-k elnyry-sam-k added this to the Sprint 13.5 milestone Apr 9, 2021
@vgenev vgenev closed this as completed Apr 12, 2021
@mdebarros mdebarros mentioned this issue Jun 28, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mdebarros mdebarros

Labels
bug Something isn't working or it has wrong behavior on a Mojaloop Core service oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it
Projects
None yet
Milestone
Sprint 13.5
Development

No branches or pull requests
3 participants
@vgenev @elnyry-sam-k @mdebarros