Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop Series Cause Write Fail/Write Timeouts/High Memory Usage #8697

Closed
ghost opened this issue Aug 14, 2017 · 3 comments · Fixed by #8801
Closed

Drop Series Cause Write Fail/Write Timeouts/High Memory Usage #8697

ghost opened this issue Aug 14, 2017 · 3 comments · Fixed by #8801

Comments

@ghost
Copy link

ghost commented Aug 14, 2017

When dropping series, we see high memory usage causing write fail/timeouts. If drop series across the entire database, influxdb would crash. when restarting influxdb due to the crash, load are still high (hoovering 7-9), a reboot is then required.

influxdb version: v1.3.2
os: redhat 7.3
spec: m4.2xlarge (8 core / 32GB)

screenshot from influxdb "_internal" stats:

image

image

image

@jwilder
Copy link
Contributor

jwilder commented Aug 15, 2017

Can you run the following when memory usage is high and attach the output to the issue?

curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"

curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt

@pingliu
Copy link

pingliu commented Sep 6, 2017

@jwilder I also have this problem.
image
image

write-timeout = "2m" also write timeout when influxdb drop series.

@jwilder
Copy link
Contributor

jwilder commented Sep 8, 2017

#8801 should fix this.

@ghost ghost removed the review label Sep 11, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants