Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Influxdb crashes after: "panic: runtime error: makeslice: cap out of range" #8952

Closed
dirkjankrijnders opened this issue Oct 11, 2017 · 1 comment

Comments

@dirkjankrijnders
Copy link

dirkjankrijnders commented Oct 11, 2017

System info:
InfluxDB starting, version 1.3.5, branch HEAD, commit 9d90010
Go version go1.8.3, GOMAXPROCS set to 4
on system:
Linux ha 4.9.41-v7+ #1023 SMP Tue Aug 8 16:00:15 BST 2017 armv7l GNU/Linux
Package from: https://repos.influxdata.com/debian jessie stable
Running on an raspberry pi 3

Steps to reproduce:

Expected behavior: A running influxdb

Actual behavior: Crash and restart (by system)

Additional info: The log says:
Oct 11 08:27:30 ha influxd[11968]: [I] 2017-10-11T08:27:30Z reading file /var/lib/influxdb/wal/telegraf/autogen/113/_00209.wal, size 359191 engine=tsm1 service=cacheloader
Oct 11 08:27:30 ha influxd[11968]: [I] 2017-10-11T08:27:30Z reading file /var/lib/influxdb/wal/telegraf/autogen/113/_00210.wal, size 10488123 engine=tsm1 service=cacheloader
Oct 11 08:27:30 ha influxd[11968]: panic: runtime error: makeslice: cap out of range
Oct 11 08:27:30 ha influxd[11968]: goroutine 89 [running]:
Oct 11 08:27:30 ha influxd[11968]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WriteWALEntry).UnmarshalBinary(0x11bf5578, 0x16cc0000, 0x28a, 0xa2de6, 0x0, 0x17dba020)
Oct 11 08:27:30 ha influxd[11968]: /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:816 +0xb80
Oct 11 08:27:30 ha influxd[11968]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WALSegmentReader).Next(0x10ec9bb0, 0x13ed1f00)
Oct 11 08:27:30 ha influxd[11968]: /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:1134 +0x448
Oct 11 08:27:30 ha influxd[11968]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load.func1(0x10ec9c28, 0x13da00e0, 0x10f288a0, 0x0, 0x0)
Oct 11 08:27:30 ha influxd[11968]: /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:621 +0x230
Oct 11 08:27:30 ha influxd[11968]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load(0x13da00e0, 0x10f288a0, 0x144f2f90, 0x13da00e0)
Oct 11 08:27:30 ha influxd[11968]: /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:645 +0x5c
Oct 11 08:27:30 ha influxd[11968]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).reloadCache(0x112f1c80, 0x0, 0x0)
Oct 11 08:27:30 ha influxd[11968]: /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1499 +0x138
Oct 11 08:27:30 ha influxd[11968]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).Open(0x112f1c80, 0x8e9200, 0x10d0e180)
Oct 11 08:27:30 ha influxd[11968]: /root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:486 +0xa8
Oct 11 08:27:30 ha influxd[11968]: github.com/influxdata/influxdb/tsdb.(*Shard).Open.func1(0x10c60480, 0x0, 0x0)
Oct 11 08:27:30 ha influxd[11968]: /root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:293 +0x258
Oct 11 08:27:30 ha influxd[11968]: github.com/influxdata/influxdb/tsdb.(*Shard).Open(0x10c60480, 0x8e92b8, 0x10d0e180)
Oct 11 08:27:30 ha influxd[11968]: /root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:304 +0x1c
Oct 11 08:27:30 ha influxd[11968]: github.com/influxdata/influxdb/tsdb.(*Store).loadShards.func1(0x10d7e600, 0x10d7a1c0, 0x10d7e680, 0x621ec0, 0x10c10b40, 0x10d82737, 0xe, 0x10d0e296, 0x7, 0x10c823ae, ...)
Oct 11 08:27:30 ha influxd[11968]: /root/go/src/github.com/influxdata/influxdb/tsdb/store.go:242 +0x3c4
Oct 11 08:27:30 ha influxd[11968]: created by github.com/influxdata/influxdb/tsdb.(*Store).loadShards
Oct 11 08:27:30 ha influxd[11968]: /root/go/src/github.com/influxdata/influxdb/tsdb/store.go:250 +0x608

Also, if this is an issue of for performance, locking, etc the following commands are useful to create debug information for the team.

curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"

curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt

These commands cannot be executed as the server never gets this far...

@dirkjankrijnders dirkjankrijnders changed the title Influxdb keeps restarting after: "panic: runtime error: makeslice: cap out of range" Influxdb crashes after: "panic: runtime error: makeslice: cap out of range" Oct 11, 2017
@dirkjankrijnders
Copy link
Author

Reproducible with the 1.3.6 and nightly tar.gz from the website as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants