Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[0.12] Panic on shutdown #6140

Closed
desa opened this issue Mar 28, 2016 · 4 comments · Fixed by #6151
Closed

[0.12] Panic on shutdown #6140

desa opened this issue Mar 28, 2016 · 4 comments · Fixed by #6151
Labels
panic
Milestone
0.12.0

Comments

@desa
Copy link
Contributor

desa commented Mar 28, 2016

Got a random panic when I tried to shutdown my local instance:

monitor] 2016/03/28 16:56:33 shutting down monitor system
[monitor] 2016/03/28 16:56:33 terminating storage of statistics
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x30 pc=0x2325c5]

goroutine 300686 [running]:
github.com/influxdata/influxdb/tsdb.(*Shard).CreateIterator(0xc841b8cfc0, 0x1907a08, 0xc82dd29aa0, 0xbed000, 0x0, 0x0, 0xc84a6aa460, 0x1, 0x1, 0x0, ...)
    /Users/michaeldesa/go/src/github.com/influxdata/influxdb/tsdb/shard.go:420 +0x115
github.com/influxdata/influxdb/tsdb.(*shardIteratorCreator).CreateIterator(0xc86fe646a0, 0x1907a08, 0xc82dd29aa0, 0xbed000, 0x0, 0x0, 0xc84a6aa460, 0x1, 0x1, 0x0, ...)
    /Users/michaeldesa/go/src/github.com/influxdata/influxdb/tsdb/shard.go:886 +0x69
github.com/influxdata/influxdb/influxql.IteratorCreators.CreateIterator.func1(0xc8200c8a00, 0x12, 0x20, 0xc8b1630990, 0xc8b1630880, 0x0, 0x0)
    /Users/michaeldesa/go/src/github.com/influxdata/influxdb/influxql/iterator.go:483 +0xdf
github.com/influxdata/influxdb/influxql.IteratorCreators.CreateIterator(0xc8200c8a00, 0x12, 0x20, 0x1907a08, 0xc82dd29aa0, 0xbed000, 0x0, 0x0, 0xc84a6aa460, 0x1, ...)
    /Users/michaeldesa/go/src/github.com/influxdata/influxdb/influxql/iterator.go:481 +0xd8
github.com/influxdata/influxdb/influxql.(*IteratorCreators).CreateIterator(0xc8331b5f40, 0x1907a08, 0xc82dd29aa0, 0xbed000, 0x0, 0x0, 0xc84a6aa460, 0x1, 0x1, 0x0, ...)
    <autogenerated>:152 +0xeb
github.com/influxdata/influxdb/influxql.buildExprIterator.func1(0xc82dd29aa0, 0x1907e28, 0xc8331b5f40, 0xc8b1631240, 0x0, 0x0, 0x0, 0x0)
    /Users/michaeldesa/go/src/github.com/influxdata/influxdb/influxql/select.go:255 +0x196
github.com/influxdata/influxdb/influxql.buildExprIterator(0x1907a08, 0xc82dd29aa0, 0x1907e28, 0xc8331b5f40, 0x1907a08, 0xc82dd29aa0, 0xbed000, 0x0, 0x0, 0xc84a6aa460, ...)
    /Users/michaeldesa/go/src/github.com/influxdata/influxdb/influxql/select.go:339 +0x13a0
github.com/influxdata/influxdb/influxql.buildFieldIterators.func1(0xc86fe64618, 0x1, 0x1, 0x1907e28, 0xc8331b5f40, 0xc8b16315d8, 0xc84a6aa840, 0x1, 0x1, 0x0, ...)
    /Users/michaeldesa/go/src/github.com/influxdata/influxdb/influxql/select.go:154 +0x5b6
github.com/influxdata/influxdb/influxql.buildFieldIterators(0xc86fe64618, 0x1, 0x1, 0x1907e28, 0xc8331b5f40, 0x0, 0x0, 0xbed000, 0x0, 0x0, ...)
    /Users/michaeldesa/go/src/github.com/influxdata/influxdb/influxql/select.go:140 +0xf8
github.com/influxdata/influxdb/influxql.Select(0xc8861ce000, 0x1907e28, 0xc8331b5f40, 0xc8b1631c90, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/michaeldesa/go/src/github.com/influxdata/influxdb/influxql/select.go:81 +0xba9
github.com/influxdata/influxdb/cluster.(*QueryExecutor).executeSelectStatement(0xc8200a4400, 0xc8861ce000, 0x2710, 0x0, 0xc82ec42480, 0xc82ec424e0, 0x0, 0x0)
@jwilder jwilder added this to the 0.12.0 milestone Mar 29, 2016
@e-dard e-dard added the panic label Mar 29, 2016
@jsternberg
Copy link
Contributor

@mjdesa any more information on how to reproduce this? Was a query currently being executed at the time?

@e-dard
Copy link
Contributor

e-dard commented Mar 29, 2016

@mjdesa this is happening due to a race between shutting down the shard and a running query that accesses it. @jsternberg is going to add a Shutdown method to the query manager so all running queries are killed on shutdown.

@jsternberg jsternberg mentioned this issue Mar 29, 2016
Merged
jsternberg added a commit that referenced this issue Mar 29, 2016
@jsternberg
Have the server kill all queries on shutdown
c1643e6 
Related to #6140, but won't actually fix that problem. It will correctly
stop new queries from being started during shutdown and will send the
interrupt signal to queries during shutdown.

Since the interrupt signal is asynchronous, there isn't currently a way
to wait for the queries to complete themselves before shutting down the
engine.
@desa
Copy link
Contributor Author

desa commented Mar 29, 2016

@jsternberg A query must have been running. It only happened once, sorry for not including more repro steps.

@e-dard e-dard mentioned this issue Mar 29, 2016
Merged
4 tasks
e-dard added a commit that referenced this issue Mar 29, 2016
@e-dard
Check if engine closed. Fixes #6140
07bd8f8
e-dard added a commit that referenced this issue Mar 31, 2016
@e-dard
Check if engine closed. Fixes #6140
8e2d1e4
@e-dard
Copy link
Contributor

e-dard commented Mar 31, 2016

Should be fixed via #6149 and #6151

@e-dard e-dard reopened this Mar 31, 2016
@e-dard e-dard closed this as completed Mar 31, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
panic
Projects
None yet
Milestone
0.12.0
Development

Successfully merging a pull request may close this issue.

Ensure safe access to Shard engine influxdata/influxdb
4 participants
@jwilder @e-dard @jsternberg @desa