[0.9.4.2] panic with index out of range

[poelzi]

I get following error in 0.9.4.2 without much information about the underlaying cause, nor which query causes the problem.

http] 2015/11/10 13:34:08 127.0.0.1 - root [10/Nov/2015:13:34:08 +0000] GET /query?db=leads_cluster&epoch=ms&q=SELECT+sum%28%22value%22%29%2F53687091200+%2B+12+%2A+0.14+AS+%22value%22+FROM+%22iptables_value%22+WHERE+%22type_instance%22+%3D+%27to_outside_microcloud%27+AND+%22type%22+%3D+%27ipt_bytes%27+AND+%22instance%22+%3D+%27filter-LEADS_ACCOUNT_OUT%27+AND+time+%3E+1447039444s+and+time+%3C+now%28%29+GROUP+BY+time%281h%29 HTTP/1.1 200 286 http://80.156.222.79:8080/dashboard/db/leads-cluster Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.7 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.7 b8674ef4-87af-11e5-828c-000000000000 90.894353ms
2015/11/10 13:34:08 http: panic serving 127.0.0.1:40862: runtime error: index out of range
goroutine 2852 [running]:
net/http.(*conn).serve.func1(0xc8200c4580, 0x7f3a1f952098, 0xc821d47a78)
        /home/philip/.gvm/gos/go1.5/src/net/http/server.go:1287 +0xb5
github.com/influxdb/influxdb/services/httpd.(*Handler).serveQuery(0xc820085400, 0x7f3a1f952328, 0xc821c7a580, 0xc82008b6c0, 0x0)
        /tmp/build/src/github.com/influxdb/influxdb/services/httpd/handler.go:316 +0x16ef
github.com/influxdb/influxdb/services/httpd.(*Handler).(github.com/influxdb/influxdb/services/httpd.serveQuery)-fm(0x7f3a1f952328, 0xc821c7a580, 0xc82008b6c0, 0x0)
        /tmp/build/src/github.com/influxdb/influxdb/services/httpd/handler.go:98 +0x48
github.com/influxdb/influxdb/services/httpd.authenticate.func1(0x7f3a1f952328, 0xc821c7a580, 0xc82008b6c0)
        /tmp/build/src/github.com/influxdb/influxdb/services/httpd/handler.go:673 +0x70
net/http.HandlerFunc.ServeHTTP(0xc8201360c0, 0x7f3a1f952328, 0xc821c7a580, 0xc82008b6c0)
        /home/philip/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.gzipFilter.func1(0x7f3a1f952250, 0xc821c7a540, 0xc82008b6c0)
        /tmp/build/src/github.com/influxdb/influxdb/services/httpd/handler.go:734 +0x2af
net/http.HandlerFunc.ServeHTTP(0xc8201360e0, 0x7f3a1f952250, 0xc821c7a540, 0xc82008b6c0)
        /home/philip/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.versionHeader.func1(0x7f3a1f952250, 0xc821c7a540, 0xc82008b6c0)
        /tmp/build/src/github.com/influxdb/influxdb/services/httpd/handler.go:743 +0xc0
net/http.HandlerFunc.ServeHTTP(0xc820136100, 0x7f3a1f952250, 0xc821c7a540, 0xc82008b6c0)
        /home/philip/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.cors.func1(0x7f3a1f952250, 0xc821c7a540, 0xc82008b6c0)
        /tmp/build/src/github.com/influxdb/influxdb/services/httpd/handler.go:776 +0x328
net/http.HandlerFunc.ServeHTTP(0xc820136120, 0x7f3a1f952250, 0xc821c7a540, 0xc82008b6c0)
        /home/philip/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.requestID.func1(0x7f3a1f952250, 0xc821c7a540, 0xc82008b6c0)
        /tmp/build/src/github.com/influxdb/influxdb/services/httpd/handler.go:786 +0x15d
net/http.HandlerFunc.ServeHTTP(0xc820136140, 0x7f3a1f952250, 0xc821c7a540, 0xc82008b6c0)
        /home/philip/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.logging.func1(0x7f3a1f952250, 0xc821c7a520, 0xc82008b6c0)
        /tmp/build/src/github.com/influxdb/influxdb/services/httpd/handler.go:794 +0x10b
net/http.HandlerFunc.ServeHTTP(0xc820136160, 0x7f3a1f952250, 0xc821c7a520, 0xc82008b6c0)
        /home/philip/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/influxdb/influxdb/services/httpd.recovery.func1(0x7f3a1f952218, 0xc822e82420, 0xc82008b6c0)
        /tmp/build/src/github.com/influxdb/influxdb/services/httpd/handler.go:804 +0x114
net/http.HandlerFunc.ServeHTTP(0xc820136180, 0x7f3a1f952218, 0xc822e82420, 0xc82008b6c0)
        /home/philip/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc820020118, 0x7f3a1f952218, 0xc822e82420, 0xc82008b6c0)
        /tmp/build/src/github.com/bmizerany/pat/mux.go:109 +0x244
github.com/influxdb/influxdb/services/httpd.(*Handler).ServeHTTP(0xc820085400, 0x7f3a1f952218, 0xc822e82420, 0xc82008b6c0)
        /tmp/build/src/github.com/influxdb/influxdb/services/httpd/handler.go:172 +0x357
net/http.serverHandler.ServeHTTP(0xc8201da240, 0x7f3a1f952218, 0xc822e82420, 0xc82008b6c0)
        /home/philip/.gvm/gos/go1.5/src/net/http/server.go:1862 +0x19e
net/http.(*conn).serve(0xc8200c4580)
        /home/philip/.gvm/gos/go1.5/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.5/src/net/http/server.go:1910 +0x3f6
[query] 2015/11/10 13:34:08 SELECT mean(value) / 100.000 AS "value" FROM "leads_cluster"."default".cpu_value WHERE type_instance = 'system' AND host = 'leads-qe8' AND time > 1447039444s AND time < now() GROUP BY time(1s)
[wal] 2015/11/10 13:34:08 Flush due to idle. Flushing 12 series with 12 points and 582 bytes from partition 1

[rossmcdonald]

@poelzi Are you finding this panic occurring multiple times? Or has it only happened once?

[poelzi]

I'm not so sure currently, it could be that it was due to memory pressure problems. I will looking into this issue further when I have some spare time

[otoolep]

I think the PR @oiooj opened would probably fix this issue. But I am surprised it happened, and need to understand better how the Series slice was actually of zero length.

@poelzi -- can you try upgrading to 0.9.5-rc2?

[oiooj]

@otoolep If nothing comes through it will return an empty result. It seems right.
https://github.com/influxdb/influxdb/blob/master/tsdb/query_executor.go#L752

...

// Stream results from the channel. We should send an empty result if nothing comes through.

...
results <- &influxql.Result{StatementID: statementID, Series: make([]*models.Row, 0)}

...

[otoolep]

It's not quite as simple as that. That line executes for any query that return no results, and a query that doesn't return results does not result in a panic.

./influx 
Connected to http://localhost:8086 version 0.9
InfluxDB shell 0.9
> use _internal
Using database _internal
> select * from HeapAlloc
> 

(no panic)