Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Percentile aggregator can panic when there aren't enough values #405

Closed
sontags opened this issue Apr 4, 2014 · 7 comments
Closed

Percentile aggregator can panic when there aren't enough values #405

sontags opened this issue Apr 4, 2014 · 7 comments
Milestone
0.5.5

Comments

@sontags
Copy link

sontags commented Apr 4, 2014

Using RHEL6.4 and InfluxDB v0.5.3 (git: 767ba83) (installed via prebuilt RPM). Starting influxdb via /etc/init.d/influxdb start does not give me any error, but influxdb dies during startup. Here's an extract of the log file

...
[2014/04/04 05:45:00 CEST] [INFO] (wal.(*WAL).rotateTheLogFile:501) Rotating log. New log file /opt/influxdb/shared/data/wal/log.270002
[2014/04/04 05:45:00 CEST] [INFO] (wal.(*WAL).processCommitEntry:332) Deleting /opt/influxdb/shared/data/wal/log.260002
[2014/04/04 09:14:51 CEST] [INFO] (main.setupLogging:52) Redirectoring logging to /opt/influxdb/shared/log.txt
[04/04/14 09:14:51] [INFO] Redirectoring logging to /opt/influxdb/shared/log.txt
[04/04/14 09:14:51] [INFO] Starting Influx Server bound to 0.0.0.0 ...
[2014/04/04 09:14:51 CEST] [INFO] (main.main:81) Starting Influx Server bound to 0.0.0.0 ...
[04/04/14 09:14:51] [INFO] 
+---------------------------------------------+
|  _____        __ _            _____  ____   |
| |_   _|      / _| |          |  __ \|  _ \  |
|   | |  _ __ | |_| |_   ___  _| |  | | |_) | |
|   | | | '_ \|  _| | | | \ \/ / |  | |  _ <  |
|  _| |_| | | | | | | |_| |>  <| |__| | |_) | |
| |_____|_| |_|_| |_|\__,_/_/\_\_____/|____/  |
+---------------------------------------------+


[04/04/14 09:14:51] [INFO] Opening database at /opt/influxdb/shared/data/db
[2014/04/04 09:14:51 CEST] [INFO] (main.main:93) 
+---------------------------------------------+
|  _____        __ _            _____  ____   |
| |_   _|      / _| |          |  __ \|  _ \  |
|   | |  _ __ | |_| |_   ___  _| |  | | |_) | |
|   | | | '_ \|  _| | | | \ \/ / |  | |  _ <  |
|  _| |_| | | | | | | |_| |>  <| |__| | |_) | |
| |_____|_| |_|_| |_|\__,_/_/\_\_____/|____/  |
+---------------------------------------------+


[2014/04/04 09:14:51 CEST] [INFO] (server.NewServer:33) Opening database at /opt/influxdb/shared/data/db
[04/04/14 09:14:51] [INFO] Opening wal in /opt/influxdb/shared/data/wal
[2014/04/04 09:14:51 CEST] [INFO] (wal.NewWAL:40) Opening wal in /opt/influxdb/shared/data/wal
[04/04/14 09:14:51] [INFO] Opening log file /opt/influxdb/shared/data/wal/log.270002
[04/04/14 09:14:51] [INFO] Opening index file /opt/influxdb/shared/data/wal/index.270002
[2014/04/04 09:14:51 CEST] [INFO] (wal.(*WAL).openLog:358) Opening log file /opt/influxdb/shared/data/wal/log.270002
[2014/04/04 09:14:51 CEST] [INFO] (wal.(*WAL).openLog:372) Opening index file /opt/influxdb/shared/data/wal/index.270002
[04/04/14 09:14:51] [INFO] Ssl will be disabled since the ssl port or certificate path weren't set
[2014/04/04 09:14:51 CEST] [INFO] (api/http.(*HttpServer).EnableSsl:55) Ssl will be disabled since the ssl port or certificate path weren't set
[04/04/14 09:14:51] [INFO] Initializing Raft HTTP server
[2014/04/04 09:14:51 CEST] [INFO] (coordinator.(*RaftServer).Serve:486) Initializing Raft HTTP server
[04/04/14 09:14:51] [INFO] Raft Server Listening at http://menetd:8090
[04/04/14 09:14:51] [INFO] ProtobufServer listening on 0.0.0.0:8099
[2014/04/04 09:14:51 CEST] [INFO] (coordinator.(*ProtobufServer).ListenAndServe:60) ProtobufServer listening on 0.0.0.0:8099
[2014/04/04 09:14:51 CEST] [INFO] (coordinator.(*RaftServer).Serve:497) Raft Server Listening at http://menetd:8090
[04/04/14 09:14:51] [INFO] Initializing Raft Server: /opt/influxdb/shared/data/raft 8090
[2014/04/04 09:14:51 CEST] [INFO] (coordinator.(*RaftServer).startRaft:320) Initializing Raft Server: /opt/influxdb/shared/data/raft 8090
[04/04/14 09:14:51] [INFO] Recovered from log
[2014/04/04 09:14:51 CEST] [INFO] (coordinator.(*RaftServer).startRaft:341) Recovered from log
[04/04/14 09:14:51] [INFO] Waiting for local server to be added
[2014/04/04 09:14:51 CEST] [INFO] (server.(*Server).ListenAndServe:85) Waiting for local server to be added
[04/04/14 09:14:53] [INFO] (raft:ed1914b) Selected as leader. Starting leader loop.
[2014/04/04 09:14:53 CEST] [INFO] (coordinator.(*RaftServer).raftEventHandler:395) (raft:ed1914b) Selected as leader. Starting leader loop.
[04/04/14 09:14:53] [INFO] Added server to cluster config: 1, http://menetd:8090, menetd:8099
[04/04/14 09:14:53] [INFO] Checking whether this is the local server new: menetd:8099, local: menetd:8099

[04/04/14 09:14:53] [INFO] Added the local server
[2014/04/04 09:14:53 CEST] [INFO] (cluster.(*ClusterConfiguration).AddPotentialServer:207) Added server to cluster config: 1, http://menetd:8090, menetd:8099
[2014/04/04 09:14:53 CEST] [INFO] (cluster.(*ClusterConfiguration).AddPotentialServer:208) Checking whether this is the local server new: menetd:8099, local: menetd:8099

[2014/04/04 09:14:53 CEST] [INFO] (cluster.(*ClusterConfiguration).AddPotentialServer:218) Added the local server
[04/04/14 09:14:53] [INFO] Setting server id to 1 and recovering
[2014/04/04 09:14:53 CEST] [INFO] (wal.(*WAL).SetServerId:109) Setting server id to 1 and recovering
[04/04/14 09:14:53] [INFO] Checking /opt/influxdb/shared/data/wal/log.270002, last: 713669, size: 801262
[2014/04/04 09:14:53 CEST] [INFO] (wal.(*WAL).recover:429) Checking /opt/influxdb/shared/data/wal/log.270002, last: 713669, size: 801262
[04/04/14 09:14:53] [INFO] DATASTORE: opening or creating shard /opt/influxdb/shared/data/db/shard_db/00001
[2014/04/04 09:14:53 CEST] [INFO] (datastore.(*LevelDbShardDatastore).GetOrCreateShard:122) DATASTORE: opening or creating shard /opt/influxdb/shared/data/db/shard_db/00001
[04/04/14 09:14:53] [INFO] Adding short term shard: 1 - start: Thu Apr 3 02:00:00 +0200 CEST 2014 (1396483200). end: Thu Apr 10 02:00:00 +0200 CEST 2014 (1397088000). isLocal: %!d(bool=true). servers: [%!s(uint32=1)]
[2014/04/04 09:14:53 CEST] [INFO] (cluster.(*ClusterConfiguration).AddShards:871) Adding short term shard: 1 - start: Thu Apr 3 02:00:00 +0200 CEST 2014 (1396483200). end: Thu Apr 10 02:00:00 +0200 CEST 2014 (1397088000). isLocal: %!d(bool=true). servers: [%!s(uint32=1)]
[04/04/14 09:14:53] [INFO] DATASTORE: opening or creating shard /opt/influxdb/shared/data/db/shard_db/00002
[2014/04/04 09:14:53 CEST] [INFO] (datastore.(*LevelDbShardDatastore).GetOrCreateShard:122) DATASTORE: opening or creating shard /opt/influxdb/shared/data/db/shard_db/00002
[04/04/14 09:14:54] [INFO] Adding long term shard: 2 - start: Tue Mar 11 01:00:00 +0100 CET 2014 (1394496000). end: Thu Apr 10 02:00:00 +0200 CEST 2014 (1397088000). isLocal: %!d(bool=true). servers: [%!s(uint32=1)]
[2014/04/04 09:14:54 CEST] [INFO] (cluster.(*ClusterConfiguration).AddShards:871) Adding long term shard: 2 - start: Tue Mar 11 01:00:00 +0100 CET 2014 (1394496000). end: Thu Apr 10 02:00:00 +0200 CEST 2014 (1397088000). isLocal: %!d(bool=true). servers: [%!s(uint32=1)]
********************************BUG********************************
Database: perf
Query: [select MIN(value) As min from /raw\..*/ group by time(5m) where time > 1396584299999999u and time < 1396595400000000u]
Error: runtime error: invalid memory address or nil pointer dereference. Stacktrace: goroutine 16 [running]:
coordinator.recoverFunc(0xc21012ae98, 0x4, 0xc2100e6a00, 0x75)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:344 +0x106
runtime.panic(0x8848e0, 0x1003e48)
    /home/vagrant/bin/go/src/pkg/runtime/panic.c:248 +0x106
cluster.(*WriteBuffer).Write(0x0, 0xc210249e70)
    /home/vagrant/influxdb/src/cluster/write_buffer.go:45 +0x1e
datastore.(*LevelDbShardDatastore).BufferWrite(0xc210085310, 0xc210249e70)
    /home/vagrant/influxdb/src/datastore/leveldb_shard_datastore.go:166 +0x35
cluster.(*ShardData).Write(0xc21014e8f0, 0xc210249e70, 0x7f44ea6e9b88, 0x0)
    /home/vagrant/influxdb/src/cluster/shard.go:186 +0xf8
coordinator.(*CoordinatorImpl).write(0xc210096060, 0xc21012ae98, 0x4, 0xc21024e190, 0x7f44ea6e9b88, ...)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:531 +0xa0
coordinator.(*CoordinatorImpl).CommitSeriesData(0xc210096060, 0xc21012ae98, 0x4, 0xc21024e190, 0x4f62f9f19b300, ...)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:516 +0x43f
coordinator.(*CoordinatorImpl).Inter
********************************BUG********************************
Database: perf
Query: [select MAX(value) As max from /raw\..*/ group by time(5m) where time > 1396584299999999u and time < 1396595400000000u]
Error: runtime error: invalid memory address or nil pointer dereference. Stacktrace: goroutine 16 [running]:
coordinator.recoverFunc(0xc21012ae98, 0x4, 0xc210230a80, 0x75)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:344 +0x106
runtime.panic(0x8848e0, 0x1003e48)
    /home/vagrant/bin/go/src/pkg/runtime/panic.c:248 +0x106
cluster.(*WriteBuffer).Write(0x0, 0xc210121cb0)
    /home/vagrant/influxdb/src/cluster/write_buffer.go:45 +0x1e
datastore.(*LevelDbShardDatastore).BufferWrite(0xc210085310, 0xc210121cb0)
    /home/vagrant/influxdb/src/datastore/leveldb_shard_datastore.go:166 +0x35
cluster.(*ShardData).Write(0xc21014e8f0, 0xc210121cb0, 0xc210000ad8, 0x1)
    /home/vagrant/influxdb/src/cluster/shard.go:186 +0xf8
coordinator.(*CoordinatorImpl).write(0xc210096060, 0xc21012ae98, 0x4, 0xc2100a4a50, 0x7f44ea6e9b88, ...)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:531 +0xa0
coordinator.(*CoordinatorImpl).CommitSeriesData(0xc210096060, 0xc21012ae98, 0x4, 0xc2100a4a50, 0x4f62f9f19b300, ...)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:516 +0x43f
coordinator.(*CoordinatorImpl).Interpo
********************************BUG********************************
Database: perf
Query: [select MEDIAN(value) As median from /raw\..*/ group by time(5m) where time > 1396584299999999u and time < 1396595400000000u]
Error: runtime error: invalid memory address or nil pointer dereference. Stacktrace: goroutine 16 [running]:
coordinator.recoverFunc(0xc21012ae98, 0x4, 0xc210200480, 0x7b)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:344 +0x106
runtime.panic(0x8848e0, 0x1003e48)
    /home/vagrant/bin/go/src/pkg/runtime/panic.c:248 +0x106
cluster.(*WriteBuffer).Write(0x0, 0xc2101a6380)
    /home/vagrant/influxdb/src/cluster/write_buffer.go:45 +0x1e
datastore.(*LevelDbShardDatastore).BufferWrite(0xc210085310, 0xc2101a6380)
    /home/vagrant/influxdb/src/datastore/leveldb_shard_datastore.go:166 +0x35
cluster.(*ShardData).Write(0xc21014e8f0, 0xc2101a6380, 0xc21011e1d0, 0x1)
    /home/vagrant/influxdb/src/cluster/shard.go:186 +0xf8
coordinator.(*CoordinatorImpl).write(0xc210096060, 0xc21012ae98, 0x4, 0xc21016f0a0, 0x7f44ea6e9b88, ...)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:531 +0xa0
coordinator.(*CoordinatorImpl).CommitSeriesData(0xc210096060, 0xc21012ae98, 0x4, 0xc21016f0a0, 0x4f62f9f19b300, ...)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:516 +0x43f
coordinator.(*CoordinatorImpl).Interpo
********************************BUG********************************
Database: perf
Query: [select PERCENTILE(value, 75) As perc75 from /raw\..*/ group by time(5m) where time > 1396584299999999u and time < 1396595400000000u]
Error: runtime error: invalid memory address or nil pointer dereference. Stacktrace: goroutine 16 [running]:
coordinator.recoverFunc(0xc21012ae98, 0x4, 0xc2100dae10, 0x83)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:344 +0x106
runtime.panic(0x8848e0, 0x1003e48)
    /home/vagrant/bin/go/src/pkg/runtime/panic.c:248 +0x106
cluster.(*WriteBuffer).Write(0x0, 0xc2101a6690)
    /home/vagrant/influxdb/src/cluster/write_buffer.go:45 +0x1e
datastore.(*LevelDbShardDatastore).BufferWrite(0xc210085310, 0xc2101a6690)
    /home/vagrant/influxdb/src/datastore/leveldb_shard_datastore.go:166 +0x35
cluster.(*ShardData).Write(0xc21014e8f0, 0xc2101a6690, 0xc2101a87f8, 0x1)
    /home/vagrant/influxdb/src/cluster/shard.go:186 +0xf8
coordinator.(*CoordinatorImpl).write(0xc210096060, 0xc21012ae98, 0x4, 0xc21016f140, 0x7f44ea6e9b88, ...)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:531 +0xa0
coordinator.(*CoordinatorImpl).CommitSeriesData(0xc210096060, 0xc21012ae98, 0x4, 0xc21016f140, 0x4f62f9f19b300, ...)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:516 +0x43f
coordinator.(*CoordinatorImpl).Interpo
panic: runtime error: index out of range

goroutine 28 [running]:
runtime.panic(0x8848e0, 0x1004037)
    /home/vagrant/bin/go/src/pkg/runtime/panic.c:266 +0xb6
engine.(*PercentileAggregator).GetValues(0xc2102cbaf0, 0xc210284875, 0x20, 0x8d0c00, 0xc21021cc20, ...)
    /home/vagrant/influxdb/src/engine/aggregator.go:818 +0x47d
engine.(*QueryEngine).runAggregates(0xc210139cc0)
    /home/vagrant/influxdb/src/engine/engine.go:559 +0x992
engine.(*QueryEngine).Close(0xc210139cc0)
    /home/vagrant/influxdb/src/engine/engine.go:193 +0x1bd
engine.(*FilteringEngine).Close(0xc21010adc0)
    /home/vagrant/influxdb/src/engine/filtering_engine.go:44 +0x30
cluster.(*ShardData).Query(0xc21013c340, 0xc2100ddc60, 0xc2102bb000)
    /home/vagrant/influxdb/src/cluster/shard.go:251 +0x46c
created by coordinator.(*CoordinatorImpl).runQuerySpec
    /home/vagrant/influxdb/src/coordinator/coordinator.go:288 +0x1c5

goroutine 1 [sleep]:
time.Sleep(0x12a05f200)
    /tmp/makerelease886106415/go/src/pkg/runtime/time.goc:31 +0x31
server.(*Server).ListenAndServe(0xc2100859a0, 0xc2100859a0, 0x0)
    /home/vagrant/influxdb/src/server/server.go:89 +0x120
main.main()
    /home/vagrant/influxdb/src/daemon/influxd.go:119 +0x5fe

goroutine 3 [syscall]:
os/signal.loop()
    /home/vagrant/bin/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
    /home/vagrant/bin/go/src/pkg/os/signal/signal_unix.go:27 +0x31

goroutine 4 [chan receive]:
code.google.com/p/log4go.ConsoleLogWriter.run(0xc21006f000, 0x7f44ea6dc0e8, 0xc210000008)
    /home/vagrant/influxdb/src/code.google.com/p/log4go/termlog.go:27 +0x60
created by code.google.com/p/log4go.NewConsoleLogWriter
    /home/vagrant/influxdb/src/code.google.com/p/log4go/termlog.go:19 +0x67

goroutine 5 [select]:
code.google.com/p/log4go.func·002()
    /home/vagrant/influxdb/src/code.google.com/p/log4go/filelog.go:84 +0x84c
created by code.google.com/p/log4go.NewFileLogWriter
    /home/vagrant/influxdb/src/code.google.com/p/log4go/filelog.go:116 +0x2d1

goroutine 6 [syscall]:
runtime.goexit()
    /home/vagrant/bin/go/src/pkg/runtime/proc.c:1394

goroutine 7 [chan receive]:
wal.(*WAL).processEntries(0xc210079d80)
    /home/vagrant/influxdb/src/wal/wal.go:242 +0x3f
created by wal.NewWAL
    /home/vagrant/influxdb/src/wal/wal.go:103 +0x9f3

goroutine 8 [sleep]:
time.Sleep(0x8bb2c97000)
    /tmp/makerelease886106415/go/src/pkg/runtime/time.goc:31 +0x31
cluster.func·001()
    /home/vagrant/influxdb/src/cluster/cluster_configuration.go:131 +0x35
created by cluster.(*ClusterConfiguration).CreateFutureShardsAutomaticallyBeforeTimeComes
    /home/vagrant/influxdb/src/cluster/cluster_configuration.go:136 +0x63

goroutine 9 [chan receive]:
main.waitForSignals(0x7f44ea6dd018, 0xc2100859a0)
    /home/vagrant/influxdb/src/daemon/null_profiler.go:23 +0x126
created by main.startProfiler
    /home/vagrant/influxdb/src/daemon/null_profiler.go:15 +0x38

goroutine 11 [IO wait]:
net.runtime_pollWait(0x7f44ea6ddfb8, 0x72, 0x0)
    /tmp/makerelease886106415/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc2100580d0, 0x72, 0x7f44ea6dbf88, 0xb)
    /home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc2100580d0, 0xb, 0x7f44ea6dbf88)
    /home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).accept(0xc210058070, 0x9f6030, 0x0, 0x7f44ea6dbf88, 0xb)
    /home/vagrant/bin/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(*TCPListener).AcceptTCP(0xc2100f6018, 0xc2100d9020, 0x0, 0x7f44ea6dd068)
    /home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(*TCPListener).Accept(0xc2100f6018, 0xc2100d9020, 0x7f44ea54cf38, 0x1, 0x1)
    /home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:243 +0x27
coordinator.(*ProtobufServer).ListenAndServe(0xc2100a1d00)
    /home/vagrant/influxdb/src/coordinator/protobuf_server.go:62 +0x1c7
created by server.(*Server).ListenAndServe
    /home/vagrant/influxdb/src/server/server.go:78 +0x3d

goroutine 12 [IO wait]:
net.runtime_pollWait(0x7f44ea6de060, 0x72, 0x0)
    /tmp/makerelease886106415/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc210085a70, 0x72, 0x7f44ea6dbf88, 0xb)
    /home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc210085a70, 0xb, 0x7f44ea6dbf88)
    /home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).accept(0xc210085a10, 0x9f6030, 0x0, 0x7f44ea6dbf88, 0xb)
    /home/vagrant/bin/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(*TCPListener).AcceptTCP(0xc2100007b8, 0x18, 0xc210102010, 0x5c7bc3)
    /home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(*TCPListener).Accept(0xc2100007b8, 0x0, 0x0, 0x0, 0x0)
    /home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(*Server).Serve(0xc2100a4640, 0x7f44ea6dd098, 0xc2100007b8, 0x0, 0x0)
    /home/vagrant/bin/go/src/pkg/net/http/server.go:1622 +0x91
coordinator.func·008()
    /home/vagrant/influxdb/src/coordinator/raft_server.go:500 +0x3a
created by coordinator.(*RaftServer).Serve
    /home/vagrant/influxdb/src/coordinator/raft_server.go:504 +0x519

goroutine 14 [select]:
github.com/goraft/raft.(*server).leaderLoop(0xc2100be360)
    /home/vagrant/influxdb/src/github.com/goraft/raft/server.go:765 +0x5fe
github.com/goraft/raft.(*server).loop(0xc2100be360)
    /home/vagrant/influxdb/src/github.com/goraft/raft/server.go:568 +0x33f
created by github.com/goraft/raft.(*server).Start
    /home/vagrant/influxdb/src/github.com/goraft/raft/server.go:472 +0x7af

goroutine 15 [select]:
coordinator.(*RaftServer).CompactLog(0xc21007a300)
    /home/vagrant/influxdb/src/coordinator/raft_server.go:300 +0x2ef
created by coordinator.(*RaftServer).startRaft
    /home/vagrant/influxdb/src/coordinator/raft_server.go:338 +0x38f

goroutine 16 [chan receive]:
coordinator.(*CoordinatorImpl).runQuerySpec(0xc210096060, 0xc2100ddc60, 0x7f44ea6e8d68, 0xc2101a52a0, 0x7f44ea6d35e0, ...)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:295 +0x42f
coordinator.(*CoordinatorImpl).runQuery(0xc210096060, 0xc210233240, 0x7f44ea6e8d10, 0xc2100bc400, 0xc21012ae98, ...)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:148 +0xdb
coordinator.(*CoordinatorImpl).RunQuery(0xc210096060, 0x7f44ea6e8d10, 0xc2100bc400, 0xc21012ae98, 0x4, ...)
    /home/vagrant/influxdb/src/coordinator/coordinator.go:139 +0x707
coordinator.(*RaftServer).runContinuousQuery(0xc21007a300, 0xc21012ae98, 0x4, 0xc2100dabd0, 0xecad0266c, ...)
    /home/vagrant/influxdb/src/coordinator/raft_server.go:471 +0x284
coordinator.(*RaftServer).checkContinuousQueries(0xc21007a300)
    /home/vagrant/influxdb/src/coordinator/raft_server.go:447 +0x4ae
coordinator.(*RaftServer).raftLeaderLoop(0xc21007a300, 0xc210136140)
    /home/vagrant/influxdb/src/coordinator/raft_server.go:410 +0x156
created by coordinator.(*RaftServer).raftEventHandler
    /home/vagrant/influxdb/src/coordinator/raft_server.go:396 +0x1d0

goroutine 19 [finalizer wait]:
runtime.park(0x450310, 0x101af68, 0x1005988)
    /home/vagrant/bin/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
    /home/vagrant/bin/go/src/pkg/runtime/mgc0.c:2279 +0x84
runtime.goexit()
    /home/vagrant/bin/go/src/pkg/runtime/proc.c:1394
@nichdiekuh
Copy link

Guess I hit the same bug (or very similar) after adding a bunch of continuous queries. The database keeps crashing during start:

File: log.txt           Line 402436 Col 0  32277880 bytes                                                                                                                                         99%
[2014/04/04 11:53:12 CEST] [INFO] (wal.(*WAL).recover:429) Checking /tmp/influxdb/development/wal/log.670001, last: 0, size: 827397
[04/04/14 11:53:13] [INFO] (raft:10f3ef2) Selected as leader. Starting leader loop.
[2014/04/04 11:53:13 CEST] [INFO] (coordinator.(*RaftServer).raftEventHandler:395) (raft:10f3ef2) Selected as leader. Starting leader loop.
[04/04/14 11:53:14] [INFO] BUFFER SIZE: %!(EXTRA int=1000)
[2014/04/04 11:53:14 CEST] [INFO] (cluster.(*ShardData).QueryResponseBufferSize:364) BUFFER SIZE: %!(EXTRA int=1000)
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x28 pc=0x51998e]

goroutine 21 [running]:
runtime.panic(0x8851a0, 0x1005728)
        /home/vagrant/bin/go/src/pkg/runtime/panic.c:266 +0xb6
cluster.(*WriteBuffer).Write(0x0, 0xc210f01000)
        /home/vagrant/influxdb/src/cluster/write_buffer.go:45 +0x1e
datastore.(*LevelDbShardDatastore).BufferWrite(0xc210072000, 0xc210f01000)
        /home/vagrant/influxdb/src/datastore/leveldb_shard_datastore.go:166 +0x35
cluster.(*ShardData).Write(0xc21003d7e0, 0xc210f01000, 0x7f3d52296b40, 0x0)
        /home/vagrant/influxdb/src/cluster/shard.go:189 +0xf8
coordinator.(*CoordinatorImpl).write(0xc210096960, 0xc2114161b0, 0x8, 0xc2100db6e0, 0x7f3d52296b40, ...)
        /home/vagrant/influxdb/src/coordinator/coordinator.go:584 +0xa0
coordinator.(*CoordinatorImpl).CommitSeriesData(0xc210096960, 0xc2114161b0, 0x8, 0xc2100db6e0, 0x4f6345f08ff00, ...)
        /home/vagrant/influxdb/src/coordinator/coordinator.go:569 +0x43f
coordinator.(*CoordinatorImpl).InterpolateValuesAndCommit(0xc210096960, 0xc2114161b0, 0x8, 0xc210d1d640, 0xc210097d20, ...)
        /home/vagrant/influxdb/src/coordinator/coordinator.go:513 +0x899
coordinator.func·006(0xc210d1d640, 0x0, 0x0)
        /home/vagrant/influxdb/src/coordinator/raft_server.go:467 +0x60
coordinator.(*ContinuousQueryWriter).Write(0xc211424e20, 0xc210d1d640, 0xc210b1d0e0, 0xc2100f4210)
        /home/vagrant/influxdb/src/coordinator/continuous_query_writer.go:18 +0x30
coordinator.func·001()
        /home/vagrant/influxdb/src/coordinator/coordinator.go:299 +0xfa
created by coordinator.(*CoordinatorImpl).getShardsAndProcessor
        /home/vagrant/influxdb/src/coordinator/coordinator.go:303 +0x36b

goroutine 1 [sleep]:
time.Sleep(0x12a05f200)
        /tmp/makerelease886106415/go/src/pkg/runtime/time.goc:31 +0x31
server.(*Server).ListenAndServe(0xc210051700, 0xc210051700, 0x0)
        /home/vagrant/influxdb/src/server/server.go:89 +0x120
main.main()
        /home/vagrant/influxdb/src/daemon/influxd.go:119 +0x5fe

goroutine 3 [syscall]:
os/signal.loop()
        /home/vagrant/bin/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
        /home/vagrant/bin/go/src/pkg/os/signal/signal_unix.go:27 +0x31

goroutine 4 [chan receive]:
code.google.com/p/log4go.ConsoleLogWriter.run(0xc21006b000, 0x7f3d64a9b200, 0xc210000008)
        /home/vagrant/influxdb/src/code.google.com/p/log4go/termlog.go:27 +0x60

@sontags
Copy link
Author

sontags commented Apr 4, 2014

Same here, most likely adding continuous queries caused the issue. Straight after adding them, they worked, just in case this is important.

@jvshahid jvshahid added this to the 0.5.5 milestone Apr 4, 2014
@jvshahid jvshahid self-assigned this Apr 4, 2014
@jvshahid jvshahid added the bug label Apr 4, 2014
@jvshahid
Copy link
Contributor

jvshahid commented Apr 4, 2014

Thanks for reporting this bug. I'll try to get a fix for it in the next release which will probably go out some time today

@jvshahid
Copy link
Contributor

jvshahid commented Apr 4, 2014

@nichdiekuh what is the version of influxdb generating this stack trace ?

@nichdiekuh
Copy link

0.5.4 on CentOS 6.4

I added a lot of continuous queries and influx run about 10-15min creating the downsampled series until it suddenly crashed.

@jvshahid
Copy link
Contributor

jvshahid commented Apr 4, 2014

@nichdiekuh can you attach the full log to the issue or send it to our support email address.

@jvshahid jvshahid mentioned this issue Apr 4, 2014
Closed
@jvshahid
Copy link
Contributor

jvshahid commented Apr 4, 2014

@nichdiekuh I think I figured out the bug in your case. which is related to the initialization process. I opened a new issue #407 to track this bug

@jvshahid jvshahid changed the title Cannot start InfluxDB Percentile aggregator can panic when there aren't enough values Apr 4, 2014
jvshahid added a commit that referenced this issue Apr 4, 2014
@jvshahid
fix #405. Start processing continuous queries after initializing the WAL
c49f110
jvshahid added a commit that referenced this issue Apr 4, 2014
@jvshahid
Revert "fix #405. Start processing continuous queries after initializ…
5a230b9 
…ing the WAL"

This reverts commit c49f110.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.5.5
Development

No branches or pull requests
3 participants
@jvshahid @nichdiekuh @sontags