WebAdmin Crashed. InfluxDB does not start. #369
Milestone
Comments
|
Hey @samuraraujo, is it possible for you to zip up your data directory and post it somewhere for us to test with? |
|
Hi Paul, I deleted the data directory to be able to start influxdb again. If it happens again I will send you the zip file. |
|
What version are you running? Can you post a gist to your config file? |
|
$ influxdb -v I did not change anything in the config file from the default ubuntu deb installation. I noticed that the default log config is in "debug" mode. Maybe the default installation should use "error". The current config makes the log files grow very fast with large data. |
|
Yeah, should probably be "info" or "warn". Also, 0.5.0 isn't the latest version. Can you upgrade to the new one? |
|
Updated! |
|
@samuraraujo is this still a current issue, or can it be closed out? |
|
I'm getting a similar error using the following version: The server was rebooted (influxdb process has probably been killed) and afterwards it continuously crashes when I try to start it up. I'm gonna do a backup of the data directory (11gb atm) and try to upgrade to 0.6.0 and check if the problem persists. Logs: |
|
@wladekb can you zip your data and send it to support at influxdb dot com |
|
@jvshahid do you really want me to send you 11GB via email? |
|
I'm only interested in the WAL, especially |
|
@jvshahid I sent WAL and a few other files, did you get it? |
|
Issue is still valid after upgrading to 0.6.0 |
|
@wladekb I didn't get the data, who did you send it to ? |
|
@jvshahid I sent it to support at influxdb dot com with subject "Issue 369 data dir" |
|
How big was the file ? None of us got the email. Is it possible to put on s3 or dropbox. |
|
18mb zipped with password (had to it due to gmail security policy) |
|
Well, we didn't get the email. Can you use google drive, s3, or dropbox. Any of those should be fine with a 18mb file. |
|
Thanks, I have it now. I'll take a look asap and get back to you. |
|
FYI I stripped all non-most-recent data files from archive. If any of them are needed I still have a copy. |
|
@wladekb can you run |
btw. it's a physical machine |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
[2014/03/28 10:44:30 BRT] DEBG recovery requestsSinceLastIndex: 749, requestNumber: 186749
[2014/03/28 10:44:30 BRT] DEBG largestrequestnumber: 186749
panic: Replay error. Stacktrace:
goroutine 17 [running]:
common.NewErrorWithStacktrace(0x7f3a8ac847b0, 0xc210162f40, 0xc210162f50, 0x1, 0x1, ...)
/home/vagrant/influxdb/src/common/error_with_stacktrace.go:22 +0x7f
wal.newErrorReplayRequest(0x7f3a8ac847b0, 0xc210162f40, 0x0)
/home/vagrant/influxdb/src/wal/replay_request.go:19 +0xc0
wal.(_log).replayFromFileLocation(0xc2100ea240, 0xc2100c2008, 0xc210125090, 0xc2100e96e0, 0xc2100e8c60)
/home/vagrant/influxdb/src/wal/log.go:254 +0x7ea
wal.func·002()
/home/vagrant/influxdb/src/wal/log.go:133 +0x25b
created by wal.(_log).dupAndReplayFromOffset
/home/vagrant/influxdb/src/wal/log.go:134 +0x16c
Caused by: proto: required field "{Unknown}" not set
goroutine 1 [running]:
runtime.panic(0x857020, 0xc2101099c0)
/home/vagrant/bin/go/src/pkg/runtime/panic.c:266 +0xb6
wal.(_WAL).SetServerId(0xc210070180, 0xc200000001)
/home/vagrant/influxdb/src/wal/wal.go:112 +0x122
server.(_Server).ListenAndServe(0xc21000b150, 0xc21000b150, 0x0)
/home/vagrant/influxdb/src/server/server.go:89 +0x10d
main.main()
/home/vagrant/influxdb/src/daemon/influxd.go:116 +0x599
goroutine 3 [runnable]:
code.google.com/p/log4go.ConsoleLogWriter.run(0xc2100482c0, 0x7f3a8ac77fe8, 0xc210000008)
/home/vagrant/influxdb/src/code.google.com/p/log4go/termlog.go:31 +0x297
created by code.google.com/p/log4go.NewConsoleLogWriter
/home/vagrant/influxdb/src/code.google.com/p/log4go/termlog.go:19 +0x67
goroutine 4 [syscall]:
os/signal.loop()
/home/vagrant/bin/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
/home/vagrant/bin/go/src/pkg/os/signal/signal_unix.go:27 +0x31
goroutine 5 [runnable]:
code.google.com/p/log4go.func·002()
/home/vagrant/influxdb/src/code.google.com/p/log4go/filelog.go:84 +0x84c
created by code.google.com/p/log4go.NewFileLogWriter
/home/vagrant/influxdb/src/code.google.com/p/log4go/filelog.go:116 +0x2d1
goroutine 6 [syscall]:
runtime.goexit()
/home/vagrant/bin/go/src/pkg/runtime/proc.c:1394
goroutine 7 [chan receive]:
wal.(*WAL).processEntries(0xc210070180)
/home/vagrant/influxdb/src/wal/wal.go:237 +0x3f
created by wal.NewWAL
/home/vagrant/influxdb/src/wal/wal.go:102 +0x9f3
goroutine 8 [sleep]:
time.Sleep(0x8bb2c97000)
/tmp/makerelease886106415/go/src/pkg/runtime/time.goc:31 +0x31
cluster.func·001()
/home/vagrant/influxdb/src/cluster/cluster_configuration.go:129 +0x35
created by cluster.(*ClusterConfiguration).CreateFutureShardsAutomaticallyBeforeTimeComes
/home/vagrant/influxdb/src/cluster/cluster_configuration.go:134 +0x63
goroutine 9 [IO wait]:
net.runtime_pollWait(0x7f3a8ac79f88, 0x72, 0x0)
/tmp/makerelease886106415/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc2100f51b0, 0x72, 0x7f3a8ac78120, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc2100f51b0, 0xb, 0x7f3a8ac78120)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).accept(0xc2100f5150, 0x9f4698, 0x0, 0x7f3a8ac78120, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(_TCPListener).AcceptTCP(0xc210000918, 0xc210036770, 0x0, 0x7f3a8ac79060)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(_TCPListener).Accept(0xc210000918, 0xc210036770, 0x7f3a8aafaf38, 0x1, 0x1)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:243 +0x27
coordinator.(_ProtobufServer).ListenAndServe(0xc21006eb40)
/home/vagrant/influxdb/src/coordinator/protobuf_server.go:62 +0x1c7
created by server.(*Server).ListenAndServe
/home/vagrant/influxdb/src/server/server.go:80 +0x3d
goroutine 10 [IO wait]:
net.runtime_pollWait(0x7f3a8ac7a030, 0x72, 0x0)
/tmp/makerelease886106415/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc21000b220, 0x72, 0x7f3a8ac78120, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc21000b220, 0xb, 0x7f3a8ac78120)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).accept(0xc21000b1c0, 0x9f4698, 0x0, 0x7f3a8ac78120, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(_TCPListener).AcceptTCP(0xc210000848, 0x18, 0xc2100f8810, 0x5d8d03)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(_TCPListener).Accept(0xc210000848, 0x0, 0x0, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(_Server).Serve(0xc2100981e0, 0x7f3a8ac79090, 0xc210000848, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/server.go:1622 +0x91
coordinator.func·008()
/home/vagrant/influxdb/src/coordinator/raft_server.go:500 +0x3a
created by coordinator.(*RaftServer).Serve
/home/vagrant/influxdb/src/coordinator/raft_server.go:504 +0x519
goroutine 13 [syscall]:
github.com/jmhodges/levigo._Cfunc_leveldb_open(0x1ff32b0, 0x1ff43a0, 0xc210111238, 0xc2100a3d60)
github.com/jmhodges/levigo/_obj/_cgo_defun.c:278 +0x31
github.com/jmhodges/levigo.Open(0xc2100a6f00, 0x2b, 0xc210000020, 0x0, 0x0, ...)
/home/vagrant/influxdb/src/github.com/jmhodges/levigo/db.go:79 +0xb6
datastore.(_LevelDbShardDatastore).GetOrCreateShard(0xc2100722a0, 0x1, 0x0, 0x0, 0x0, ...)
/home/vagrant/influxdb/src/datastore/leveldb_shard_datastore.go:123 +0x2d2
cluster.(_ShardData).SetLocalStore(0xc21006bb60, 0x7f3a8ac78e20, 0xc2100722a0, 0x1, 0x0, ...)
/home/vagrant/influxdb/src/cluster/shard.go:163 +0x10e
cluster.(_ClusterConfiguration).AddShards(0xc210078380, 0xc2100a5b00, 0x1, 0x4, 0x0, ...)
/home/vagrant/influxdb/src/cluster/cluster_configuration.go:824 +0x9bc
coordinator.(_CreateShardsCommand).Apply(0xc2100a5a40, 0x7f3a8ac7a240, 0xc2100a0240, 0x7f3a8ac7abe0, 0xc2100a5a40, ...)
/home/vagrant/influxdb/src/coordinator/command.go:242 +0x92
github.com/goraft/raft.func·005(0xc2100467e0, 0x7f3a8ac78650, 0xc2100a5a40, 0x76, 0x76, ...)
/home/vagrant/influxdb/src/github.com/goraft/raft/server.go:204 +0x364
github.com/goraft/raft.(_Log).setCommitIndex(0xc2100f5690, 0x9, 0x0, 0x0)
/home/vagrant/influxdb/src/github.com/goraft/raft/log.go:371 +0x36e
github.com/goraft/raft.(_server).processCommand(0xc2100a0240, 0x7f3a8ac7a690, 0xc2100a3b10, 0xc2100a67b0)
/home/vagrant/influxdb/src/github.com/goraft/raft/server.go:877 +0x47e
github.com/goraft/raft.(_server).leaderLoop(0xc2100a0240)
/home/vagrant/influxdb/src/github.com/goraft/raft/server.go:800 +0x474
github.com/goraft/raft.(_server).loop(0xc2100a0240)
/home/vagrant/influxdb/src/github.com/goraft/raft/server.go:590 +0x33f
created by github.com/goraft/raft.(*server).Start
/home/vagrant/influxdb/src/github.com/goraft/raft/server.go:460 +0x345
goroutine 14 [select]:
coordinator.(_RaftServer).CompactLog(0xc210073000)
/home/vagrant/influxdb/src/coordinator/raft_server.go:300 +0x2ef
created by coordinator.(_RaftServer).startRaft
/home/vagrant/influxdb/src/coordinator/raft_server.go:338 +0x388
goroutine 15 [select]:
coordinator.(_RaftServer).raftLeaderLoop(0xc210073000, 0xc2100a7740)
/home/vagrant/influxdb/src/coordinator/raft_server.go:407 +0x29c
created by coordinator.(_RaftServer).raftEventHandler
/home/vagrant/influxdb/src/coordinator/raft_server.go:396 +0x1d0
goroutine 16 [chan receive]:
github.com/goraft/raft.(_server).Do(0xc2100a0240, 0x7f3a8ac7a690, 0xc2100a3b10, 0x7f3a8ac7a690, 0xc2100a3b10, ...)
/home/vagrant/influxdb/src/github.com/goraft/raft/server.go:852 +0xfd
created by github.com/goraft/raft.(_server).leaderLoop
/home/vagrant/influxdb/src/github.com/goraft/raft/server.go:782 +0x220
goroutine 18 [finalizer wait]:
runtime.park(0x44fbf0, 0x1018da8, 0x10037c8)
/home/vagrant/bin/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
/home/vagrant/bin/go/src/pkg/runtime/mgc0.c:2279 +0x84
runtime.goexit()
/home/vagrant/bin/go/src/pkg/runtime/proc.c:1394
The text was updated successfully, but these errors were encountered: