Influxdb stop fails with runtime error if configured graphite DB does not exist

[aheusingfeld]

It seems that a new "feature" in 0.9.3 is that InfluxDB refuses to start if the configured database for the graphite plugin does not exist. Apart from complicating the initialization process e.g. of a docker based deployment (see https://github.com/tutumcloud/influxdb/tree/master/0.9), the error handling in this scenario looks a bit strange so I thought I'd let you know.

I tested with the following config.toml locally and within the docker container:

reporting-disabled = false
[meta]
  dir = "/root/.influxdb/meta"
  hostname = "localhost"
  bind-address = ":8088"
  retention-autocreate = true
  election-timeout = "1s"
  heartbeat-timeout = "1s"
  leader-lease-timeout = "500ms"
  commit-timeout = "50ms"
  cluster-tracing = false

[data]
  dir = "/root/.influxdb/db"
  max-wal-size = 104857600 # Maximum size the WAL can reach before a flush. Defaults to 100MB.
  wal-flush-interval = "10m" # Maximum time data can sit in WAL before a flush.
  wal-partition-flush-delay = "2s" # The delay time between each WAL partition being flushed.
  wal-dir = "/root/.influxdb/wal"
  wal-enable-logging = true
  wal-ready-series-size = 30720
  wal-compaction-threshold = 0.5
  wal-max-series-size = 1048576
  wal-flush-cold-interval = "5m0s"
  wal-partition-size-threshold = 20971520

[cluster]
  force-remote-mapping = false
  write-timeout = "5s" # The time within which a write operation must complete on the cluster.
  shard-writer-timeout = "5s" # The time within which a shard must respond to write.
  shard-mapper-timeout = "5s"

[retention]
  enabled = true
  check-interval = "10m"

[shard-precreation]
  enabled = true
  check-interval = "10m0s"
  advance-period = "30m0s"

[admin]
  enabled = true
  bind-address = ":8083"
  https-enabled = false
  https-certificate = "/etc/ssl/influxdb.pem"

[http]
  enabled = true
  bind-address = ":8086"
  auth-enabled = false
  log-enabled = true
  write-tracing = false
  pprof-enabled = false
  https-enabled = false
  https-certificate = "/etc/ssl/influxdb.pem"

[[graphite]]
  enabled = true
  bind-address = ":2003"
  protocol = "udp"
  consistency-level = "one"
  name-separator = "."
  database = "graphite"
  batch-size = 1000
  batch-timeout = "1s"
  name-schema = "application.instance.profile.measurement*"
  ignore-unnamed = true

[collectd]
  enabled = false

[opentsdb]
  enabled = false

[[udp]]
  enabled = false

[monitoring]
  enabled = false
  write-interval = "24h"

[continuous_queries]
  log-enabled = true
  enabled = true
  recompute-previous-n = 2
  recompute-no-older-than = "10m"
  compute-runs-per-interval = 10
  compute-no-more-than = "2m"

[hinted-handoff]
  enabled = true
  dir = "/root/.influxdb/hh"
  max-size = 1073741824
  max-age = "168h"
  retry-rate-limit = 0
  retry-interval = "1s"

this resulted in the following (shortened) log output:

[graphite] 2015/09/12 16:29:29 Failed to ensure target database graphite exists: rpc FetchData failed: short read: got 0, exp 16
[snapshot] 2015/09/12 16:29:29 snapshot listener closed
[tcp] 2015/09/12 16:29:29 cluster service accept error: network connection closed
[shard-precreation] 2015/09/12 16:29:29 Precreation service terminating
[continuous_querier] 2015/09/12 16:29:29 continuous query service terminating
[retention] 2015/09/12 16:29:29 retention policy enforcement terminating
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x50 pc=0x1cdf58]

goroutine 1 [running]:
github.com/influxdb/influxdb/tsdb.(*PointBatcher).Stop(0x0)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/tsdb/batcher.go:109 +0x18
github.com/influxdb/influxdb/services/graphite.(*Service).Close(0xc8201500c0, 0x0, 0x0)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/services/graphite/service.go:129 +0x60
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Close(0xc82012c380, 0x0, 0x0)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:372 +0x141
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open(0xc82012c380, 0x0, 0x0)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:348 +0x55
github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run(0xc820134100, 0xc82000a430, 0x1, 0x1, 0x0, 0x0)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:112 +0x992
main.(*Main).Run(0xc820101f00, 0xc82000a430, 0x1, 0x1, 0x0, 0x0)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/cmd/influxd/main.go:81 +0x67e
main.main()
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/cmd/influxd/main.go:42 +0x29b

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/Cellar/go/1.5/libexec/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall]:
os/signal.loop()
    /usr/local/Cellar/go/1.5/libexec/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/Cellar/go/1.5/libexec/src/os/signal/signal_unix.go:28 +0x37

goroutine 27 [sleep]:
time.Sleep(0x3b9aca00)
    /usr/local/Cellar/go/1.5/libexec/src/runtime/time.go:59 +0xf9
github.com/influxdb/influxdb/meta.(*Store).syncNodeInfo(0xc82012a280, 0x0, 0x0)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/meta/store.go:292 +0x82
created by github.com/influxdb/influxdb/meta.(*Store).Open
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/meta/store.go:253 +0x540

goroutine 28 [select]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).monitorErrorChan(0xc82012c380, 0xc82011e6c0)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:442 +0x145
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:319 +0xbff

goroutine 527 [chan receive]:
github.com/influxdb/influxdb/meta.proxy(0xc820118270, 0xc820224240, 0x0, 0x0)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/meta/proxy.go:42 +0x29a
github.com/influxdb/influxdb/meta.(*rpc).proxyLeader(0xc82017d780, 0xc820224240)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/meta/rpc.go:66 +0x440
github.com/influxdb/influxdb/meta.(*rpc).handleRPCConn(0xc82017d780, 0x1b640b0, 0xc820224240)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/meta/rpc.go:80 +0x1e6
github.com/influxdb/influxdb/meta.(*Store).serveRPCListener.func1(0xc82012a280, 0x1b640b0, 0xc820224240)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/meta/store.go:737 +0x75
created by github.com/influxdb/influxdb/meta.(*Store).serveRPCListener
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/meta/store.go:738 +0x29f

goroutine 578 [chan send]:
github.com/influxdb/influxdb/meta.broker(0x1b640b0, 0xc820224240, 0x1b640b0, 0xc820118270, 0xc8202d27e0, 0xc8202d28a0)
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/meta/proxy.go:56 +0xfa
created by github.com/influxdb/influxdb/meta.proxy
    /private/tmp/influxdb20150827-14196-z5nyki/influxdb-0.9.3/src/github.com/influxdb/influxdb/meta/proxy.go:16 +0x1d7

[otoolep]

Auto-creation of whatever database is specified with your Graphite inputs does work. Something else is up with your system, that is prevent configurations such as database creation taking place. I have not seen this issue before.

Are you running a cluster?

[aheusingfeld]

No, only this single instance installed via Homebrew on OS X (brew info influxdb tells me it is influxdb: stable 0.9.3, HEAD) and started via influxd -config config.toml with the above mentioned config.

I will retest this on another host on Monday to check that the error messages are really identical.

[cdgraff]

Hi @otoolep I has similar error but starting with OpenTSDB engine.

Here the log https://gist.github.com/cdgraff/1be844aa5fcd22d962e6

I has setup by default... start well and with the first data arrive crash...

[otoolep]

@cdgraff -- the trace you posted looks exactly like the panic #4136 will fix.

[otoolep]

@aheusingfeld -- are you still having issues?

[otoolep]

I can't reproduce this with the given config and master.

[aheusingfeld]

You're right, this seems to be fixed in 0.9.4. Thanks!