Panic on 1.0-rc1 (runtime: goroutine stack exceeds 1000000000-byte limit)

[liv3d]

System info:
InfluxDB 1.0-rc1
CentOS 7

Similar to #7180 (but I don't have a 23 million series database this time, it's currently at 610k), we tried upgrading to rc1 but I got another/different panic on start up 😞

log output is here: https://gist.github.com/liv3d/1adddd6452be7a5de462f7ac3b2a611e

@jwilder / @benbjohnson I can grab a backup of any data or any more stats if you want more information. Otherwise is there a nightly with #7196 in and we can try that?

[kostasb]

I was able to consistently reproduce this error with the following steps on 1.0-rc1:

> create database md
> use md
Using database md
> alter retention policy autogen on md duration 1h replication 1 shard duration 1m
> insert tester,taga=ena fielda=1
> insert tester,taga=beta fielda=2
> CREATE CONTINUOUS QUERY mycq1 ON md BEGIN SELECT count(distinct(fielda)) AS countfielda INTO md.autogen.cq1 FROM md.autogen.tester GROUP BY time(1m), taga END

> CREATE CONTINUOUS QUERY mycq2 ON md BEGIN SELECT sum(countfielda) AS sumcount INTO md.autogen.cq2 FROM md.autogen.cq1 GROUP BY time(5m), taga END
> insert tester,taga=ena fielda=1
> insert tester,taga=ena fielda=2
> insert tester,taga=ena fielda=3
> insert tester,taga=ena fielda=4
> insert tester,taga=beta fielda=1

...wait a couple mins

> show shards
name: md
--------
id  database    retention_policy    shard_group start_time      end_time        expiry_time     owners
4   md      autogen         4       2016-08-29T14:25:00Z    2016-08-29T14:26:00Z    2016-08-29T15:26:00Z    
2   md      autogen         2       2016-08-29T14:28:00Z    2016-08-29T14:29:00Z    2016-08-29T15:29:00Z    
3   md      autogen         3       2016-08-29T14:29:00Z    2016-08-29T14:30:00Z    2016-08-29T15:30:00Z    

> select * from cq2
name: cq2
---------
time            sumcount    taga
1472480700000000000 2       beta
1472480700000000000 4       ena


...wait a couple minutes

> insert cq1,taga=ena countfielda=1
...(write is accepted while it should be dropped with "field type conflict" error)

> select * from cq1
ERR: unexpected EOF
> 

Configuration standard except for:

  cache-snapshot-write-cold-duration = "5s"
  compact-full-write-cold-duration = "5s"

Logfile with stack trace attached:

overflow.log.gz

[jwilder]

Fixed via #7232