[0.9.5] HTTP 500 errors on insertions (after timeout)

[joshliberty]

I have a cluster of production servers writing web requests to a single influxdb node.
They're doing up to 100 requests per second - not something I'd expect to crash a node.

But after the system is up for a few hours (sometimes days), the clients start getting 500 errors. The log is littered with these messages:

[http] 2015/11/21 12:42:01 10.60.195.206 - root [21/Nov/2015:12:41:46 -0600] POST /write?db=mydb HTTP/1.1 500 32 - python-requests/2.4.3 CPython/2.7.3 Linux/3.2.0-4-amd64 84aa8e82-907f-11e5-b414-000000000000 15.000491819s

When this happens, I'm also getting timeouts on continuous queries & the following error:

[monitor] 2015/11/21 19:04:56 failed to store statistics: timeout

I'm not batching my queries, since they're web requests and I have no real way to batch them.

If useful, a typical minute looks like this:

[wal] 2015/11/21 22:17:06 Flush due to idle. Flushing 31 series with 31 points and 1544 bytes from partition 1
[wal] 2015/11/21 22:17:16 Flush due to idle. Flushing 31 series with 31 points and 1544 bytes from partition 1
[wal] 2015/11/21 22:17:26 Flush due to idle. Flushing 31 series with 31 points and 1544 bytes from partition 1
[wal] 2015/11/21 22:17:36 Flush due to idle. Flushing 31 series with 31 points and 1544 bytes from partition 1
[wal] 2015/11/21 22:17:46 Flush due to idle. Flushing 31 series with 31 points and 1544 bytes from partition 1
[wal] 2015/11/21 22:17:56 Flush due to idle. Flushing 31 series with 31 points and 1544 bytes from partition 1

[joshliberty]

This also happens with the new storage engine, tsm1.

[beckettsean]

@caligula1989 when the system is non-responsive to writes and queries, does the /ping endpoint still respond?

Does a restart clear the issue?

[beckettsean]

@corylanou @rossmcdonald this sounds pretty familiar, is it another occurrence of the potential deadlock?

[joshliberty]

A restart clears the issue for couple of hours, but it'll happen again.
I'm not sure about the ping endpoint, we check again next time it happens. If I had to venture a guess however, it would work, since queries such as "show measurements" still work on other DBs at least.

[rossmcdonald]

@caligula1989 Thank you for the extra information. Can you also try sending a SIGQUIT signal to the process the next time this happens, and then send us the resulting stack trace? You can send a SIGQUIT to InfluxDB by using the command:

kill -SIGQUIT <PID of InfluxDB>

And then capture the stack trace output using the command:

sed -n '/SIGQUIT/,/InfluxDB starting/p' /var/log/influxdb/influxd.log > stack-trace.out

I can send you a link to an S3 bucket if it's too large to paste here.

[beckettsean]

Believed fixed by #4913, which is available in the just-released version 0.9.5.1.