HAproxy health-check causes node shutdown

[seancribbs]

HAproxy uses a naïve health-check mechanism that simply opens a TCP socket and then closes it immediately after it is accepted by the server. This causes the call to peername/1 in riak_api_pb_server to return {error, enotconn} instead of the expected {ok, Peername}. The resulting badmatch propagates to the listener process (it was waiting on a reply via gen_fsm:sync_send_event), which then crashes the application supervisor after too many restarts.

Using this example config:

listen riak_pbc_demo_10 0.0.0.0:9109
    balance leastconn
    option tcplog
    mode tcp
    server 127.0.0.1 127.0.0.1:8087 check

[seancribbs]

It's worth noting that this was not an issue on 1.4.x because we didn't have the security features that require checking the peername, so the process would simply shutdown normally at a time after the set_socket call.

[seancribbs]

^^ @cipy

[roooms]

I tested two haproxy configs and I only see this issue in pre11 when option nolinger [0] is used.

Saying that, having tested the configuration @cipy is using (including nolinger) the pre11 crashes as reported, and this branch stopped the crashing behaviour. So +1

[0] http://cbonte.github.io/haproxy-dconv/configuration-1.4.html#option%20nolinger

[seancribbs]

Thanks @roooms. Can I get an Eng +1?

[engelsanchez]

This looks racy. How about setting a monitor on the server, and expect a 'DOWN' message within a given timeout instead?

[seancribbs]

Good idea.

[engelsanchez]

Quick note: if you were to run the eunit test on the old code, it is skipped because of way the crash happens, not marked as failed.

[engelsanchez]

So probably a good idea to start without linking to the test process here to really verify what happens to it.

[engelsanchez]

The test looks much better, and now fails with the old code, passes with the fixes. I did not test the HAProxy configuration myself, as Dan has already and know nothing about HAProxy. I hope between my 👍 and his it's enough.

💃 ⛵

[seancribbs]

Thanks @engelsanchez @roooms @cipy