There should be a way to force a node to leave the cluster

[jvshahid]

There are a few cases when we need to force a node to leave the cluster, see this thread for an example. We need to find a way to force a node to leave the cluster in case there's no quorum.

[kapilt]

this is pretty critical for multi-node software. When i take a node down.. i get a dogpile in the logs of all the other servers of the form

[2014/05/17 17:16:00 EDT] EROR Error while reading messsage size: &{%!d(string=use of closed network connection)}
[2014/05/17 17:16:00 EDT] EROR Error while reading messsage size: &{%!d(string=use of closed network connection)}
[2014/05/17 17:16:00 EDT] EROR Error while reading messsage size: &{%!d(string=use of closed network connection)}
[2014/05/17 17:16:00 EDT] EROR Error while reading messsage size: &{%!d(string=use of closed network connection)}
[2014/05/17 17:16:01 EDT] EROR Error while reading messsage size: &{%!d(string=use of closed network connection)}
[2014/05/17 17:16:01 EDT] EROR Error while reading messsage size: &{%!d(string=use of closed network connection)}
[2014/05/17 17:16:01 EDT] EROR Error while reading messsage size: &{%!d(string=use of closed network connection)}
[2014/05/17 17:16:01 EDT] EROR Error while reading messsage size: &{%!d(string=use of closed network connection)}
[2014/05/17 17:16:01 EDT] EROR Error while reading messsage size: &{%!d(string=use of closed network connection)}

[jvshahid]

I attached a patch to this issue that will add a way to forcibly remove a node from a cluster. This is a destructive action. Also, don't merge this in. I have to push a change to the influxdb-go client.

[pauldix]

lgtm

[ikreymer]

I had a similar example as in the thread (accidentally started a node with a seed list had it join a cluster), then stopped the node, removed the seed list, deleted the node with curl -XDELETE from the cluster,
rm -rf the shared/data dir, and then restarted the node.

The cluster doesn't show the node and the node itself appears to list only itself as the server, however, upon startup, it still gets a bunch of:

[INFO] (github.com/influxdb/influxdb/coordinator.(*ProtobufServer).handleConnection:77) ProtobufServer: client connected

from each of the nodes in the cluster. I've tried restarting the new node a couple of times, and it seems that the other nodes still try to connect to it, though it looks like it hasn't been added to the cluster, which is good.

Seems like there should not be any attempted connections from the old cluster once a node has been removed?