Support incremental cluster joins

[jwilder]

Overview

This PR is a follow up to #3372 and implements more the of functionality for #2966. Specifically, it allows servers to become raft peers when joining. This allows single node cluster to be expanded to larger clusters and automatically use a large number of nodes for raft consensus to provide better availability. By default, this is still hard-code to 3 nodes max, but could be made configurable. The first three nodes added to the cluster will become raft peer members. The remaining are data-only nodes.

To add a new member to the cluster, you should start influxd with the -join flag. The -join flag tags a host:port value. Multiple host:ports can be specific by comma separating them. For example:

influxd -join host1:8088,host2:8088,host3:8088

The port should be the cluster port (default 8088). Not the API port (default 8086).

The existing [meta].Peers config var can also be used and is equivalent to the -join flag. If a both are specified, the -join flag override the config value.

If nodes are restarted that have previously joined a cluster, their existing cluster state on disk will be used and any join or config variable will be ignored.

Additionally, the SHOW SERVERS statement now has a raft column that indicates whether the node is running raft or not.

Not implemented

• Leaving the cluster - The infrastructure is in place to remove a raft peer, but is not currently exposed. Removing a non-raft peer is not supported at this time.
• Promoting/demoting raft peers - The infrastructure to promote/demote nodes to become raft peers is in place but not exposed or wired up currently.
• Changing hostnames - The hostname:port used when joining the cluster can't be changed after the fact. This will be addressed in Should update metastore and cluster if IP or hostname changes #3421

[pauldix]

overall lgtm. w00t clustering!!

[beckettsean]

@jwilder is there a difference between the following scenarios? (assume servers are launched in alphabetical order)

B, C, & D join A

Server A: influxd -config /path/to/config
Server B: influxd -config /path/to/config -join serverA:8088
Server C: influxd -config /path/to/config -join serverA:8088
Server D: influxd -config /path/to/config -join serverA:8088

B joins A, C joins B & A, D joins C, B, & A

Server A: influxd -config /path/to/config
Server B: influxd -config /path/to/config -join serverA:8088
Server C: influxd -config /path/to/config -join serverA:8088,serverB:8088
Server D: influxd -config /path/to/config -join serverA:8088,serverB:8088,serverC:8088

Putting it another way, why would I specify multiple servers in the -join flag?

[beckettsean]

"The existing [meta].Peers config var can also be used and is equivalent to the -join flag. If a both are specified, the -join flag override the config value."

So what's the reason for the -join flag? If I set the peers correctly in [meta] the server does the right thing on a restart. If I just use the -join flag it only does the right thing on the first launch (or I must pass -join every time).

[pauldix]

@beckettsean peers and join only get used on the first startup. Generally I would prefer to not even have peers since people think it's something that gets used all the time if it's in the config.

Once it has joined the cluster, it writes a file to the local disk that contains the information. On restart it will see this file is there and use that to reconnect to the cluster.

[pauldix]

There's no difference in the situation you describe. The list is just to have other ones to try to join to if the first fails. The only thing that matters is the order in which the four systems join the cluster because the 4th won't run the consensus protocol

[beckettsean]

Thanks, Paul. I tend to agree, that if [meta] peers is a one-time thing it shouldn't be in the config at all. Maybe we can deprecate that with 0.9.3 and remove in 0.9.4?

Is the cluster file on disk human readable and/or editable? If I blow it away presumably the node forgets about the cluster on restart unless there's a -join or [meta] section involved?

Understood about the first three make the consensus group and all subsequent are not participants. It makes sense that the -join command could specify fallback servers in case one doesn't respond, and that makes it idempotent with respect to startup timing for each process.

[kfitzpatrick]

It would be great to have something that could be run while the database is up or in the config file that I can change and bounce the service. When I'm spinning up machines I don't want to have to pass a special argument to the machine. it makes the monit scripts a pain to deal with and maintain automatically.

Also, if a new node is added, then I have to go back into all the existing nodes somehow and change everything.

Reason this matters: I would like to add the ability to add nodes dynamically to a cluster instead of having to bring them all up at the same time in our hosted environment.

[pauldix]

@kfitzpatrick you don't need to do anything to existing nodes. They all get notified of the new member automatically. This design allows you to bring up new nodes at any time and join them to a cluster

[kfitzpatrick]

So let's see if I got this. I create node A. I bring up another node (B) and have to start it with "--join A:8088". Or, if it's not removed, add A to B's Peers in the config and bring it up and now they all know about each other.

Correct, @pauldix ?

[pauldix]

right, except I'm not sure what you mean by "Or, if it's not removed". I assume you meant that if you don't specify the join argument you have to list A in B's peers? If so, then you're correct

[toddboom]

i think he meant "if it's not removed from the config file". maybe?