Avoid routing to cluster replicas that are loading data

[mp911de]

Nodes that have joined the cluster and still load data should not receive any traffic unless they are fully replicated. As per @madolson, if the master_repl_offset is 0, we should skip that node from command routing.

[mp911de]

Removing replicas based on the repl-offset will also remove replicas that cannot contact the master/upstream node although the node is functional otherwise. That is quite a significant change in comparison to leniently accept all replicas.

[jhmartin]

Is there a middle ground? Accept nodes that have /ever/ replicated successfully, and ignore nodes that have /never/ replicated?

[madolson]

Starting in 6.2, repl-offset is preserved until the point the data is flushed, which means that it is no longer able to usefully serve traffic (except for pubsub, but I would posit we don't care about that here).

[mp911de]

For the time being, we have that switch avoids command routing to replicas without data within the routing mechanism. If it turns out to be too invasive, we can move things into the individual ReadFrom strategies so it can be controlled by applications whether they want to talk to replicas without data.

[mp911de]

Refined the description, we're using INFO REPLICATION and the master_repl_offset field to determine the repl-offset.