ReadMode: introduce AutoFollowerRead mechanism by adding a new ReadMode == PreferLeader

[LykxSassinator]

Description

Introduce AutoFollowerRead mechanism to improve the Read availability on Read workloads when setting with ReadMode == PreferLeader. Here, a new read mode PreferLeader has been introduced for enabling this mechanism.

Close #670.

Signed-off-by: lucasliang nkcs_lykx@hotmail.com

[LykxSassinator]

@tonyxuqqi PTAL,thx

[disksing]

I wonder If the load of different stores is different (for example, some stores are all point get requests, while others are full table scans), then the method of determining store slow may not be accurate.

[LykxSassinator]

I wonder If the load of different stores is different (for example, some stores are all point get requests, while others are full table scans), then the method of determining store slow may not be accurate.

Thx for your comments.
What u've pointed about the influence of different workloads has been considered in the design of the detection algorithm.

Design in brief introduction

One store in TiKV clusters may be slow because of network delays or IO delays on disk. Currently, we cannot have a very concise method to aggregate all informations into the Client and make the determination on which store is slow.
So, we use a measurement, so-called Slow Score, to represent the severity of slow on each store. And only when the Slow Score exceed the threshold(80 in current implementation), will the corresponding store be marked as SLOW. Then, the read flows will be redirected to other nodes.

And about the calculation of Slow Score, we introduce two dimensions to determine it on each store:

• Requests in one timing tick. This dimension can be regarded as QPS on each store.
• Average timecost on each request in one timing tick. This dimension is used to detect whether the relative store is busy on processing requests.

And as we known, if one store is slow, its Requests will keep decreasing gradually, but Average timecost will keep ascending.

So, we just introduce an appropriate algorithm(just like Sliding window) to combine these two factors and convert them to Slow Score, for representing the business of each node. That is:

• Slow Score focus the severity of slowness on each store, and each store has its own counting statistics for calculating Slow Score.
• Slow Score of one store will gradually ascend if this store is gradually slow on processing requests. And this store will be marked as SLOW if Slow Score exceed the threshold.

Of course, about the point on:

store slow may not be accurate.

Slow Score can be more accurate if we make the statistics divided by the type of commands, which can be reviewed in later generation of this algorithm. We just wanna introduce an appropriate and simple way, for less costs of resources, to optimize the replica_read_mode == leader_and_follower mode.

[disksing]

@LykxSassinator Thank you for the clarification.

[LykxSassinator]

cc @disksing PTAL, thx.