Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calculate follower staleness by using replicate messages timestamp #4868

Closed
hectorgcr opened this issue Jun 24, 2020 · 0 comments
Closed

Calculate follower staleness by using replicate messages timestamp #4868

hectorgcr opened this issue Jun 24, 2020 · 0 comments
Assignees
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature

Comments

@hectorgcr
Copy link
Contributor

When reading from followers (consistency level ONE for YCQL), we have a flag to limit the staleness of the data that a follower can serve. Our current implementation uses the time that the follower last heard from the leader (through an update consensus message) to do this calculation. The problem with this is that a follower could be behind even if it's still receiving messages from the leader. For example, after a node restarts, the leader will try to catch up the follower as soon as the follower opens its tablet, but these messages will contain old log entries. So instead of using the time we last heard from the leader, we should use the time of the last log entry in a consensus update, or the last time we last heard from the leader if the message was empty (an empty message means that the follower is up-to-date, and it is used as a heartbeat between leaders and followers).

@hectorgcr hectorgcr added kind/enhancement This is an enhancement of an existing feature area/docdb YugabyteDB core features labels Jun 24, 2020
@hectorgcr hectorgcr self-assigned this Jun 24, 2020
hectorgcr added a commit that referenced this issue Jul 13, 2020
…e as a measure of staleness

Summary:
When doing read from followers, (also known as consistency ONE in YCQL
or consistent prefix reads), we use the time we received the last consensus
request from the leader to measure the staleness of a follower.

This diff changes that, and instead it uses the difference between the
follower's hybrid time, and its safe time that is derived from the the leader's
safe time and propagated through consensus request messages.

This diff also changes the default value of flag `max_stale_read_bound_time_ms`
to 10 seconds to avoid serving stale request without the user being aware.

Test Plan:
Ran a yb-sample app with local reads enabled, stopped one of the nodes for
~40s, started it again, and checked through log messages that the staleness
was recorded correctly (~40s) as soon as the tablet was bootstrapped.

Reviewers: amitanand, mikhail

Reviewed By: mikhail

Subscribers: kannan, mikhail, ybase, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D8698
hectorgcr added a commit that referenced this issue Jul 16, 2020
…ime and its safe time as a measure of staleness

Summary:
When doing read from followers, (also known as consistency ONE in YCQL
or consistent prefix reads), we use the time we received the last consensus
request from the leader to measure the staleness of a follower.

This diff changes that, and instead it uses the difference between the
follower's hybrid time, and its safe time that is derived from the the leader's
safe time and propagated through consensus request messages.

This diff also changes the default value of flag `max_stale_read_bound_time_ms`
to 10 seconds to avoid serving stale request without the user being aware.

Test Plan:
Ran a yb-sample app with local reads enabled, stopped one of the nodes for
~40s, started it again, and checked through log messages that the staleness
was recorded correctly (~40s) as soon as the tablet was bootstrapped.

Reviewers: bogdan

Reviewed By: bogdan

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D8906
hectorgcr added a commit that referenced this issue Jul 16, 2020
…ime and its safe time as a measure of staleness

Summary:
When doing read from followers, (also known as consistency ONE in YCQL
or consistent prefix reads), we use the time we received the last consensus
request from the leader to measure the staleness of a follower.

This diff changes that, and instead it uses the difference between the
follower's hybrid time, and its safe time that is derived from the the leader's
safe time and propagated through consensus request messages.

This diff also changes the default value of flag `max_stale_read_bound_time_ms`
to 10 seconds to avoid serving stale request without the user being aware.

Backport to 2.2.0

Test Plan:
Ran a yb-sample app with local reads enabled, stopped one of the nodes for
~40s, started it again, and checked through log messages that the staleness
was recorded correctly (~40s) as soon as the tablet was bootstrapped.

Reviewers: amitanand, mikhail

Subscribers: bogdan, ybase, mikhail, kannan

Differential Revision: https://phabricator.dev.yugabyte.com/D8922
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature
Projects
None yet
Development

No branches or pull requests

1 participant