-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Calculate follower staleness by using replicate messages timestamp #4868
Labels
Comments
hectorgcr
added
kind/enhancement
This is an enhancement of an existing feature
area/docdb
YugabyteDB core features
labels
Jun 24, 2020
hectorgcr
added a commit
that referenced
this issue
Jul 13, 2020
…e as a measure of staleness Summary: When doing read from followers, (also known as consistency ONE in YCQL or consistent prefix reads), we use the time we received the last consensus request from the leader to measure the staleness of a follower. This diff changes that, and instead it uses the difference between the follower's hybrid time, and its safe time that is derived from the the leader's safe time and propagated through consensus request messages. This diff also changes the default value of flag `max_stale_read_bound_time_ms` to 10 seconds to avoid serving stale request without the user being aware. Test Plan: Ran a yb-sample app with local reads enabled, stopped one of the nodes for ~40s, started it again, and checked through log messages that the staleness was recorded correctly (~40s) as soon as the tablet was bootstrapped. Reviewers: amitanand, mikhail Reviewed By: mikhail Subscribers: kannan, mikhail, ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D8698
hectorgcr
added a commit
that referenced
this issue
Jul 16, 2020
…ime and its safe time as a measure of staleness Summary: When doing read from followers, (also known as consistency ONE in YCQL or consistent prefix reads), we use the time we received the last consensus request from the leader to measure the staleness of a follower. This diff changes that, and instead it uses the difference between the follower's hybrid time, and its safe time that is derived from the the leader's safe time and propagated through consensus request messages. This diff also changes the default value of flag `max_stale_read_bound_time_ms` to 10 seconds to avoid serving stale request without the user being aware. Test Plan: Ran a yb-sample app with local reads enabled, stopped one of the nodes for ~40s, started it again, and checked through log messages that the staleness was recorded correctly (~40s) as soon as the tablet was bootstrapped. Reviewers: bogdan Reviewed By: bogdan Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D8906
hectorgcr
added a commit
that referenced
this issue
Jul 16, 2020
…ime and its safe time as a measure of staleness Summary: When doing read from followers, (also known as consistency ONE in YCQL or consistent prefix reads), we use the time we received the last consensus request from the leader to measure the staleness of a follower. This diff changes that, and instead it uses the difference between the follower's hybrid time, and its safe time that is derived from the the leader's safe time and propagated through consensus request messages. This diff also changes the default value of flag `max_stale_read_bound_time_ms` to 10 seconds to avoid serving stale request without the user being aware. Backport to 2.2.0 Test Plan: Ran a yb-sample app with local reads enabled, stopped one of the nodes for ~40s, started it again, and checked through log messages that the staleness was recorded correctly (~40s) as soon as the tablet was bootstrapped. Reviewers: amitanand, mikhail Subscribers: bogdan, ybase, mikhail, kannan Differential Revision: https://phabricator.dev.yugabyte.com/D8922
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
When reading from followers (consistency level ONE for YCQL), we have a flag to limit the staleness of the data that a follower can serve. Our current implementation uses the time that the follower last heard from the leader (through an update consensus message) to do this calculation. The problem with this is that a follower could be behind even if it's still receiving messages from the leader. For example, after a node restarts, the leader will try to catch up the follower as soon as the follower opens its tablet, but these messages will contain old log entries. So instead of using the time we last heard from the leader, we should use the time of the last log entry in a consensus update, or the last time we last heard from the leader if the message was empty (an empty message means that the follower is up-to-date, and it is used as a heartbeat between leaders and followers).
The text was updated successfully, but these errors were encountered: