Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CL.ONE (read from followers) serving reads before node is fully caught up #456

Closed
rameshpoti opened this issue Aug 31, 2018 · 2 comments
Closed

CL.ONE (read from followers) serving reads before node is fully caught up #456

rameshpoti opened this issue Aug 31, 2018 · 2 comments
Assignees
@amitanandaiyer
Labels
kind/bug This issue is a bug

Comments

@rameshpoti
Copy link

CassandraKeyValue sample app in Read only load and when a node is restarted seeing the following error when the node joins back to the cluster.

2018-08-30 23:48:40,389 [FATAL|com.yugabyte.sample.apps.CassandraKeyValue|CassandraKeyValue] Read key: key:33191 expected 1 row in result, got 0

Initial assumption : Until a "follower" is caught up replaying its local RAFT log, it shouldn't serve reads. Until then it should just redirect to one of the other two peers. (edited)
@kmuthukk
Copy link
Collaborator

Thanks @rameshpoti for reporting this issue. Will keep you posted once the fix is available.

@kmuthukk kmuthukk added the kind/bug This issue is a bug label Aug 31, 2018
@amitanandaiyer amitanandaiyer self-assigned this Aug 31, 2018
@amitanandaiyer
Copy link
Contributor

This seems to be caused by the follower not resetting safe_time_to_read correctly.

We seem to have been setting safe_time_to_read based on the flushed sst files. Not accounting for the committed entries in rocksdb that haven't been flushed.

But, the committed entries are present in the Write Ahead Log and replayed upon restart. The fix is to update the initial safe_time_to_read based on the applied entries from WAL in addition to the flushed files.

yugabyte-ci pushed a commit that referenced this issue Sep 12, 2018
@amitanandaiyer
[ENG-3874] #456: Update committed hybrid time from WAL records
2fdad35 
Summary:
Currently we update local safe time based on the written SST files. But fail to do so for WAL records. This may cause a follower -- who has restarted -- to miss those entries, until he reconnects with the Leader.

This fix is to update the safe time based on entries in the log that are known to be committed.

For entries that are in the log, that cannot be determined to have committed (because there are no further log entries telling us they are); the follower will have to wait to hear from the leader before serving them.

Test Plan: added unit test

Reviewers: mikhail, kannan, sergei

Reviewed By: sergei

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D5442
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amitanandaiyer amitanandaiyer

Labels
kind/bug This issue is a bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kmuthukk @rameshpoti @amitanandaiyer