Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

raft issue: break linear consistency because of two leaders and raft lease read. #5352

Closed
tangyuanzhang opened this issue Feb 17, 2023 · 10 comments
Assignees
Labels
affects/none PR/issue: this bug affects none version. process/fixed Process of bug severity/none Severity of bug type/bug Type: something is unexpected

Comments

@tangyuanzhang
Copy link
Contributor

image
leader_ and isBlindFollower_ act the same, I think isBlindFollower_ == True eq leader_ == HostAddr("", 0)

We should increase the limit when voting. The follwer host leader_ != HostAddr("", 0) and leader lease valid recorded by the current host should refuse to vote to prevent split-brain

@tangyuanzhang tangyuanzhang added the type/bug Type: something is unexpected label Feb 17, 2023
@github-actions github-actions bot added affects/none PR/issue: this bug affects none version. severity/none Severity of bug labels Feb 17, 2023
@tangyuanzhang
Copy link
Contributor Author

isBlindFollower_ maybe cause split brain, A(leader)B(follower) C(followe), restart B and C, B start election
,C restart ok,dont start election,C vote for B。 B send request to A is lost, B is leader and A is leader

@liwenhui-soul
Copy link
Contributor

why does B start election?

@tangyuanzhang
Copy link
Contributor Author

start to election has gap

@critical27
Copy link
Contributor

isBlindFollower_ maybe cause split brain, A(leader)B(follower) C(followe), restart B and C, B start election ,C restart ok,dont start election,C vote for B。 B send request to A is lost, B is leader and A is leader

Both leader is possible, but they are not same term. This is a very common case for raft.

isBlindFollower_ is only used for speed up first election.

@critical27
Copy link
Contributor

I don't think this is a issue, reopen it if necessary

@github-actions github-actions bot added the process/fixed Process of bug label Mar 9, 2023
@tangyuanzhang
Copy link
Contributor Author

tangyuanzhang commented Mar 9, 2023

But if there is data written to B now, it can be submitted successfully, but the read data is routed to A, then the data cannot be read or old. Does this cause inconsistency?

@critical27
Copy link
Contributor

I got your point.
It won't cause inconsistency. Because at least B or C has same log as A, let's say it is B, only B could be elected as leader at higher term.

Read could be routed to A, but write to A won't be success anymore. So again, no inconsistency happened.

@tangyuanzhang tangyuanzhang changed the title raft some issue raft issue: break linear consistency because of two leaders and raft lease read. Mar 14, 2023
@qiwei9743
Copy link

qiwei9743 commented Mar 14, 2023

It indeed break the linear consistency. nebula implement pure lease read but didn't implement checkquorum and read index as ETCD does. The old leader may read the stale data from old leader after new write over B and C(in shixiang's case). Although it's a small gap, nebula can't declare it implement a strong consistency.

@critical27
Copy link
Contributor

critical27 commented Mar 20, 2023

It indeed break the linear consistency. nebula implement pure lease read but didn't implement checkquorum and read index as ETCD does. The old leader may read the stale data from old leader after new write over B and C(in shixiang's case). Although it's a small gap, nebula can't declare it implement a strong consistency.

Perhaps you need to describe in more details how inconsistency happen. Liked I said in #5352 (comment), two leader doesn't mean split brain. Only when two leader of same term is. New leader of B or C can only trigger election after an election timeout passed. If B or C triggers election, which means, A's lease is valid as well. It won't serve read any more.

@critical27
Copy link
Contributor

critical27 commented Mar 20, 2023

Although I agree that blind follower that may break the precondition that election could only after an election timeout passed.

I think we could remove the blind follower? WDYT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects/none PR/issue: this bug affects none version. process/fixed Process of bug severity/none Severity of bug type/bug Type: something is unexpected
Projects
None yet
Development

No branches or pull requests

4 participants