vault: recover from standby losing etcd lease (#3031) #3511
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Fixes #3031 |
xiang90
reviewed
Oct 31, 2017
physical/etcd/etcd3.go
Outdated
| } | ||
|
|
||
| if needNewSession { | ||
| session, err := concurrency.NewSession(c.etcd, concurrency.WithTTL(etcd3LockTimeoutInSeconds)) |
There was a problem hiding this comment.
move the code to line 273 directly?
|
@bhiggins the fix looks right. can you simplify the code? have you tested it manually? if yes, then it looks good to me. |
|
@bhiggins thanks! |
This change makes these errors transient instead of permanent: [ERROR] core: failed to acquire lock: error=etcdserver: requested lease not found After this change, there can still be one of these errors when a standby vault that lost its lease tries to become leader, but on the next lock acquisition attempt a new session will be created. With this new session, the standby will be able to become the leader.
bhiggins
force-pushed
the
etcd-lost-lease-recovery
branch
from
bb34891
to
68b1abc
Compare
|
Simplified code as requested. Yes, I tested this patch against 0.7.3 by having three vaults running in HA config. I manually deleted the standby's leases out of etcd, and then killed the leader. After that, as described in the commit message, the "lease not found" error occurs once but then on the next lock attempt a new session is created and the standby becomes the leader. |
|
lgtm. defer to @jefferai |
|
Hah, it's the other way, when it comes to etcd3 I defer to @xiang90 :-D Merging. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change makes these errors transient instead of permanent:
[ERROR] core: failed to acquire lock: error=etcdserver: requested lease not found
After this change, there can still be one of these errors when a
standby vault that lost its lease tries to become leader, but on the
next lock acquisition attempt a new session will be created. With this
new session, the standby will be able to become the leader.