NRG: Wait for goroutines to shutdown when recreating group #5832
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
neilalexander
changed the title
neilalexander
force-pushed
the
neil/nrgwg
branch
from
9613916 to
01a15f8
Compare
neilalexander
changed the title
neilalexander
force-pushed
the
neil/nrgwg
branch
from
01a15f8 to
0ec0f43
Compare
neilalexander
changed the title
neilalexander
force-pushed
the
neil/nrgwg
branch
6 times, most recently
from
61b9a2e to
3cc6288
Compare
neilalexander
force-pushed
the
neil/nrgwg
branch
from
3cc6288 to
751dec0
Compare
neilalexander
changed the title
|
Have rebased, this one is ready for review. |
server/raft.go
Outdated
| // Wait for goroutines to shut down before trying to clean up | ||
| // the log below, otherwise we might end up deleting the store | ||
| // from underneath running goroutines. | ||
| n.Unlock() |
There was a problem hiding this comment.
We have these patterns for sure elsewhere but I try hard to avoid them if at all possible.
There was a problem hiding this comment.
I don't think we can avoid it here, it's the only way we can potentially avoid a deadlock if the runAs goroutine is finishing up processing some operations.
neilalexander
force-pushed
the
neil/nrgwg
branch
3 times, most recently
from
03b9f1d to
4a02a47
Compare
neilalexander
force-pushed
the
neil/nrgwg
branch
2 times, most recently
from
566c9cc to
dcd8167
Compare
server/raft.go
Outdated
| @@ -1728,12 +1731,17 @@ func (n *raft) Stop() { | |||
| n.shutdown(false) | |||
| } | |||
|
|
|||
| func (n *raft) WaitForStop() { | |||
| n.wg.Wait() | |||
There was a problem hiding this comment.
Should we check that the state is set to stopped to avoid calling in here and never returning?
server/raft.go
Outdated
| @@ -1744,12 +1752,12 @@ func (n *raft) shutdown(shouldDelete bool) { | |||
| if n.state.Swap(int32(Closed)) == int32(Closed) { | |||
| // If we get called again with shouldDelete, in case we were called first with Stop() cleanup | |||
| if shouldDelete { | |||
| n.wg.Wait() | |||
There was a problem hiding this comment.
Since we hold the lock here this may never return.
This should fix some logical races where multiple sets of goroutines can fight over the same store directory, i.e. when shutting down and recreating a group rapidly. Signed-off-by: Neil Twigg <neil@nats.io>
Signed-off-by: Neil Twigg <neil@nats.io>
neilalexander
force-pushed
the
neil/nrgwg
branch
from
dcd8167 to
d2ed0d9
Compare
neilalexander
force-pushed
the
neil/nrgwg
branch
3 times, most recently
from
3624123 to
dc65149
Compare
Signed-off-by: Neil Twigg <neil@nats.io>
neilalexander
force-pushed
the
neil/nrgwg
branch
from
dc65149 to
5622199
Compare
|
I have shuffled things around a bit:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This should fix some logical races where multiple sets of goroutines can fight over the same store directory, i.e. when shutting down and recreating a group rapidly.
Signed-off-by: Neil Twigg neil@nats.io