Fix racy requester shutdown logic #286
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change resolves the panic in bojand#268. The requester shutdown logic was not correctly waiting for the worker ticker to shutdown which could cause the shutdown logic to run concurrently with new workers being created. This doesn't fail race tests because the workers array is protected by a mutex, but the length of `b.workers` used for `b.workers[i].Stop()` is not guaranteed to be the same as the length used to collect errors from errC. This is possible for example with the default concurrency schedule, and a service that completes `-c` requests faster than it takes to create `-c` workers. This means that the load test finishes before the first worker `TickValue` has been processed. The workers that are created after the `b.workers[i].Stop()` loop run forever and the process doesn't respond to SIGINT, it has to be SIGKILLed.
|
Thanks for the PR! Should be in 0.96.0. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change resolves the panic in #268. The requester shutdown logic was not
correctly waiting for the worker ticker to shutdown which could cause
the shutdown logic to run concurrently with new workers being created.
This doesn't fail race tests because the workers array is protected by
a mutex, but the length of
b.workersused forb.workers[i].Stop()is not guaranteed to be the same as the length used to collect errors
from errC.
This is possible for example with the default concurrency schedule, and
a service that completes
-crequests faster than it takes to create-cworkers. This means that the load test finishes before the firstworker
TickValuehas been processed. The workers that are createdafter the
b.workers[i].Stop()loop run forever and the process doesn'trespond to SIGINT, it has to be SIGKILLed.