Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement close notifier and timeout on executors #4889

Merged
merged 1 commit into from
Nov 25, 2015

Conversation

corylanou
Copy link
Contributor

This PR does two things:

  1. Adds the CloseNotifier implementation for our Query handler endpoint. This will pass in a closing channel to the executors. If the HTTP client disconnects, we will cancel the query on the server.
  2. Added a timeout within the executor loop while retrieving results. We have a suspicion that we may stop reading on a channel for some reason (short read) and thus the execute never exits. As a result, it could leave a bolt transaction open and create a deadlock situation.

@@ -330,6 +331,7 @@ func (s *Service) runContinuousQueryAndWriteResult(cq *ContinuousQuery) error {
if res.Err != nil {
return res.Err
}
close(closing)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move the close(closing) right after the creation (L320) and do it as a defer. That way it'll handle it for panics too. If it closes and the results are done reading the it shouldn't do anything anyway.

@corylanou corylanou force-pushed the mapper-execute-close branch from a7bb47a to 24e05df Compare November 25, 2015 02:00
break
case <-time.After(30 * time.Second):
// This should never happen, so if it does, it is a problem
out <- &models.Row{Err: fmt.Errorf("execute was closed by caller")}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error text still looks wrong.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh, I missed this one. will fix.

@otoolep
Copy link
Contributor

otoolep commented Nov 25, 2015

@corylanou -- can you scrub the error messages? We can't differentiate between cancellation and timeout right now.

@corylanou corylanou force-pushed the mapper-execute-close branch from 24e05df to be488b7 Compare November 25, 2015 03:10
@corylanou
Copy link
Contributor Author

@otoolep fixed the duplicated error messages.

@otoolep
Copy link
Contributor

otoolep commented Nov 25, 2015

OK, seems good now. I never like to see magic numbers in the code, and if nothing else I would think the 30 seconds value should be const out in tsdb/config.go (but not configurable). However, if this code is going anyway with the refactor by @benbjohnson it's doesn't matter.

corylanou added a commit that referenced this pull request Nov 25, 2015
Implement close notifier and timeout on executors
@corylanou corylanou merged commit ce8cf05 into master Nov 25, 2015
@corylanou corylanou deleted the mapper-execute-close branch November 25, 2015 14:24
func (e *ShowMeasurementsExecutor) Execute() <-chan *models.Row {
func (e *ShowMeasurementsExecutor) Execute(closing <-chan struct{}) <-chan *models.Row {
// It's important that all resources are released when execution completes.
e.close()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think e.close() could not be invoked here.
Please ref issue #4926 and pr #4927
Thanks! @corylanou

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants