Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout calls if they have waited too long in the queue #87

Closed
rkarthik007 opened this issue Mar 7, 2018 · 1 comment
Closed

Timeout calls if they have waited too long in the queue #87

rkarthik007 opened this issue Mar 7, 2018 · 1 comment
Assignees
Labels
kind/enhancement This is an enhancement of an existing feature

Comments

@rkarthik007
Copy link
Collaborator

When thing slow down, it is possible that it takes a lot of time for the operations in the rpc queue to start getting handled.If they have waited too long, there is no real purpose in still trying to handle them. We might as well fail them, the way we do when we hit a back-pressure like scenario.

Adding a GFLAG to limit the amount of time an op waits in the queue. If we take longer than that to start handling the request. The request will be promptly failed, so that the resources can be used to work on more timely requests. 

@rkarthik007 rkarthik007 added the kind/enhancement This is an enhancement of an existing feature label Mar 7, 2018
yugabyte-ci pushed a commit that referenced this issue Apr 10, 2018
Summary:
We do not expect calls to spend too long waiting
in the queue. If they spend way too long, it means that
the system is already in a bad state, and we may be applying
backpressure to prevent new requests from joining the queue.

To avoid spending time on calls that are too old, we will
abort calls, if they have just been waiting in the queue for too long
and move on to process more "timely" requests.

Test Plan:
Added a test.
+ Jenkins

Reviewers: kannan, bharat, hector, mikhail

Reviewed By: bharat, hector, mikhail

Subscribers: bharat, ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D4251
@amitanandaiyer
Copy link
Contributor

Committed in d537ed2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement This is an enhancement of an existing feature
Projects
None yet
Development

No branches or pull requests

2 participants