Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

clientv3/balancer: handle network partition in health check #8669

Merged
merged 5 commits into from
Oct 9, 2017
Merged

clientv3/balancer: handle network partition in health check #8669

merged 5 commits into from
Oct 9, 2017

Commits on Oct 9, 2017

  1. clientv3: only health-check when timeout elapses since last failure 

    Otherwise network-partitioned member with active health-check
server would not be gray-listed, making health-balancer stuck
with isolated endpoint.

Also clarifies some log messages.

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
    @gyuho
    gyuho committed Oct 9, 2017
    Configuration menu
    Copy the full SHA
    1704443 View commit details
    Browse the repository at this point in the history

  2. clientv3/balancer: mark partitioned member as unhealthy 

    Previous behavior is when server returns errors, retry
wrapper does not do anything, while passively expecting
balancer to gray-list the isolated endpoint. This is
problematic when multiple endpoints are passed, and
network partition happens.

This patch adds 'endpointError' method to 'balancer' interface
to actively(possibly even before health-check API gets called)
handle RPC errors and gray-list endpoints for the time being,
thus speeding up the endpoint switch.

This is safe in a single-endpoint case, because balancer will
retry no matter what in such case.

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
    @gyuho
    gyuho committed Oct 9, 2017
    Configuration menu
    Copy the full SHA
    fbed568 View commit details
    Browse the repository at this point in the history

  3. clientv3/integration: add balancer network partition tests 

    Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
    @gyuho
    gyuho committed Oct 9, 2017
    Configuration menu
    Copy the full SHA
    8224c74 View commit details
    Browse the repository at this point in the history

  4. words: whitelist more words 

    Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
    @gyuho
    gyuho committed Oct 9, 2017
    Configuration menu
    Copy the full SHA
    826de3c View commit details
    Browse the repository at this point in the history

  5. clientv3: pin any endpoint when all unhealthy 

    Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
    @gyuho
    gyuho committed Oct 9, 2017
    Configuration menu
    Copy the full SHA
    e9e17e3 View commit details
    Browse the repository at this point in the history